Named after its electrical counterpart, a software circuit breaker monitors calls to an external dependency and trips open — stopping all outgoing calls — when the failure rate exceeds a threshold. The breaker enters a half-open state after a configurable timeout, allowing a probe request through. If the probe succeeds, the breaker closes and normal traffic resumes; if it fails, the open state is maintained.
In a scraping context, a circuit breaker prevents a worker pool from hammering a target site that has gone down or started returning consistent errors, which would waste resources and could trigger IP bans. It also protects downstream consumers from cascading failures when a scraping tier is degraded.
Circuit breakers are especially useful in distributed scraping systems where many worker nodes target the same site. Without a breaker, all workers independently retry on the same failing endpoint, amplifying the load exactly when the site is most vulnerable.