Why circuit breakers are the missing layer above retry

Retry logic protects you from single failures. Rate limit budgets protect you from bursts. Neither protects you from sustained provider degradation. That is the job of the circuit breaker.

Here is the failure mode retry alone cannot fix: your AI provider enters a rolling degradation — not fully down, but returning 503s for 40% of requests. Your retry loop dutifully attempts each failing request 5 times. Every attempt consumes tokens (billed) and rate limit budget (finite). After thirty minutes, you have spent your daily token quota, exhausted your retry budget, added seconds of latency to every user request, and none of your users got a response.

The correct response — the response the circuit breaker gives you — is to stop trying after enough failures pile up. Fail every subsequent request immediately for a period. Then probe carefully to see if the provider recovered. If it did, resume normal traffic. If not, keep the breaker open.

The three problems circuit breakers solve

1. Wasted resources during outages

Every request against a failing provider costs tokens (billed even on failure in some setups), consumes rate limit headroom, and holds a client thread. A breaker stops all three costs at the outage's start.

2. Cascading latency

Retry loops add latency to every request. During degradation, every request pays the retry cost. Users experience a slow site, not a broken one — harder to detect, worse to endure. A breaker fails fast, exposing the outage to your monitoring instead of hiding it in latency percentiles.

3. Recovery stampedes

When a provider recovers from an outage, every client hits it at once — often re-triggering the outage. A properly implemented breaker's half-open probe (Chapter 4) prevents this coordination.

Where the circuit breaker sits in your stack

# Order of operations for every AI API call:
# 
# 1. Circuit breaker check  — is the path open?
# 2. Client-side rate limiter  — do we have budget?
# 3. Idempotency check  — has this exact request already run?
# 4. Retry loop with backoff  — up to N attempts
# 5. Actual API call
# 6. Update breaker state based on outcome
# 7. Fallback if retries exhausted
# 
# Skip step 1 and every subsequent step is wasted during an outage.

This guide is the complete pattern. It covers every state transition, every threshold decision, the interaction with retries from G2, distributed coordination for multi-instance services, and a copy-paste-ready Python implementation. All of it comes from patterns tested in production against the actual providers.

The three states — closed, open, half-open

Every circuit breaker is a state machine with three states. Understanding the transitions between them is understanding the whole pattern.

Closed — normal operation

The breaker is closed. Requests flow through freely. Each result (success or failure) is recorded. When the failure count in a rolling window crosses a threshold, the breaker transitions to open.

Open — failing fast

The breaker is open. Every request fails immediately without touching the provider. This state has a duration — the "reset timeout." Common durations: 30 seconds for transient issues, 5 minutes for sustained outages. When the timeout expires, the breaker transitions to half-open.

Half-open — probing recovery

The breaker allows a small number of test requests through (often just one at a time). If the test succeeds, the breaker returns to closed. If it fails, the breaker returns to open and the reset timeout resets.

from enum import Enum

class CircuitState(Enum):
    CLOSED = "closed"      # normal operation
    OPEN = "open"          # failing fast
    HALF_OPEN = "half_open"  # probing recovery

The state diagram

              [failures > threshold]
    CLOSED  ————————————————————————>  OPEN
      ^                                  |
      |                                  | [reset_timeout elapsed]
      |                                  v
      |    [probe succeeds]           HALF_OPEN
      +————————————————————————————————— |
                                         |
                                         | [probe fails]
                                         v
                                        OPEN
                          (reset_timeout resets from now)

Why three states, not two

A two-state breaker (closed / open) would need a full reset moment where all traffic resumes at once. Because ten thousand clients coordinate on the same reset timeout, that moment re-triggers the outage. Half-open gives you a controlled probe — one request, one client — that determines recovery without a stampede.

Success counting during half-open

Some implementations require multiple consecutive successes in half-open before transitioning to closed. This adds resilience against a provider that intermittently succeeds during recovery. Two-of-three or three-of-five are common patterns. Single-probe (one success closes the breaker) works if your traffic volume is high enough to detect regression quickly.

Triggering thresholds — count, percentage, sliding windows

The breaker opens when failures cross a threshold. The threshold definition matters enormously — a poorly sized threshold either flaps open constantly (annoying but survivable) or opens too rarely to prevent damage (dangerous).

Threshold type 1: consecutive failures

if consecutive_failures >= 5:
    breaker.open()

Simple and cheap. Works well when failures are truly transient — one failure resets the counter. Fails when failures are intermittent (fail, succeed, fail, succeed) because the counter never accumulates.

Threshold type 2: failure count in window

if failures_in_last_30s >= 10:
    breaker.open()

Better for intermittent failures. Counts total failures over a rolling window (typically 30-60 seconds). Insensitive to success rate — a service doing 1,000 rps with 10 failures/minute has a 0.02% error rate but still trips the breaker.

Threshold type 3: failure percentage in window

if (failures / total) > 0.5 and total >= 20:
    breaker.open()

The right default for most services. Trips when failure percentage crosses a threshold, but requires a minimum sample size to avoid tripping on the first request in a low-traffic period.

Sliding window implementation

from collections import deque
import time, threading

class SlidingWindowCounter:
    def __init__(self, window_seconds=30):
        self.window = window_seconds
        self.events = deque()  # (timestamp, success:bool)
        self.lock = threading.Lock()
    
    def record(self, success):
        with self.lock:
            now = time.time()
            # Drop events older than window
            while self.events and self.events[0][0] < now - self.window:
                self.events.popleft()
            self.events.append((now, success))
    
    def stats(self):
        with self.lock:
            now = time.time()
            while self.events and self.events[0][0] < now - self.window:
                self.events.popleft()
            total = len(self.events)
            failures = sum(1 for _, ok in self.events if not ok)
            return total, failures

Choosing thresholds for AI APIs

For AI provider calls, we recommend:

• Window: 30 seconds. Long enough to sample meaningfully, short enough to react to real outages.
• Minimum sample size: 20 requests. Prevents tripping on the first 429 in a quiet minute.
• Failure percentage: 50%. High enough to distinguish real degradation from normal transients (5-10% healthy baseline).
• Reset timeout: 30 seconds initially, exponentially backoff on repeated open transitions.

The half-open probe — probing recovery without a stampede

The half-open state is where circuit breakers earn their complexity. Done wrong, it re-triggers the outage on every recovery. Done right, it detects recovery precisely without adding load.

The classic mistake: unlimited probe traffic

# WRONG — allows unlimited requests during half-open
if state == HALF_OPEN:
    result = call_api()
    if result.ok:
        state = CLOSED
    else:
        state = OPEN

If a hundred goroutines simultaneously transition from open to half-open, all hundred try their probe simultaneously. That is not a probe — that is normal traffic hitting a recovering provider. Guaranteed to re-open the breaker.

The correct pattern: single-request probe

from threading import Lock

class ProbeLimiter:
    def __init__(self, max_probes=1):
        self.max = max_probes
        self.active = 0
        self.lock = Lock()
    
    def acquire(self):
        with self.lock:
            if self.active >= self.max:
                return False
            self.active += 1
            return True
    
    def release(self):
        with self.lock:
            self.active -= 1

# During half-open:
if state == HALF_OPEN:
    if not probe_limiter.acquire():
        # Another thread is already probing — fail fast
        raise CircuitOpenError()
    try:
        result = call_api()
        if result.ok:
            state = CLOSED
        else:
            state = OPEN
            reset_timer.restart()
    finally:
        probe_limiter.release()

Requiring multiple consecutive successes

A single successful probe may reflect a moment of recovery rather than a real one. For higher-stakes services, require N-of-M consecutive successes.

class HalfOpenProbe:
    def __init__(self, required_successes=3):
        self.required = required_successes
        self.consecutive = 0
    
    def record(self, success):
        if success:
            self.consecutive += 1
            if self.consecutive >= self.required:
                return "close"  # transition to CLOSED
        else:
            return "open"  # transition to OPEN, reset timer
        return "continue"  # stay HALF_OPEN, probe again

Exponential reset timeout

If the breaker keeps flipping to half-open and back to open, the provider is not recovered. Increase the reset timeout exponentially on repeated failures — but cap it at a reasonable maximum (5 minutes) so recovery is eventually detected.

class ResetTimer:
    def __init__(self, base=30.0, cap=300.0):
        self.base = base
        self.cap = cap
        self.failures = 0
        self.next_probe_at = time.time() + base
    
    def elapsed(self):
        return time.time() >= self.next_probe_at
    
    def on_probe_failure(self):
        self.failures += 1
        # Exponential backoff, capped
        delay = min(self.cap, self.base * (2 ** self.failures))
        self.next_probe_at = time.time() + delay
    
    def on_probe_success(self):
        self.failures = 0  # reset the escalation

Per-provider vs global breakers — scoping decisions

A single global breaker treats every AI call as the same. That is almost always wrong. Different providers, models, and API surfaces fail independently. Your breakers should reflect this reality.

The scoping hierarchy

# Bad: single global breaker for all AI calls
GLOBAL_BREAKER  # trips when ANY provider degrades
                # penalizes healthy providers for one bad one

# Better: per-provider breakers
BREAKERS = {
    "anthropic": CircuitBreaker(),
    "openai": CircuitBreaker(),
    "gemini": CircuitBreaker(),
}

# Best: per-provider, per-model, per-region breakers
BREAKERS = {
    ("anthropic", "claude-sonnet-4-6"): CircuitBreaker(),
    ("anthropic", "claude-opus-5"): CircuitBreaker(),
    ("openai", "gpt-5-6"): CircuitBreaker(),
    ("bedrock", "us-east-1", "claude-sonnet-4-6"): CircuitBreaker(),
    ("bedrock", "us-west-2", "claude-sonnet-4-6"): CircuitBreaker(),
}

Why granular scoping matters

Claude Opus on Anthropic direct can fail without Claude Sonnet failing. Anthropic API can degrade while Bedrock's Claude is fine. Vertex us-central1 can throttle while Vertex europe-west4 is healthy. If your global breaker fires for any of these, it takes down paths that are still healthy.

The lookup pattern

from typing import Tuple, Dict

class BreakerRegistry:
    def __init__(self, factory):
        self.factory = factory  # callable that creates a fresh breaker
        self.breakers: Dict[Tuple, CircuitBreaker] = {}
        self.lock = threading.Lock()
    
    def get(self, key: Tuple):
        # Lazy creation — breakers created on first request per key
        with self.lock:
            if key not in self.breakers:
                self.breakers[key] = self.factory()
            return self.breakers[key]

registry = BreakerRegistry(
    factory=lambda: CircuitBreaker(
        failure_threshold=0.5,
        window_seconds=30,
        reset_timeout=30,
    )
)

# Call site
key = ("anthropic", "claude-sonnet-4-6")
breaker = registry.get(key)
result = breaker.call(lambda: anthropic_client.messages.create(...))

When to collapse the hierarchy

Very fine-grained breakers have too little traffic to reach minimum sample size. If your service does 10 requests per minute per (provider, model, region), a 30-second window rarely accumulates enough samples to trip. Collapse to (provider, model) or just (provider) until traffic justifies finer scope.

Correlated failures across scopes

If Anthropic direct fails, Claude on Bedrock usually stays up (they're independent infrastructure). But sometimes an underlying issue (a bad model checkpoint, a shared network path) affects both. Your fallback logic must handle the case where multiple scopes fail simultaneously.

def call_with_fallback(messages):
    # Try Anthropic direct
    try:
        return registry.get(("anthropic", "sonnet-4-6")).call(
            lambda: anthropic_client.messages.create(...)
        )
    except CircuitOpenError:
        pass
    # Try Bedrock (different scope key = different breaker)
    try:
        return registry.get(("bedrock", "us-east-1", "sonnet-4-6")).call(
            lambda: bedrock_client.invoke_model(...)
        )
    except CircuitOpenError:
        pass
    # Try downgraded model
    return registry.get(("anthropic", "haiku-4-5")).call(
        lambda: anthropic_client.messages.create(model="claude-haiku-4-5", ...)
    )

What triggers a breaker — signal quality matters

Not every failure should trip the breaker. A 400 Bad Request from a bad prompt is your fault, not the provider's — tripping the breaker helps nothing. Signal classification is as critical here as it was for retry logic.

The three failure categories

1. Trips the breaker — provider is degraded

• 429 Too Many Requests (persistent, not one-off).
• 500, 502, 503, 504 Server Errors.
• 529 Overloaded (Anthropic).
• Network errors: connection refused, connection reset, DNS failure.
• Read/write timeouts that exceed reasonable variance.

2. Ignored by breaker — your fault, not the provider's

• 400 Bad Request — malformed input.
• 401 Unauthorized — credentials issue.
• 403 Forbidden — permissions.
• 404 Not Found — wrong endpoint or model.
• 422 Unprocessable Entity — validation error.

3. Signal but not tripping — content or model refusals

• stop_reason: refusal (Claude) — model chose not to answer.
• finish_reason: content_filter (OpenAI) — input or output filtered.
• blockReason: SAFETY (Gemini) — safety filter.

These are the model working correctly, not a provider outage. Don't trip. But do log — if refusal rate spikes for reasons unrelated to input distribution, that's a signal worth investigating.

Implementation of the classifier

from anthropic import (
    RateLimitError, APIStatusError, BadRequestError, APIConnectionError
)

def is_breaker_signal(exception) -> bool:
    # Returns True if this exception counts as a breaker-tripping failure.
    if isinstance(exception, BadRequestError):
        return False  # 400 — your fault
    if isinstance(exception, RateLimitError):
        return True  # 429 — provider capacity
    if isinstance(exception, APIStatusError):
        code = exception.status_code
        if code in (401, 403, 404, 422):
            return False  # your fault
        if code in (429, 500, 502, 503, 504, 529):
            return True  # provider fault
    if isinstance(exception, (APIConnectionError, ConnectionError, TimeoutError)):
        return True  # network — treat as provider fault
    return False  # unknown — don't trip on unknowns

def is_soft_success(response) -> bool:
    # Called on successful responses.
    # Returns True if response is a soft failure (refusal, filter).
    # Soft failures don't trip breaker but are logged.
    stop_reason = getattr(response, "stop_reason", None)
    if stop_reason in ("refusal", "content_filter"):
        return True
    return False

Latency as a signal

Explicit failures are the primary trigger, but severe latency degradation is also a signal. A provider that responds slowly is degraded even if requests eventually succeed. Some breakers include a p99 latency threshold: if p99 exceeds N seconds over the window, treat as degraded.

class LatencyAwareBreaker(CircuitBreaker):
    def __init__(self, *args, latency_threshold=15.0, **kwargs):
        super().__init__(*args, **kwargs)
        self.latency_threshold = latency_threshold
        self.recent_latencies = deque(maxlen=100)
    
    def record(self, success, latency=None):
        super().record(success)
        if latency is not None:
            self.recent_latencies.append(latency)
            # Trip if p95 latency exceeds threshold
            if len(self.recent_latencies) >= 20:
                sorted_l = sorted(self.recent_latencies)
                p95 = sorted_l[int(len(sorted_l) * 0.95)]
                if p95 > self.latency_threshold:
                    self._trip()

Combining with retries — order of operations

Circuit breakers and retry logic (from G2) are complementary, not alternatives. But their interaction has to be designed carefully. Wrong ordering causes either failure amplification or premature circuit opening.

The correct order: breaker outside retry

def call_with_breaker_and_retry(fn, breaker, max_attempts=5):
    # Check breaker BEFORE any attempt
    if not breaker.allow_request():
        raise CircuitOpenError()
    
    # Retry loop happens INSIDE the breaker envelope
    for attempt in range(max_attempts):
        try:
            result = fn()
            breaker.record_success()
            return result
        except Exception as e:
            if is_breaker_signal(e):
                breaker.record_failure()
            if not is_retryable(e) or attempt == max_attempts - 1:
                raise
            time.sleep(full_jitter(attempt))

The breaker check happens once at the start of the whole retry sequence. The breaker records outcomes throughout, but a single logical operation (with N retries) is one thing to the breaker.

What NOT to do — breaker inside retry

# WRONG — breaker check inside each retry attempt
for attempt in range(max_attempts):
    if not breaker.allow_request():  # check each iteration
        raise CircuitOpenError()
    try:
        return fn()
    except Exception as e:
        breaker.record_failure()
        time.sleep(full_jitter(attempt))

The problem: retries during a rate limit spike each get counted as separate failures. Five retries against a rate-limited provider count as five failures. The breaker trips faster than the failure rate actually justifies.

Adaptive: retry count as breaker signal

An advanced variant treats the number of retries needed as the signal, not the raw success/failure. A request that succeeds on the first try is a healthy signal. A request that succeeds only after 3 retries is a soft failure — something is degrading.

def call_with_adaptive_signal(fn, breaker, max_attempts=5):
    if not breaker.allow_request():
        raise CircuitOpenError()
    
    for attempt in range(max_attempts):
        try:
            result = fn()
            # Successful on first try = full success
            # Required retries = degraded signal
            if attempt == 0:
                breaker.record_success()
            elif attempt < 3:
                breaker.record_soft_success()  # partial credit
            else:
                breaker.record_failure()  # succeeded but too costly
            return result
        except Exception as e:
            if not is_retryable(e) or attempt == max_attempts - 1:
                breaker.record_failure()
                raise
            time.sleep(full_jitter(attempt))

Circuit breaker with fallback in one wrapper

def robust_call(primary_fn, fallback_fn, breaker):
    if breaker.allow_request():
        try:
            return call_with_retry(primary_fn, max_attempts=5)
        except CircuitOpenError:
            pass
        except Exception as e:
            if not is_breaker_signal(e):
                raise  # don't fall back on your own bugs
    
    # Primary circuit open or exhausted retries — fallback
    return fallback_fn()

Distributed circuit breakers — sharing state across instances

Per-process breakers work correctly on a single instance. Across a fleet of ten instances, each has its own view of provider health — and each has to independently trip. That's ten instances doing full-degradation-detection before any of them fail fast.

The tradeoff

Distributed breakers share state across instances. All ten instances see the same failure counts, all trip at the same moment, all recover together. But you add a hard dependency on the state store (Redis, DynamoDB, etc.) — if it's down, your breakers don't work.

When to use distributed vs per-instance

• 1-3 instances: per-instance is fine. Coordination overhead exceeds benefit.
• 4-10 instances: per-instance still workable. Fleet reaches degraded state within one window.
• 10+ instances: distributed increasingly valuable. Amplification of independent detection is significant.
• Latency-critical or shared-quota: distributed regardless of instance count. You want unified state.

Redis-backed sliding window

import redis, time, json

class DistributedBreaker:
    def __init__(self, redis_client, key, config):
        self.r = redis_client
        self.key = key
        self.window = config.get("window_seconds", 30)
        self.threshold = config.get("failure_threshold", 0.5)
        self.min_samples = config.get("min_samples", 20)
        self.reset_timeout = config.get("reset_timeout", 30)
    
    def _state_key(self):
        return f"breaker:{self.key}:state"
    
    def _events_key(self):
        return f"breaker:{self.key}:events"
    
    def allow_request(self):
        state_data = self.r.get(self._state_key())
        if not state_data:
            return True  # default open (allow)
        state = json.loads(state_data)
        
        if state["state"] == "closed":
            return True
        if state["state"] == "open":
            if time.time() >= state["reopens_at"]:
                # Transition to half-open by claiming the probe
                probe_claimed = self.r.set(
                    f"breaker:{self.key}:probe_lock",
                    "1",
                    nx=True, ex=30
                )
                if probe_claimed:
                    self._transition("half_open")
                    return True
                # Someone else is probing — fail fast
                return False
            return False
        if state["state"] == "half_open":
            # Only the probe holder gets through
            return self.r.exists(f"breaker:{self.key}:probe_lock") == 0
        return True
    
    def record(self, success):
        # Add to sliding window
        now = time.time()
        pipe = self.r.pipeline()
        pipe.zadd(self._events_key(), {f"{now}:{'ok' if success else 'fail'}": now})
        pipe.zremrangebyscore(self._events_key(), 0, now - self.window)
        pipe.zrange(self._events_key(), 0, -1)
        _, _, events = pipe.execute()
        
        total = len(events)
        failures = sum(1 for e in events if b"fail" in e)
        
        if total >= self.min_samples and failures / total >= self.threshold:
            self._transition("open", reopens_at=now + self.reset_timeout)
    
    def _transition(self, new_state, reopens_at=None):
        state = {"state": new_state}
        if reopens_at:
            state["reopens_at"] = reopens_at
        self.r.set(self._state_key(), json.dumps(state), ex=600)

The probe lock

Note the probe_lock pattern: exactly one instance across the fleet gets to send the half-open probe. This prevents the recovery-time stampede that plagues naive distributed breakers.

Falling back when Redis is unavailable

Distributed breakers depend on Redis. If Redis fails, your breakers become permissive (allow everything) or restrictive (block everything) — both problematic. The safe pattern is per-instance fallback: each instance falls back to a local breaker with tighter thresholds when the distributed breaker cannot be reached.

class HybridBreaker:
    def __init__(self, distributed, local):
        self.dist = distributed  # Redis-backed
        self.local = local       # per-process fallback
    
    def allow_request(self):
        try:
            return self.dist.allow_request()
        except redis.ConnectionError:
            return self.local.allow_request()
    
    def record(self, success):
        try:
            self.dist.record(success)
        except redis.ConnectionError:
            pass  # Redis down — silently
        # Always record locally too
        self.local.record(success)

Multi-tier breakers — fallback chains done right

A single breaker protects one path. Real production usually has multiple paths: primary provider, secondary provider, degraded fallback. Coordinating breakers across a fallback chain requires care.

The fallback chain

# Primary: Anthropic direct
# Secondary: Bedrock (same model)
# Tertiary: Downgraded model (Sonnet -> Haiku)
# Quaternary: Cached / degraded response

primary_breaker = CircuitBreaker(name="anthropic-sonnet")
secondary_breaker = CircuitBreaker(name="bedrock-sonnet")
tertiary_breaker = CircuitBreaker(name="anthropic-haiku")

def robust_completion(messages):
    for breaker, fn in [
        (primary_breaker, lambda: anthropic_call(messages, "sonnet-4-6")),
        (secondary_breaker, lambda: bedrock_call(messages, "sonnet-4-6")),
        (tertiary_breaker, lambda: anthropic_call(messages, "haiku-4-5")),
    ]:
        try:
            return breaker.call(fn)
        except CircuitOpenError:
            continue
        except Exception as e:
            if not is_breaker_signal(e):
                raise  # don't fall back on your own bugs
            # Otherwise fall through to next
    return cached_response()  # ultimate fallback

Each tier has independent state

The primary breaker can be open while secondary is closed. This is the whole point of having multiple tiers: your paths fail independently. If they didn't, there'd be no reason for fallback.

The correlated failure trap

Sometimes all your paths fail together. Anthropic direct and Bedrock's Claude both go down at the same moment because there's a shared underlying issue. Your fallback chain rapidly cascades — each breaker takes a moment to trip, so during the cascade, real user requests hit real failing paths.

Mitigation: an ultimate fallback that doesn't call any AI provider. A cached response, a degraded UX message, a rules-based response. Anything that acknowledges "AI is unavailable right now" without adding to the failing provider's load.

def ultimate_fallback(user_input):
    # No AI call. Deterministic degraded response.
    return {
        "content": "Our AI service is temporarily unavailable. "
                   "Please try again in a few minutes.",
        "is_degraded": True,
    }

Metrics per tier

Instrument each tier separately. When fallback-tier usage spikes, you know your primary is failing. When ultimate-fallback usage spikes, everything is failing — page someone.

tier_used = Counter(
    "ai_tier_used_total",
    "Which fallback tier served the request",
    ["tier"]  # values: "primary", "secondary", "tertiary", "ultimate"
)

# In the fallback chain
try:
    result = primary_breaker.call(fn)
    tier_used.labels(tier="primary").inc()
    return result
except CircuitOpenError:
    # ... continue to next tier, label accordingly

Health-based routing

Advanced: route to whichever tier is healthiest, not just first-tried. If primary is open but secondary is closed, go directly to secondary rather than checking primary first. Requires each breaker to expose its current state as a fast read.

Observability — watching state transitions

A circuit breaker without observability is a black box. When something goes wrong, you can't tell whether the breaker prevented an outage or caused one. Instrument every state transition.

The metrics you need

from prometheus_client import Counter, Gauge, Histogram

state_transitions = Counter(
    "circuit_breaker_transitions_total",
    "State transitions",
    ["breaker", "from_state", "to_state", "reason"]
)

current_state = Gauge(
    "circuit_breaker_state",
    "Current state (0=closed, 1=half_open, 2=open)",
    ["breaker"]
)

rejected_requests = Counter(
    "circuit_breaker_rejected_total",
    "Requests failed fast due to open breaker",
    ["breaker"]
)

failure_rate = Gauge(
    "circuit_breaker_failure_rate",
    "Current failure rate in window (0.0 to 1.0)",
    ["breaker"]
)

Alert on the transitions, not the state

An open breaker for two seconds is normal — a transient. An open breaker for two minutes is an outage. Alert on the transition to open (informational) and on sustained open state (critical).

groups:
- name: circuit_breakers
  rules:
  - alert: BreakerOpened
    expr: increase(circuit_breaker_transitions_total{to_state="open"}[5m]) > 0
    for: 0m
    labels: {severity: info}
    annotations:
      summary: "{{ $labels.breaker }} transitioned to open"
  
  - alert: BreakerSustainedOpen
    expr: circuit_breaker_state{state="open"} == 2
    for: 5m
    labels: {severity: critical}
    annotations:
      summary: "{{ $labels.breaker }} has been open for 5+ minutes"
  
  - alert: BreakerFlapping
    expr: rate(circuit_breaker_transitions_total[15m]) > 0.2
    for: 15m
    labels: {severity: warning}
    annotations:
      summary: "{{ $labels.breaker }} flapping between states"

The dashboard row

Add one row per breaker to your service dashboard: state (colored), failure rate over time, rejected request rate. During any incident, this row tells you immediately which paths are open, which are recovering, and how much traffic is being shed.

Logging state changes

class ObservableBreaker(CircuitBreaker):
    def _transition(self, new_state, reason=""):
        old_state = self.state
        super()._transition(new_state, reason)
        logger.warning(
            "Circuit breaker transition",
            extra={
                "breaker": self.name,
                "from_state": old_state.value,
                "to_state": new_state.value,
                "reason": reason,
                "window_stats": self.window.stats(),
            }
        )
        state_transitions.labels(
            breaker=self.name,
            from_state=old_state.value,
            to_state=new_state.value,
            reason=reason,
        ).inc()
        current_state.labels(breaker=self.name).set(
            {"closed": 0, "half_open": 1, "open": 2}[new_state.value]
        )

Post-incident review

After every incident, review breaker behavior. Did the right breaker trip? Did it trip fast enough? Did the half-open probe detect recovery correctly? Did any fallback tier get overwhelmed? Answers to these tune your thresholds over time.

When NOT to use circuit breakers

Circuit breakers are the right pattern for most production AI services. But not all. Applying them where they don't help adds complexity, or worse, introduces bugs.

When circuit breakers are the wrong answer

1. Truly non-critical background work

A nightly batch that summarizes documents can just retry from scratch tomorrow. Circuit breaker adds no value; retry-with-backoff and eventual failure is enough. The complexity you'd add is unjustified for work that has no user-facing impact.

2. Low-volume services

If your service makes 10 AI calls per hour, you don't get enough samples for the breaker to work correctly. Minimum sample size means the breaker rarely trips, and when it does, the small volume means the "protection" is imperceptible. Rely on retry and manual monitoring instead.

3. When the failure is your fault, not the provider's

If your bug is producing malformed prompts that get 400s, the breaker sees "failures" and trips — but the "outage" is entirely on your side. Every request in-flight fails identically because your code is broken. Circuit breakers help when the provider is degraded; they don't help when you are.

4. When you have no fallback path

A circuit breaker's value is failing fast so you can either wait or fall back. If you have no fallback and can't wait, all the breaker does is make user requests fail sooner. Compare "wait 30 seconds and get an answer" to "fail immediately with an error" — for some workloads, the wait is preferable.

5. Very short outages

If the provider's outage lasts less than one breaker window (typically 30 seconds), the breaker may not have time to trip — and by the time it would, the outage is over. Retry with jittered backoff handles these cases without any breaker involvement.

The over-engineering trap

Junior engineers often want to apply every pattern from every book. Multi-tier distributed breakers with per-region granularity, latency-aware triggers, adaptive thresholds — the whole stack. For a service doing 100 requests per minute, this is over-engineered. The extra code has bugs that create worse outages than the breaker prevents.

The right progression for a growing service:

1. Start with retry + manual monitoring. No breaker yet.
2. Add per-provider breakers when you have >3 providers or multi-tier fallback.
3. Add distributed coordination when you have >10 instances or shared quotas.
4. Add per-model / per-region granularity only after seeing scope-limited failures in real incidents.

When circuit breakers ARE the right answer

• User-facing services where latency matters and slow responses degrade UX badly.
• Multi-provider setups where fallback exists and you need to route around failures.
• High-volume services where amplification of retries during outages consumes real budget.
• Services with SLA commitments where sustained degradation triggers penalties.
• Services with billing implications where each failed call still costs tokens.

The complete circuit breaker implementation

Every idea in this guide, packaged as a copy-paste Python class. Read it, understand every piece, adapt for your provider mix and traffic profile.

import time, threading, logging
from collections import deque
from enum import Enum
from typing import Callable, Optional
from dataclasses import dataclass, field

logger = logging.getLogger(__name__)


class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


class CircuitOpenError(Exception):
    # Raised when the breaker is open and rejects a request.
    pass


@dataclass
class CircuitBreakerConfig:
    name: str = "default"
    failure_threshold: float = 0.5         # 50% failure rate
    window_seconds: float = 30.0            # sliding window size
    min_samples: int = 20                   # min sample size for tripping
    reset_timeout: float = 30.0             # initial reset delay
    reset_timeout_cap: float = 300.0        # max reset delay (5 min)
    required_successes_in_half_open: int = 2


class CircuitBreaker:
    def __init__(self, config: CircuitBreakerConfig):
        self.config = config
        self.state = CircuitState.CLOSED
        self.events = deque()  # (timestamp, success:bool)
        self.reopens_at: Optional[float] = None
        self.consecutive_reset_failures = 0
        self.half_open_successes = 0
        self.probe_active = False
        self.lock = threading.Lock()
        self._on_state_change: Optional[Callable] = None
    
    def on_state_change(self, callback: Callable):
        self._on_state_change = callback
    
    def allow_request(self) -> bool:
        with self.lock:
            now = time.time()
            if self.state == CircuitState.CLOSED:
                return True
            if self.state == CircuitState.OPEN:
                if self.reopens_at and now >= self.reopens_at:
                    if not self.probe_active:
                        self.probe_active = True
                        self._transition(CircuitState.HALF_OPEN)
                        return True
                return False
            if self.state == CircuitState.HALF_OPEN:
                if not self.probe_active:
                    self.probe_active = True
                    return True
                return False  # someone else is probing
            return False
    
    def record_success(self):
        with self.lock:
            now = time.time()
            self._add_event(now, True)
            if self.state == CircuitState.HALF_OPEN:
                self.half_open_successes += 1
                if self.half_open_successes >= self.config.required_successes_in_half_open:
                    self._transition(CircuitState.CLOSED)
                    self.half_open_successes = 0
                    self.consecutive_reset_failures = 0
                    self.probe_active = False
                else:
                    self.probe_active = False
    
    def record_failure(self):
        with self.lock:
            now = time.time()
            self._add_event(now, False)
            if self.state == CircuitState.HALF_OPEN:
                # Probe failed — back to open with longer timeout
                self.probe_active = False
                self.half_open_successes = 0
                self.consecutive_reset_failures += 1
                self._schedule_next_probe()
                self._transition(CircuitState.OPEN)
            elif self.state == CircuitState.CLOSED:
                total, failures = self._window_stats(now)
                if total >= self.config.min_samples and \
                   failures / total >= self.config.failure_threshold:
                    self._schedule_next_probe()
                    self._transition(CircuitState.OPEN)
    
    def _schedule_next_probe(self):
        delay = min(
            self.config.reset_timeout_cap,
            self.config.reset_timeout * (2 ** self.consecutive_reset_failures)
        )
        self.reopens_at = time.time() + delay
    
    def _window_stats(self, now):
        cutoff = now - self.config.window_seconds
        while self.events and self.events[0][0] < cutoff:
            self.events.popleft()
        total = len(self.events)
        failures = sum(1 for _, ok in self.events if not ok)
        return total, failures
    
    def _add_event(self, timestamp, success):
        self.events.append((timestamp, success))
        # Prune periodically to prevent unbounded growth
        cutoff = timestamp - self.config.window_seconds
        while self.events and self.events[0][0] < cutoff:
            self.events.popleft()
    
    def _transition(self, new_state: CircuitState):
        old_state = self.state
        if old_state == new_state:
            return
        self.state = new_state
        logger.warning(
            f"Breaker {self.config.name}: {old_state.value} -> {new_state.value}"
        )
        if self._on_state_change:
            self._on_state_change(old_state, new_state)
    
    def call(self, fn, *args, **kwargs):
        # Convenience wrapper: check breaker, call fn, record outcome.
        if not self.allow_request():
            raise CircuitOpenError(
                f"Circuit breaker {self.config.name} is open"
            )
        try:
            result = fn(*args, **kwargs)
            self.record_success()
            return result
        except Exception as e:
            if self._is_breaker_signal(e):
                self.record_failure()
            raise
    
    def _is_breaker_signal(self, exception) -> bool:
        # Override or configure per your provider set.
        status = getattr(exception, "status_code", None)
        if status in (400, 401, 403, 404, 422):
            return False
        if status in (429, 500, 502, 503, 504, 529):
            return True
        return isinstance(exception, (ConnectionError, TimeoutError))


# Usage
config = CircuitBreakerConfig(
    name="anthropic-sonnet",
    failure_threshold=0.5,
    window_seconds=30,
    min_samples=20,
    reset_timeout=30,
    required_successes_in_half_open=2,
)
breaker = CircuitBreaker(config)

def emit_state_metric(old, new):
    metrics.increment("breaker.transition",
                      tags={"from": old.value, "to": new.value})

breaker.on_state_change(emit_state_metric)

def call_claude(messages):
    return breaker.call(
        lambda: anthropic_client.messages.create(
            model="claude-sonnet-4-6", max_tokens=1024, messages=messages
        )
    )

What this implementation gives you

• All three states with correct transitions.
• Sliding window with configurable size and minimum sample threshold.
• Percentage-based failure detection (more robust than raw counts).
• Exponential reset timeout on repeated open transitions.
• Single-probe half-open (prevents recovery stampedes).
• Configurable required successes in half-open (N-of-M pattern).
• Callback for state transition observability.
• Thread-safe with a single lock.

What it doesn't give you

• Async support — write an async twin using asyncio.Lock.
• Distributed coordination — wrap with the Redis-backed variant from Chapter 8.
• Latency-based triggering — add p95 latency tracking as in Chapter 6.
• Automatic retry integration — combine with the retry decorator from G2 explicitly.

The final combined wrapper

from retry_decorator import retry_ai_call  # from G2

@retry_ai_call(max_attempts=5)
def call_with_retry(messages):
    return anthropic_client.messages.create(
        model="claude-sonnet-4-6", max_tokens=1024, messages=messages
    )

def robust_completion(messages, fallback_fn):
    try:
        return breaker.call(lambda: call_with_retry(messages))
    except (CircuitOpenError, Exception) as e:
        if isinstance(e, CircuitOpenError) or breaker.state == CircuitState.OPEN:
            return fallback_fn(messages)
        raise

Frequently asked questions

Library & further reading

Everywhere else on AI Error Hub that touches reliability, retries, breakers, and outages.