Published on

Building a Circuit Breaker Across Two Repos

Authors
Written by Ghostpen � How it works
This post was AI-generated from development pipeline artifacts and reviewed by me before publishing.

Introduction

I have been building an ecosystem of AI tools for analyzing gaming VODs. The core app, AutoAgent, watches video-on-demand recordings and extracts game knowledge using vision APIs. All LLM calls go through a shared library I built called llm-router, which routes requests across Anthropic, OpenAI, and Google based on rate limit headroom and cost.

Anthropic has been the primary vision provider because they support batch processing -- 50% cost savings. But when their API goes down with 500s and 529s, the entire learning pipeline stops. I wanted graceful failover to OpenAI and Google so that when Anthropic is having a bad day, work continues on another provider.

This is the story of how that feature shipped -- nine issues across two repos, with an AI development pipeline doing the implementation while I made the architectural calls and reviewed the output.

The Architectural Decision

My first question was where the failover logic belongs. The obvious place was AutoAgent's WatchLoop -- just catch the exception and retry with a different provider:

try:
    response = await router.complete(request)
except AnthropicError:
    request.provider = "openai"
    response = await router.complete(request)

I challenged that immediately. AutoAgent submits LLMRequest objects and gets LLMResponse objects back. It should not need to know whether Anthropic is returning 529s or if OpenAI's rate limit headers look different from Google's. That is provider-specific knowledge leaking above the abstraction boundary.

The circuit breaker had to go into llm-router. The router already selects providers based on rate limit headroom. It already tracks errors. It just did not use that signal to avoid sick providers. The foundation was there -- it just needed the decision logic on top.

Planning -- And Replanning

The first pass at task breakdown produced 11 issues with titles like "Circuit breaker state + error classification on _ProviderState." No explicit file paths, no verification steps, no acceptance criteria. Vague.

I scrapped them and replanned. The second attempt produced 9 issues -- 4 in llm-router, 5 in AutoAgent. Each issue was atomic: max 3 files, explicit file paths with reasons, verification commands, and acceptance criteria checkboxes. Two fewer issues but each one was actually completable without guessing at scope. The task boundaries mattered more than the task count.

What Got Built

The implementation started with error classification -- before the router can make health decisions, it needs to know what kind of error it is looking at. The classification function handles five categories:

def _classify_error(exc):
    # Timeouts -- not a server failure, different handling
    if type(exc).__name__ in ("TimeoutError", "APITimeoutError", "DeadlineExceeded"):
        return "timeout", None

    status_code = getattr(exc, "status_code", None)
    if status_code in (500, 502, 503, 529):
        return "server_error", status_code
    if status_code == 429:
        return "rate_limit", status_code    # Already handled by headroom
    if 400 <= status_code < 500:
        return "client_error", status_code  # Retrying won't help
    return "unknown", status_code

A 500 means the provider is sick. A 429 is a rate limit -- already handled. A 400 is a client error -- retrying will not fix bad input. 30 tests cover these categories.

The circuit breaker state ended up on _ProviderState, which already tracked rate limit headroom:

class _ProviderState:
    BREAKER_THRESHOLD = 3
    COOLDOWN_SECONDS = 300  # 5 minutes, fixed

    def __init__(self, provider):
        self.consecutive_errors = 0
        self.unhealthy_since = None
        self.half_open_lock = False

Three consecutive failures marks the provider unhealthy. After a 5-minute cooldown, one probe request tests recovery. Simple on paper.

What the Review Caught

This is where directing an AI pipeline pays for itself. The automated review step -- a separate model with fresh eyes -- found four critical issues that the developer agent missed.

Thundering herd on the probe. When a provider's cooldown expires, _select_provider() allows a probe request through. But in a concurrent system, multiple requests could all see the provider as ready-to-probe simultaneously and all send probes at once. That is the exact opposite of "gentle recovery." The fix was a lock boolean:

if state._is_half_open() and not state.half_open_lock:
    state.half_open_lock = True   # One probe at a time
    return state, False, None

Cooldown reset bug. Each new error was resetting unhealthy_since to the current timestamp, which meant the 5-minute cooldown kept restarting under sustained failures. A provider that was truly down would never get a recovery probe. The fix: only set the timestamp on the transition from healthy to unhealthy.

Interleaved error ambiguity. A successful response on provider B was clearing provider A's consecutive error count. Each provider's health is independent -- a success on one tells you nothing about the other.

Unbounded counter. Minor, but consecutive_errors had no cap. Capped it at the threshold value.

These are the kind of bugs you only discover under production load. The review step caught them in code review, before a single request hit the circuit breaker. That is 8 critical issues across two tasks, all resolved before shipping.

The AutoAgent Side

With 84 tests passing in llm-router, the AutoAgent integration was straightforward plumbing:

ALTER TABLE jobs ADD COLUMN provider_used TEXT;
ALTER TABLE jobs ADD COLUMN was_failover BOOLEAN DEFAULT FALSE;

The WatchLoop aggregates provider attribution across a batch. The UI shows which provider served each job with colored health dots in the header. All five AutoAgent tasks shipped without fix loops -- clean passes through review and QA.

What I Learned

The review step is the most valuable part of this pipeline. I made the architectural decisions -- where the circuit breaker lives, how errors are classified, what the threshold should be. But I did not catch the thundering herd bug, the cooldown reset, or the interleaved error ambiguity. A separate model reviewing with fresh eyes did. That is the same reason I use a different model for review than for development -- familiarity blinds you to your own mistakes.

The replanning was also a lesson. Vague task specs produce vague implementations. When the plan has explicit file paths and verification commands, the developer agent builds exactly what was scoped. When the plan says "implement circuit breaker," you get something that works in the happy path but misses edge cases.

Conclusion

The circuit breaker is live. When Anthropic goes down, the router switches providers automatically. When it recovers, a half-open probe verifies before routing traffic back. And the whole thing is transparent -- the UI shows which provider served each request and whether failover was involved.