Design a circuit breaker around unreliable tools

When a dependency is failing, retries can turn a partial outage into a system-wide one. A circuit breaker stops calls after a measured failure condition and gives the dependency time to recover.

Define the measurable control

Define closed, open and half-open states. Choose a rolling window, failure classes, minimum traffic, open duration and probe limit. Keep business rejection separate from dependency failure. The fallback must be explicit: cached safe data, queued review, partial answer or refusal.

The AI reliability lab makes the decision reproducible with authored data:

from reliability_cases import circuit_breaker_case

result = circuit_breaker_case()
assert result["state"] == "open"
assert result["executed"] == 3
assert result["blocked"] == 2
assert result["automatic_write"] is False

Three consecutive failures open the fixture’s circuit. Two later attempts are blocked and the work goes to review; no automatic write occurs. A production implementation would use time-based windows and concurrency-safe state, but the state transition and fallback remain the testable contract.

Challenge the result

One global circuit can penalize healthy tenants or operations. Partition it by dependency and relevant scope without creating so many breakers that each lacks signal. In half-open state, permit a bounded probe and prevent a stampede.

Keep a reviewable evidence pack:

• state-transition specification
• failure classification and threshold rationale
• open and half-open tests
• fallback capacity and user message

This work aligns with the evaluation, security, cloud operations, reliability and FinOps sequence in the FDE for Professionals course. The linked course describes the learning pathway; this article’s numbers are synthetic and do not report a model, client, audit or production result.

Practice task

Inject five failures into a fake tool. Verify which calls execute, which are blocked and which event closes the circuit again.

Continue learning

This article is part of the Advanced AI reliability and assurance sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Measure retry amplification across an agent workflow.
• Continue with Create a model-change regression release process.

Reference: Azure Architecture Center: Circuit Breaker.