Reliability Patterns

Overview

Your agent works perfectly in development. It handles tool calls, reasons through complex tasks, and produces clean outputs. Then you deploy it to production, where it serves hundreds of users hitting real APIs, and everything falls apart. The LLM provider returns a 429 rate limit error. A downstream API times out. A tool call succeeds on the provider's side but the response never arrives. The agent retries blindly, hammering the already-stressed service until it collapses entirely.

The patterns in this lesson exist because distributed systems fail in ways that local development never reveals. In Module 6 Lesson 05, you learned fallback chains for routing requests to alternative resources when the primary one fails. In Module 6 Lesson 06, you learned idempotency for making retries safe by preventing duplicate side effects. This lesson builds on both foundations and introduces the full toolkit of reliability patterns that production agent systems need: retries with exponential backoff, circuit breakers, bulkheads, timeouts, graceful degradation, rate limiting, and dead letter queues.

These patterns are not theoretical. They come from decades of distributed systems engineering -- from the telecommunications switches of the 1980s to the microservices architectures of today. Agent systems inherit all the failure modes of distributed systems, plus new ones unique to LLM interactions: variable latency, token budget exhaustion, and non-deterministic outputs. The reliability patterns you learn here are the engineering that makes the difference between an agent that demos well and one that runs in production.

The Reliability Pattern Decision Tree

When something goes wrong in a production agent, the first question is not "how do I fix it?" but "which reliability pattern applies here?" The following decision tree maps failure types to the appropriate pattern.

flowchart TD
  FAIL["Failure Detected"] --> TYPE{"What type<br/>of failure?"}
  TYPE -->|"Transient error<br/>(timeout, 429, 503)"| RETRY["Retry with<br/>Exponential Backoff"]
  TYPE -->|"Repeated failures<br/>from same service"| CB["Circuit Breaker"]
  TYPE -->|"One slow service<br/>affecting others"| BH["Bulkhead<br/>Isolation"]
  TYPE -->|"Call taking<br/>too long"| TO["Timeout +<br/>Cancel"]
  TYPE -->|"Service fully<br/>unavailable"| GD["Graceful<br/>Degradation"]
  TYPE -->|"Too many requests<br/>hitting a service"| RL["Rate Limiting"]
  TYPE -->|"Message processing<br/>keeps failing"| DLQ["Dead Letter<br/>Queue"]

  RETRY -->|"Retries exhausted"| CB
  CB -->|"Breaker OPEN"| GD
  GD -->|"Fallback also fails"| DLQ

  style FAIL fill:#fed7d7,stroke:#fc8181
  style TYPE fill:#fefcbf,stroke:#d69e2e
  style RETRY fill:#ebf8ff,stroke:#3182ce
  style CB fill:#ebf8ff,stroke:#3182ce
  style BH fill:#ebf8ff,stroke:#3182ce
  style TO fill:#ebf8ff,stroke:#3182ce
  style GD fill:#e9d8fd,stroke:#b794f4
  style RL fill:#ebf8ff,stroke:#3182ce
  style DLQ fill:#fef0f0,stroke:#fc8181

Notice how the patterns chain together. A transient error triggers a retry. If retries keep failing, the circuit breaker opens. Once the breaker is open, the system falls back to graceful degradation. And if the fallback also fails, the message lands in a dead letter queue for later inspection. Each pattern handles a specific failure mode, and together they form a layered defense.

Retry with Exponential Backoff

The simplest reliability pattern is the retry -- if something fails, try again. But naive retries (retry immediately, retry forever) are dangerous. If an API returns a 429 because it is overloaded, retrying immediately adds more load. If multiple agent instances all retry at the same interval, they create a thundering herd -- a synchronized wave of requests that overwhelms the recovering service.

Exponential backoff solves this by increasing the wait time between each retry attempt: 1 second, 2 seconds, 4 seconds, 8 seconds. Each successive attempt waits longer, giving the service time to recover. Adding jitter -- a random offset to each wait time -- prevents multiple clients from synchronizing their retries.

A few design choices deserve attention. The retryable_exceptions parameter is critical -- you should only retry errors that are transient. A 400 Bad Request is not transient; retrying it with the same payload will produce the same error. A 429 Rate Limit or 503 Service Unavailable is transient; the same request may succeed after a delay. Retrying non-transient errors wastes time and obscures the real problem.

The max_delay cap prevents absurdly long waits. Without it, exponential backoff on the tenth retry would produce a delay of over 17 minutes. In practice, if you have waited 30-60 seconds and the service is still down, further waiting provides diminishing returns -- you should escalate to a fallback instead.

Connection to Module 6 Lesson 06: Retries are only safe when the action being retried is idempotent. If your agent is retrying a tool call that sends an email or charges a credit card, you must use idempotency keys (as covered in Module 6 Lesson 06) to prevent duplicate side effects. The retry decorator handles the when of retrying; idempotency handles the safety of retrying.

Circuit Breakers

Retries handle transient failures, but what happens when a service is down for minutes or hours? If every request triggers five retries with exponential backoff, your agent wastes time and resources on a service that is not coming back anytime soon. Worse, the accumulated retry traffic from many agents can prevent the service from recovering -- a phenomenon called a cascading failure.

The circuit breaker pattern solves this by tracking the health of each downstream service and short-circuiting requests when a service is unhealthy. The pattern borrows its name from electrical circuit breakers that trip when current exceeds safe levels, protecting the circuit from damage.

A circuit breaker has three states:

• CLOSED (normal operation): Requests pass through. The breaker tracks failures. If the failure count exceeds a threshold within a time window, the breaker trips to OPEN.
• OPEN (service is down): All requests are immediately rejected without attempting the call. This protects both your agent (no wasted time) and the failing service (no additional load). After a configurable timeout, the breaker transitions to HALF_OPEN.
• HALF_OPEN (testing recovery): The breaker allows a small number of probe requests through. If they succeed, the service has recovered and the breaker resets to CLOSED. If they fail, the service is still down and the breaker returns to OPEN.

sequenceDiagram
  participant A as Agent
  participant CB as Circuit Breaker
  participant S as External Service

  Note over CB: State: CLOSED

  A->>CB: Request 1
  CB->>S: Forward request
  S-->>CB: Success
  CB-->>A: Response

  A->>CB: Request 2
  CB->>S: Forward request
  S-->>CB: Failure (500)
  CB-->>A: Error (failures: 1/3)

  A->>CB: Request 3
  CB->>S: Forward request
  S-->>CB: Failure (timeout)
  CB-->>A: Error (failures: 2/3)

  A->>CB: Request 4
  CB->>S: Forward request
  S-->>CB: Failure (503)
  Note over CB: Threshold reached!<br/>State: CLOSED → OPEN
  CB-->>A: Error (breaker tripped)

  A->>CB: Request 5
  Note over CB: State: OPEN<br/>Reject immediately
  CB-->>A: CircuitOpenError<br/>(no call to service)

  A->>CB: Request 6
  CB-->>A: CircuitOpenError

  Note over CB: Recovery timeout elapsed<br/>State: OPEN → HALF_OPEN

  A->>CB: Request 7 (probe)
  CB->>S: Forward probe request
  S-->>CB: Success!
  Note over CB: Probe succeeded!<br/>State: HALF_OPEN → CLOSED
  CB-->>A: Response

  A->>CB: Request 8
  CB->>S: Forward request
  S-->>CB: Success
  CB-->>A: Response

The sequence diagram shows the full lifecycle. Requests 1 through 4 flow through normally, but three consecutive failures trip the breaker. Requests 5 and 6 are rejected instantly -- the agent gets a fast failure instead of waiting for a timeout. After the recovery period, request 7 acts as a probe. When it succeeds, the breaker resets and normal traffic resumes.

The implementation is thread-safe, which matters when your agent handles concurrent requests. The _lock prevents race conditions where two threads could both read the failure count as 2, both increment it, but only one trips the breaker. In production, you would also add metrics emission (total trips, time spent open, recovery success rate) to feed the monitoring systems covered in Module 12 Lesson 06.

Bulkheads

The bulkhead pattern borrows its name from the watertight compartments in ship hulls. If one compartment floods, the bulkheads prevent the water from spreading to other compartments. In software, a bulkhead isolates resources so that a failure in one area cannot exhaust the resources of another.

For an agent that calls multiple external services -- an LLM API, a search API, a database, a vector store -- a bulkhead ensures that a slow or failing service cannot consume all available resources and starve the others. Without bulkheads, a search API that starts responding slowly could tie up all your agent's threads, preventing it from making LLM calls that are working perfectly fine.

In practice, bulkheads take the form of separate resource pools per service: dedicated connection pools, thread pools, or semaphores that limit how many concurrent calls can go to each service.

The key insight is that bulkheads, circuit breakers, and retries compose naturally. A call flows through the bulkhead (capacity gate), then through the circuit breaker (health gate), and finally through the retry decorator (transient error handler). Each layer addresses a different failure mode.

Timeouts and Graceful Degradation

Timeouts are the simplest reliability pattern but also the most frequently neglected. Every external call should have a timeout. Without one, a hanging service can block your agent indefinitely. LLM calls are particularly susceptible -- a model generating a long response can take 30 seconds or more, and the difference between "still thinking" and "stuck" is impossible to detect from the client side.

Graceful degradation is what your agent does when a service is unavailable and all retries, circuit breakers, and fallbacks have been exhausted. Instead of crashing, the agent continues with reduced functionality. A search agent that cannot reach the search API might fall back to its training knowledge. A code review agent that cannot access the repository API might review only the diff that was provided in the prompt.

The connection to Module 6 Lesson 05 is direct: fallback chains are the mechanism for implementing graceful degradation. When the primary service is down, the fallback chain routes to the next alternative. The circuit breaker tells you when to degrade; the fallback chain tells you how.

Rate Limiting

Rate limiting is the proactive counterpart to retries. Instead of waiting for a 429 error and then backing off, you limit your own request rate to stay under the provider's limits. This is especially important for agent systems that can generate bursts of API calls -- a ReAct agent that decides to call a tool twenty times in rapid succession, or a multi-agent system where ten agents all call the same API simultaneously.

A simple token bucket rate limiter works well for most agent use cases: you have a bucket that holds a fixed number of tokens, tokens are consumed by each request, and tokens are replenished at a fixed rate. When the bucket is empty, requests wait until tokens are available.

Practical tip: Most LLM providers publish their rate limits. The Anthropic API, for example, has per-minute rate limits on both requests and tokens. Build your rate limiter around these published limits, with a safety margin of 80-90% of the stated limit. It is better to slightly under-utilize your quota than to trigger rate limit errors that degrade the experience.

Dead Letter Queues

When an agent processes messages from a queue -- user requests, task assignments, webhook events -- some messages will fail repeatedly despite retries. A dead letter queue (DLQ) captures these persistently failing messages instead of dropping them silently.

Without a DLQ, a message that fails three retries simply vanishes. You might not notice for hours or days that certain request types are silently failing. With a DLQ, failed messages accumulate in a separate queue where operators can inspect them, fix the underlying issue, and reprocess them.

In agent systems, DLQs are valuable for:

• Tool calls that fail due to unexpected input formats -- the DLQ preserves the exact input that caused the failure, making debugging straightforward.
• Multi-step tasks where one step fails permanently -- instead of losing the entire task, the DLQ captures the partial state so an operator or a different agent can complete it.
• Webhook processing where external events arrive in unexpected formats -- the DLQ prevents data loss while you update the parser.

Key takeaway: Dead letter queues transform silent failures into visible, actionable incidents. They are the difference between "we think the system is working" and "we know exactly which requests failed and why."

Composing Reliability Patterns

These patterns are most powerful when composed together. A production agent call flows through multiple layers, each handling a different failure mode. Here is how a well-protected external call looks end-to-end:

1. Rate limiter -- ensures you do not exceed the API's request quota.
2. Bulkhead -- ensures this service cannot consume all available capacity.
3. Circuit breaker -- fast-fails if the service is known to be down.
4. Timeout -- caps the maximum wait time for any single call.
5. Retry with backoff -- handles transient errors from the call itself.
6. Graceful degradation -- returns a reduced-quality result if all else fails.
7. Dead letter queue -- captures permanently failing requests for later inspection.

Each layer is independent and reusable. You can apply circuit breakers to your LLM calls and rate limiting to your search calls without coupling the patterns together. The composition happens at the call site, not inside the pattern implementations.

Summary

Reliability patterns are the engineering discipline that separates prototype agents from production systems. Each pattern addresses a specific failure mode, and together they form a layered defense against the inevitable failures of distributed systems.

• Retry with exponential backoff handles transient errors by spacing out retry attempts with increasing delays and random jitter to prevent thundering herds -- but only retry idempotent operations (see Module 6 Lesson 06)
• Circuit breakers track service health across three states (CLOSED, OPEN, HALF_OPEN) and short-circuit requests to failing services, protecting both the agent and the service from cascading failure
• Bulkheads isolate resources per service so that a slow or failing dependency cannot exhaust the agent's capacity and starve calls to healthy services
• Timeouts cap the maximum wait time for any external call, preventing indefinite hangs from blocking the agent
• Graceful degradation continues operation with reduced functionality when a service is unavailable, using the fallback chains from Module 6 Lesson 05 as the implementation mechanism
• Rate limiting proactively throttles outgoing requests to stay under provider quotas instead of reactively handling 429 errors
• Dead letter queues capture messages that fail processing repeatedly, preserving them for inspection and reprocessing instead of silently dropping them
• These patterns compose: a production call flows through rate limiting, then a bulkhead, then a circuit breaker, then a timeout, then retries -- each layer handling a different failure mode independently

In the next lesson, you will learn about Guardrails and Safety -- the patterns that protect not just your system from failures, but your users from harmful or incorrect agent outputs.