Fallback and Escalation Chains

Overview

In Module 3 Lesson 04, you learned how to handle tool failures with retries, backoff, and fallback tools -- when one tool fails, try an alternative that achieves the same goal through a different method. That pattern keeps your agent running when individual tools break.

But tool failures are not the only kind of problem. Sometimes the tool works fine, but the model is not capable enough. A fast, cheap model might misunderstand a nuanced query. A single model tier might produce a low-confidence answer on a complex question. And sometimes, no model is sufficient -- the decision requires human judgment, policy expertise, or access to information the agent does not have.

This lesson introduces fallback and escalation chains -- a pattern for cascading requests from fast and cheap resources to slow and powerful ones, and ultimately to human operators when automation is not enough. This is the design pattern that controls who handles what, and it is the key to building agents that are both cost-effective and reliable.

Model Cascading

The simplest form of escalation is model cascading: routing a request through progressively more capable (and more expensive) LLMs until one produces a satisfactory result.

The idea is straightforward. Most requests are easy. A small, fast model like Claude Haiku handles them in milliseconds at minimal cost. But some requests are harder -- they need the reasoning depth of Claude Sonnet. And a few requests are genuinely complex, requiring the full capability of Claude Opus. Instead of always using the most powerful model (expensive, slow) or always using the cheapest model (fast but sometimes wrong), you cascade through them.

flowchart TD
  REQ[Incoming Request] --> T1{Tier 1: Haiku}
  T1 -->|confidence >= 0.85| DONE1[Return Response]
  T1 -->|confidence < 0.85| T2{Tier 2: Sonnet}
  T2 -->|confidence >= 0.80| DONE2[Return Response]
  T2 -->|confidence < 0.80| T3{Tier 3: Opus}
  T3 -->|confidence >= 0.70| DONE3[Return Response]
  T3 -->|confidence < 0.70| HUMAN[Escalate to Human]
  style REQ fill:#ebf8ff,stroke:#3182ce
  style T1 fill:#f0fff4,stroke:#68d391
  style T2 fill:#fefcbf,stroke:#d69e2e
  style T3 fill:#fed7d7,stroke:#fc8181
  style HUMAN fill:#e9d8fd,stroke:#b794f4
  style DONE1 fill:#f0fff4,stroke:#68d391
  style DONE2 fill:#fefcbf,stroke:#d69e2e
  style DONE3 fill:#fed7d7,stroke:#fc8181

There are two critical design decisions in any cascade: what triggers escalation, and how you measure confidence.

Triggering escalation can be based on several signals. The model can self-report confidence (by asking it to rate its own certainty). You can use output validation -- if the response fails a format check or contains hedging language ("I'm not sure", "it might be"), escalate. You can use task complexity heuristics -- long queries, multi-step reasoning, or domain-specific jargon might skip directly to a higher tier. In practice, combining these signals works better than relying on any single one.

Measuring confidence is an active area of research. One practical approach is to ask the model to return a confidence score alongside its answer. Another is to run a lightweight classifier on the response to detect uncertainty markers. The key insight is that the confidence threshold should decrease as you move up the cascade -- you demand high confidence from the cheap model (only accept its answer if it is very sure) but accept lower confidence from the expensive model (because there is nowhere better to escalate to, except a human).

Confidence-Based Routing in Practice

Here is a complete implementation of a model cascade with confidence-based routing. The cascade tries each tier in order, extracts a confidence score from the model's response, and escalates if confidence falls below the tier's threshold.

A few things to notice in this implementation:

• The confidence threshold decreases at each tier (0.85, 0.80, 0.70). You are strict with the cheap model and lenient with the expensive one.
• Every tier result is recorded in tiers_tried, creating an audit trail. When you get to Module 12's monitoring and cost lessons, this trail becomes invaluable for understanding where your budget goes.
• When JSON parsing fails, confidence defaults to 0.0 -- this is a safe failure mode that forces escalation rather than silently accepting a bad response.
• The resolved_by field tells the caller which tier handled the request. This is the metric you will track in production to tune your thresholds.

Tool Fallbacks vs. Model Escalation

It is worth pausing to clarify how this pattern relates to the tool fallback pattern from Module 3 Lesson 04.

Tool fallbacks (Module 3) handle the case where a tool breaks. The search API returns a 500 error, so you try a different search API. The primary database is down, so you query the read replica. The tool's function stays the same -- you are just finding a different way to achieve it.

Model escalation (this lesson) handles the case where the model is not capable enough. The tool works fine, but the model's answer is not good enough. You are not switching tools -- you are switching the brain that reasons about the results.

In a well-designed agent, both patterns coexist. A request might first cascade through models (escalation), and within each model tier, a failed tool call might trigger a tool fallback. They operate at different layers of the system.

The Escalation Sequence

Let's trace a concrete scenario to see how escalation works end-to-end. A customer support agent receives a complex billing dispute. Watch how the request flows through the cascade.

sequenceDiagram
  participant U as User
  participant R as Router
  participant H as Tier 1: Haiku
  participant S as Tier 2: Sonnet
  participant O as Tier 3: Opus
  participant HU as Human Agent

  U->>R: "My invoice has duplicate charges<br/>from two different billing cycles"
  R->>H: Route to cheapest tier
  H-->>R: Response (confidence: 0.45)
  Note right of H: Below 0.85 threshold<br/>Complex billing query
  R->>S: Escalate to Tier 2
  S-->>R: Response (confidence: 0.72)
  Note right of S: Below 0.80 threshold<br/>Needs policy knowledge
  R->>O: Escalate to Tier 3
  O-->>R: Response (confidence: 0.68)
  Note right of O: Below 0.70 threshold<br/>Requires account access
  R->>HU: Escalate to human
  Note right of HU: Human reviews all 3<br/>automated responses
  HU-->>U: Final resolution with<br/>manual account adjustment

This sequence reveals something important: when the request finally reaches a human, the human does not start from scratch. They have three automated responses to review, each with a confidence score. The Opus response (68% confidence) is probably mostly correct -- the human just needs to verify and authorize the account adjustment. The cascade has not just escalated the request; it has pre-processed it, saving the human significant effort.

Human-in-the-Loop Escalation

The final tier of any escalation chain is a human. Designing this handoff well is the difference between a system that frustrates operators and one that makes them more productive.

When to escalate to a human:

• All automated tiers have been tried and confidence remains below threshold
• The request involves high-stakes decisions -- financial transactions, account deletions, legal implications
• The agent detects ambiguity it cannot resolve -- contradictory instructions, missing context, policy edge cases
• Safety guardrails are triggered -- the request touches sensitive topics or the response might cause harm

What to include in the escalation:

• The original user request
• Every automated response with its confidence score
• The specific reason for escalation (low confidence, safety trigger, policy gap)
• Relevant context the agent has gathered (tool results, conversation history)

What to avoid:

• Do not make the human re-do work the agent already completed. If Opus produced a 68% confidence answer, show it to the human as a starting point.
• Do not escalate silently. Tell the user their request is being reviewed by a human and give an estimated response time.
• Do not treat escalation as failure. Knowing when not to answer autonomously is a feature of a well-designed agent, not a deficiency.

Notice the HIGH_STAKES check -- even when a model answers with high confidence, certain topics should always get human review. A model might be 95% confident about how to process a refund, but company policy might require human authorization for all refunds above a certain amount. Confidence-based routing and policy-based routing work together.

Designing Effective Escalation Chains

Building a cascade is straightforward. Building a good cascade takes thought. Here are the design principles that separate effective chains from ones that waste money or frustrate users.

Principle 1: Match tiers to task difficulty distribution. Before choosing thresholds, analyze your request distribution. If 80% of requests are simple lookups, a cheap model with a high threshold will handle most traffic at minimal cost. If 50% of requests are complex reasoning tasks, starting with a cheap model wastes latency -- you are better off starting at a middle tier or using a complexity classifier to skip tiers.

Principle 2: Make thresholds tunable, not hardcoded. Your initial thresholds will be wrong. In production, you will tune them based on actual resolution rates. Store thresholds in configuration, not in code. Track the distribution of which tier resolves each request and adjust. Module 12 Lesson 06 covers the monitoring infrastructure you need for this.

Principle 3: Add circuit breakers at each tier. If Haiku starts returning errors (API outage, rate limit), skip it and go directly to Sonnet. Do not waste time and tokens waiting for a tier that is down. This connects directly to the reliability patterns you will see in Module 12 Lesson 04.

Principle 4: Preserve context across tiers. When escalating from Haiku to Sonnet, pass the original request -- not Haiku's response. Each tier should reason from the source material, not from a potentially flawed lower-tier answer. However, do pass the lower-tier confidence score as metadata so the higher tier understands why it was called.

Key takeaway: The goal of an escalation chain is not to try everything. It is to find the cheapest resource that can handle the request well enough, and to know when "well enough" requires a human.

Connecting the Patterns

Fallback and escalation chains do not exist in isolation. They work with the other patterns from this module:

• Tool Registry (Lesson 02): The cascade itself can be registered as a "meta-tool" in the registry, so agents invoke it like any other capability.
• Middleware and Hooks (Lesson 03): Middleware can log every escalation, track tier resolution rates, and enforce rate limits per tier.
• Context Management (Lesson 04): When escalating, you may need to summarize the conversation so far to fit the higher-tier model's context window -- especially if earlier tiers consumed tokens on failed attempts.
• Checkpointing (next lesson): If an escalation to a human takes hours or days, checkpointing ensures the agent can resume the conversation when the human responds, without losing state.

And looking ahead to Module 12: the cascade's tiers_tried audit trail feeds directly into the cost monitoring and optimization infrastructure you will build there. Every escalation is a data point that helps you tune thresholds, identify patterns in what gets escalated, and measure the true cost per resolution.

Summary

Fallback and escalation chains give your agent the ability to match the right resource to each request -- cheap and fast for easy tasks, powerful and expensive for hard ones, and human when automation is not enough.

• Model cascading routes requests through progressively more capable models (Haiku, Sonnet, Opus), stopping at the first tier that meets its confidence threshold
• Confidence-based routing uses self-reported confidence scores (or output validation) to decide whether to accept an answer or escalate
• Confidence thresholds decrease as you move up the cascade -- strict with cheap models, lenient with expensive ones
• Tool fallbacks (Module 3 Lesson 04) and model escalation (this lesson) operate at different layers: tool fallbacks swap implementations, model escalation swaps reasoning capability
• Human-in-the-loop is the final escalation tier -- it should receive pre-processed context, not raw requests, so humans augment the agent rather than replace it
• Design cascades with tunable thresholds, circuit breakers, and audit trails to enable the production monitoring and cost optimization covered in Module 12

In the next lesson, you will learn Idempotency and Checkpointing -- how to make agent actions safe to retry and how to save state so that long-running workflows (including those waiting on human escalation) can survive crashes and resume exactly where they left off.