LLM Agents Academy

Overview

In the previous lessons, you learned how to organize tools with a registry and intercept actions with middleware. Both patterns assume that the LLM sees the information it needs. But what happens when there is too much information to fit?

Every LLM has a context window -- a fixed limit on how many tokens it can process in a single request. Think of it as the agent's working memory. Everything the agent knows right now must fit inside this window: the system prompt that defines its behavior, the conversation history that tracks what has happened, the tool results it needs to reason about, and enough remaining space for it to generate a response.

For a simple chatbot that answers one question and stops, this is rarely a problem. But agents are not simple chatbots. A ReAct agent that searches the web, reads documents, and iterates through ten reasoning steps can easily accumulate hundreds of thousands of tokens of conversation history. A Plan-and-Execute agent that calls a code analysis tool might receive a single tool result that consumes half the context window. A customer support agent that handles a complex case over fifty messages will eventually run out of room.

When the context window fills up, bad things happen. The API call fails with a token limit error. Or worse, you naively truncate the conversation and the agent forgets critical instructions or earlier decisions, leading to contradictory behavior and loops.

Context window management is the design pattern that prevents this. It treats the context window as a scarce resource, allocates budgets to different content categories, and applies strategies to keep the most relevant information visible to the LLM while gracefully evicting or compressing the rest.

Anatomy of the Context Window

Before you can manage the context window, you need to understand what fills it. Every LLM request is composed of distinct content categories, each competing for the same fixed pool of tokens.

graph TD
  subgraph ContextWindow["Context Window (e.g., 128k tokens)"]
      direction TB
      SYS["System Prompt<br/>─────────────────<br/>Agent instructions, persona,<br/>tool definitions, guardrails<br/>~2k-8k tokens (fixed)"]
      HIST["Conversation History<br/>─────────────────<br/>User messages, assistant responses,<br/>previous reasoning steps<br/>~grows unbounded"]
      TOOL["Tool Results<br/>─────────────────<br/>Search results, API responses,<br/>file contents, database rows<br/>~varies wildly per call"]
      GEN["Generation Budget<br/>─────────────────<br/>Space reserved for the LLM<br/>to produce its response<br/>~1k-4k tokens (reserved)"]
  end
  SYS --> HIST
  HIST --> TOOL
  TOOL --> GEN
  style ContextWindow fill:#f7fafc,stroke:#2d3748,stroke-width:2px
  style SYS fill:#fed7d7,stroke:#fc8181
  style HIST fill:#fefcbf,stroke:#d69e2e
  style TOOL fill:#bee3f8,stroke:#3182ce
  style GEN fill:#c6f6d5,stroke:#38a169

Each category has different characteristics:

System prompt is essentially fixed for the lifetime of the agent. It defines who the agent is, what tools it has, and what rules it follows. You set it once and it occupies the same space every turn. This is your floor -- the minimum context you always need.
Conversation history grows with every turn. Each user message, assistant response, and reasoning step adds tokens. In a ReAct agent, a single turn might add hundreds of tokens (thought + action + observation). Over dozens of turns, history can consume the entire window.
Tool results are the wildcard. A web search might return 500 tokens. A code file might return 15,000. A database query might return 50,000. You often cannot predict the size until after the tool runs.
Generation budget is the space you must reserve for the LLM's response. If you fill the context window completely with input tokens, the model has no room to generate output. You need to reserve at least 1,000-4,000 tokens depending on the expected response length.

The fundamental tension is this: the system prompt and generation budget are relatively fixed, but history and tool results grow without bound. Something has to give. The question is what gives and how.

The Context Budget

The first principle of context management is making the implicit budget explicit. Instead of hoping everything fits and crashing when it does not, you define upfront how tokens are allocated.

graph LR
  TOTAL["Total Window<br/>128,000 tokens"] --> ALLOC["Budget Allocator"]
  ALLOC --> S["System<br/>8,000<br/>(fixed)"]
  ALLOC --> G["Generation<br/>4,000<br/>(reserved)"]
  ALLOC --> FLEX["Flexible Pool<br/>116,000"]
  FLEX --> H["History<br/>up to 80,000"]
  FLEX --> T["Tool Results<br/>up to 36,000"]
  style TOTAL fill:#e2e8f0,stroke:#4a5568,stroke-width:2px
  style ALLOC fill:#ebf8ff,stroke:#3182ce
  style S fill:#fed7d7,stroke:#fc8181
  style G fill:#c6f6d5,stroke:#38a169
  style FLEX fill:#fefcbf,stroke:#d69e2e
  style H fill:#fefcbf,stroke:#ecc94b
  style T fill:#bee3f8,stroke:#63b3ed

A token budget defines hard limits and soft limits for each category:

Hard limits cannot be exceeded. The system prompt always stays. The generation budget is always reserved. These are subtracted first.
Soft limits represent the maximum a category can use when there is room. History might be allowed up to 80,000 tokens, but if a large tool result comes in, history gets compressed to make space.
The flexible pool is what remains after hard limits are subtracted. History and tool results share this pool, with a priority system determining who yields when space is tight.

This budgeting approach transforms context management from a reactive "oh no, we exceeded the limit" crash into a proactive allocation problem.

Context Management Strategies

When the flexible pool runs out of space, you need a strategy to reclaim tokens. There are four main strategies, each with different trade-offs. In practice, production agents combine multiple strategies.

Sliding Window

The simplest strategy. Keep the most recent N messages and drop everything older.

How it works: Set a maximum message count or token count for history. When a new message arrives and the limit is exceeded, remove the oldest message. The system prompt is never removed -- only conversation history slides.

Strengths: Simple to implement. Predictable memory usage. Works well for short conversations where recent context is all that matters.

Weaknesses: Catastrophic information loss. If the user stated a critical requirement in message 3 and the window only holds the last 20 messages, that requirement disappears at message 23. The agent acts as if it was never mentioned. Worse, it does not know it lost that information -- it simply behaves inconsistently.

When to use: Prototypes, simple chatbots, or as a fallback inside a more sophisticated strategy. Never as the sole strategy for agents that run for many turns.

Summarization

Replace older messages with a compressed summary that preserves the key information in fewer tokens.

How it works: When history exceeds a threshold, take the oldest block of messages (say, the first 20 turns) and pass them to an LLM with a prompt like "Summarize the key decisions, facts, and open questions from this conversation." Replace the 20 messages with the summary, which typically uses 10-20% of the original tokens.

Strengths: Preserves semantic content. Critical facts survive even as the raw messages are evicted. The agent "remembers" what happened, just not the exact wording.

Weaknesses: Lossy -- subtle details can be dropped. Adds latency and cost (you are making an extra LLM call). Summaries can be inaccurate. Recursive summarization (summarizing summaries) compounds errors.

When to use: Long-running agents where conversation history contains important decisions, requirements, or facts that must persist across many turns.

Priority-Based Retention

Assign importance scores to messages and evict the least important ones first.

How it works: Each message gets a priority score based on its role and content. System messages get the highest priority (never evicted). Messages containing tool results from the current task get high priority. User messages with explicit instructions get medium-high priority. Assistant reasoning steps that led to completed sub-tasks get low priority -- the outcome matters, not the deliberation.

Strengths: Intelligent eviction. The most relevant information survives regardless of when it appeared in the conversation. You can encode domain-specific importance rules.

Weaknesses: Requires careful priority tuning. Getting the heuristics wrong means evicting important messages. More complex to implement than a simple window.

When to use: Multi-step agents where some turns are clearly more important than others. Particularly effective when combined with summarization -- summarize low-priority blocks, keep high-priority messages verbatim.

Chunking and Retrieval

Instead of keeping everything in the context window, store messages externally and retrieve only what is relevant to the current step.

How it works: As the conversation grows, older messages are moved to an external store (a vector database, a simple list, or even a file). Before each LLM call, a retrieval step identifies which stored messages are relevant to the current query and injects them back into the context. This is essentially RAG applied to the agent's own conversation history.

Strengths: Scales to arbitrarily long conversations. Only relevant information enters the context, regardless of when it appeared. Memory-efficient.

Weaknesses: Retrieval can miss relevant information (recall is imperfect). Adds significant infrastructure complexity. Latency increases with retrieval steps. The agent may behave oddly if retrieval returns messages out of chronological order without proper framing.

When to use: Agents that run for hundreds or thousands of turns. Agents that need to recall specific facts from much earlier in the conversation. Often used alongside summarization -- store detailed messages externally, keep summaries in-context, and retrieve originals on demand.

Strategy Decision Tree

Choosing the right combination of strategies depends on your agent's characteristics. Use this decision tree as a starting point.

flowchart TD
  START["How long does your<br/>agent conversation run?"] -->|"< 20 turns"| SHORT["Short conversation"]
  START -->|"20-100 turns"| MEDIUM["Medium conversation"]
  START -->|"> 100 turns"| LONG["Long conversation"]
  SHORT --> SW["Sliding Window<br/>is sufficient"]
  MEDIUM --> Q1{"Do early messages<br/>contain critical context?"}
  Q1 -->|"No"| SW2["Sliding Window<br/>+ token budget"]
  Q1 -->|"Yes"| Q2{"Can you define<br/>priority rules?"}
  Q2 -->|"Yes"| PRI["Priority-Based Retention<br/>+ Summarization"]
  Q2 -->|"No"| SUM["Periodic Summarization<br/>+ Sliding Window"]
  LONG --> Q3{"Do you need to recall<br/>specific past details?"}
  Q3 -->|"No"| SUM2["Rolling Summarization<br/>+ Priority Retention"]
  Q3 -->|"Yes"| CHUNK["Chunking + Retrieval<br/>+ Summarization"]
  style START fill:#ebf8ff,stroke:#3182ce,stroke-width:2px
  style SHORT fill:#c6f6d5,stroke:#38a169
  style MEDIUM fill:#fefcbf,stroke:#d69e2e
  style LONG fill:#fed7d7,stroke:#fc8181
  style SW fill:#f0fff4,stroke:#68d391
  style SW2 fill:#f0fff4,stroke:#68d391
  style PRI fill:#fef3c7,stroke:#d97706
  style SUM fill:#fef3c7,stroke:#d97706
  style SUM2 fill:#fff5f5,stroke:#fc8181
  style CHUNK fill:#fff5f5,stroke:#fc8181
  style Q1 fill:#f7fafc,stroke:#4a5568
  style Q2 fill:#f7fafc,stroke:#4a5568
  style Q3 fill:#f7fafc,stroke:#4a5568

Most production agents land in the middle column: conversations long enough that naive truncation loses information, but not so long that full retrieval infrastructure is justified. Priority-based retention combined with periodic summarization is the sweet spot for the majority of use cases.

Building a Context Manager

Let's put these concepts into code. The following implementation provides a ContextManager class that tracks token usage, enforces budgets, and applies strategies automatically.

context_manager.py

Let's walk through the key design decisions:

Token counting uses tiktoken, the same tokenizer the model uses. Estimating token counts with character-based heuristics (like "4 characters per token") leads to budget overruns. Always count exactly.
Priority is assigned by default based on role, but callers can override it. This lets the agent mark specific messages as critical ("the user just gave a new requirement") or disposable ("this was a verbose debug dump").
Eviction is priority-first, age-second. A disposable message from turn 50 is evicted before a medium-priority message from turn 5. Within the same priority, older messages go first.
Truncation is the last resort. If evicting all evictable messages still does not free enough space, the incoming message itself is truncated. This preserves the beginning of tool results (which usually contain the most relevant information) and appends a marker so the agent knows the content was cut.

Using the Context Manager

Here is how you wire the context manager into an agent loop.

agent_with_context.py

Notice the priority assignments: the agent's intermediate reasoning steps are marked LOW because once a step leads to a tool call and its result, the reasoning that led to that decision is less important than the result itself. The tool results are HIGH because they contain the facts the agent needs to reason about now. If the context fills up, old reasoning steps are evicted first, preserving the factual tool outputs.

Adding Summarization

The basic context manager evicts messages entirely. A more sophisticated approach summarizes them before eviction, preserving their semantic content in fewer tokens.

summarizing_context.py

Two important details in this implementation:

Summaries are marked is_summarizable=False to prevent recursive summarization. Summarizing a summary compounds information loss and can produce nonsensical results after a few rounds.
Summarization triggers proactively at 75% utilization, not reactively when the window is full. This avoids the latency spike of summarizing a large block right when the agent needs to respond quickly.

Token Counting: Getting It Right

Token counting is the foundation that everything else depends on. If your counts are wrong, your budgets are wrong, and your eviction strategies fire too early or too late.

token_counting.py

A common mistake is counting only the content tokens and ignoring the per-message overhead -- the tokens the model uses for role markers, separators, and message boundaries. For a conversation with 100 messages, this overhead alone can be 400+ tokens. Ignore it and your budget calculations silently drift.

Common Pitfalls

Pitfall 1: Evicting the system prompt. Never. The system prompt defines the agent's behavior. Evicting it turns your carefully designed agent into a generic chatbot. Always mark it as CRITICAL.

Pitfall 2: Summarizing too aggressively. If you summarize after every 5 messages, you spend more tokens (and latency) on summarization calls than you save. Summarize in batches -- 10-20 messages at a time -- and only when utilization crosses a threshold.

Pitfall 3: Ignoring tool result size. A single tool call can return tens of thousands of tokens. If your context manager does not account for tool results, a large API response can blow through your budget in one turn. Always check tool result size before inserting it and truncate if necessary.

Pitfall 4: Not reserving generation space. If you fill the context window to 99% with input tokens, the model has almost no room to respond. It may produce truncated or incoherent output. Always reserve a generation budget and treat it as a hard limit.

Connection to Memory Systems

Context window management handles the agent's short-term working memory -- what is visible in the current LLM call. But what about information that was evicted? What about facts the agent learned in a previous conversation entirely?

This is where memory systems come in, which you will explore in depth in Module 7. Memory systems provide long-term storage that persists beyond the context window and across conversations. The context manager and memory system work together:

The context manager decides what stays in the window and what gets evicted
The memory system stores what was evicted so it can be retrieved later
At the start of each turn, the context manager can query the memory system to pull in relevant information from past conversations

Think of it like human cognition: your context window is your conscious attention (limited, focused, fast), while your memory system is your long-term memory (vast, associative, requires recall). A well-designed agent uses both.

Key insight: Context management and memory are two sides of the same coin. Context management controls what the agent is thinking about right now. Memory controls what the agent can think about when prompted. Neither is sufficient alone.

Summary

The context window is the agent's working memory -- a fixed-size budget that must hold everything the agent needs to reason about. Managing it well is the difference between agents that handle complex, multi-step tasks and agents that crash or forget critical information midway through.

The context window is divided into four categories: system prompt (fixed), conversation history (grows), tool results (variable), and generation budget (reserved)
A token budget makes allocation explicit, giving each category a guaranteed share and defining a flexible pool for history and tool results
Sliding window is simple but loses information indiscriminately -- suitable only for short conversations or as a fallback
Summarization compresses older messages, preserving semantic content at 10-20% of the original token cost
Priority-based retention assigns importance scores and evicts the least important messages first, keeping critical context regardless of age
Chunking and retrieval scales to very long conversations by storing messages externally and retrieving relevant ones on demand
Most production agents combine priority-based retention with periodic summarization for the best balance of simplicity and information preservation
Always count tokens exactly using the model's tokenizer, including per-message overhead
Always reserve generation space -- a full context window with no room for output produces truncated or broken responses

In the next lesson, we explore Fallback and Escalation Chains -- what happens when a tool call fails, a model cannot handle a query, or the agent reaches the limits of what automation can solve. You will learn to build cascading strategies that degrade gracefully instead of failing catastrophically.

Context Window Management