LLM Agents Academy
Module 13: Advanced Patterns & Real-World Applications

Advanced Applications Lab

Design and build a production-grade multi-agent system integrating advanced patterns

~15 min readIncludes code

Overview

This lesson brings together the advanced patterns from this module into a single integrative exercise. Over the past fourteen modules and one hundred one lessons, you built your understanding from the ground up -- starting with what an LLM agent is, progressing through prompting, tool use, architectures, design patterns, memory, frameworks, multi-modal capabilities, multi-agent systems, evaluation, production deployment, and enterprise applications. Each module introduced concepts in isolation so you could master them one at a time. But real-world agent systems do not use one concept at a time. They combine all of them into a single, cohesive system where prompting strategies, tool orchestration, memory, guardrails, monitoring, and multi-agent coordination work together.

This lab is your opportunity to put the advanced patterns together. You will design and build a Research & Report System -- a production-grade multi-agent application that accepts a research question, dispatches specialized agents to gather and verify information, synthesizes the findings into a structured report, and does all of this behind guardrails and monitoring. Every module you have completed contributes a specific piece to this system. By the end, you will have a skeleton you can extend into a real product. Module 14 extends these patterns further into enterprise contexts with governance, compliance, and domain-specific deployment concerns.

The Complete Capstone Architecture

Before writing any code, study the full system architecture. This diagram shows every component, how they connect, and which module taught you the underlying concept.

graph TD
  User["User Query"] --> API["FastAPI Gateway"]
  subgraph ProductionLayer["Production Layer (Module 12)"]
      API --> InputG["Input Guardrails"]
      InputG --> Monitor["Monitoring Middleware"]
  end
  Monitor --> Supervisor["Supervisor Agent (Module 10)"]
  subgraph MultiAgent["Multi-Agent Core (Modules 4, 5, 9)"]
      Supervisor -->|"plan"| Planner["Task Planner"]
      Supervisor -->|"delegate"| Research["Research Agent"]
      Supervisor -->|"delegate"| Writer["Writer Agent"]
      Supervisor -->|"delegate"| FactChecker["Fact-Checker Agent"]
      Research -->|"retrieve"| RAG["RAG Pipeline (Module 7)"]
      RAG --> VectorDB["Vector Store"]
      RAG --> WebSearch["Web Search Tool (Module 3)"]
      FactChecker -->|"debate"| Research
      Writer -->|"draft"| Report["Report Draft"]
      FactChecker -->|"verify"| Report
  end
  subgraph MemoryLayer["Memory Layer (Module 7)"]
      Supervisor --> ConvMem["Conversation Memory"]
      Research --> LongMem["Long-Term Memory"]
  end
  subgraph EvalLayer["Evaluation Layer (Module 11)"]
      Report --> Judge["LLM-as-Judge"]
      Judge -->|"score < threshold"| Supervisor
      Judge -->|"score >= threshold"| OutputG["Output Guardrails"]
  end
  OutputG --> Response["Final Report"]
  subgraph Observability["Observability (Module 12)"]
      Monitor -->|"metrics"| Prometheus["Prometheus"]
      Monitor -->|"traces"| Traces["Trace Store"]
      Prometheus --> Grafana["Grafana"]
  end
  style ProductionLayer fill:#fed7d7,stroke:#c53030
  style MultiAgent fill:#ebf8ff,stroke:#3182ce
  style MemoryLayer fill:#fefcbf,stroke:#b7791f
  style EvalLayer fill:#c6f6d5,stroke:#38a169
  style Observability fill:#e9d8fd,stroke:#805ad5
  style Supervisor fill:#bee3f8,stroke:#2b6cb0
  style Research fill:#bee3f8,stroke:#2b6cb0
  style Writer fill:#bee3f8,stroke:#2b6cb0
  style FactChecker fill:#bee3f8,stroke:#2b6cb0

The system has five layers, each color-coded to the module that taught it. The Production Layer (red) handles input validation, guardrails, and monitoring before any agent logic runs. The Multi-Agent Core (blue) contains the supervisor and its three specialized agents. The Memory Layer (yellow) gives agents access to conversation history and long-term knowledge. The Evaluation Layer (green) uses LLM-as-Judge to score the final report before it reaches the user. The Observability Layer (purple) captures metrics and traces across the entire pipeline.

This is not a toy architecture. Production research systems at companies like Perplexity, You.com, and Elicit use variations of exactly this pattern: a coordinator that dispatches specialized sub-agents, each with access to retrieval and memory, with quality checks before the response ships.

What Every Module Contributes

Before building, let us map every module to its role in the capstone. This is the curriculum you have completed, and every piece has a purpose here.

| Module | Topic | Role in the Capstone | |--------|-------|---------------------| | 1 | Foundations of LLM Agents | The agent loop (perceive, reason, act) that every agent in the system follows | | 2 | Prompting & Reasoning | System prompts for each specialized agent; chain-of-thought for the planner | | 3 | Tool Use & Function Calling | Web search, document retrieval, and citation extraction tools | | 4 | Agent Architectures | ReAct loop for the research agent; plan-and-execute for the supervisor | | 5 | Agent Design Patterns | Routing pattern for task delegation; reflection for iterative improvement | | 6 | Memory & Knowledge | RAG pipeline for grounding research; conversation memory for multi-turn | | 7 | Agent Frameworks & SDKs | Framework choice for implementation (Anthropic SDK, LangGraph, or CrewAI) | | 8 | Multi-Modal Agents | Chart and image analysis if the research involves visual data | | 9 | Multi-Agent Systems | Supervisor orchestration, inter-agent communication, debate protocol | | 10 | Evaluation & Testing | LLM-as-Judge scoring, evaluation harness, regression testing | | 11 | Production & Safety | Guardrails, circuit breakers, monitoring, containerized deployment | | 12 | Advanced Patterns | Research agent patterns, domain-specific tuning, alignment considerations |

Every row is load-bearing. If you remove any module, the capstone loses a critical capability -- and that is the point. A production agent system is the sum of all these disciplines working together.

Step 1: Define the Agent Contracts

Start by defining the data structures that flow between agents. In Module 10, you learned that multi-agent systems need clear contracts -- each agent must know exactly what it receives and what it returns. These shared types are the backbone of the system.

contracts.py

These dataclasses are deliberately simple -- no framework-specific types, no LLM client imports, no business logic. They are pure data. This is a design pattern from Module 6: separate the data contracts from the agent logic so that agents can be tested, replaced, and composed independently. The ResearchTask flows from the supervisor to a worker agent. The FactCheckResult flows from the fact-checker back to the supervisor. The FinalReport is the system's output.

Step 2: Build the Supervisor Agent

The supervisor agent is the orchestrator from Module 10. It receives the user's question, decomposes it into sub-tasks using chain-of-thought reasoning (Module 2), and delegates those tasks to specialized agents. It does not do the research itself -- it coordinates.

supervisor.py

Notice the phased execution pattern. The supervisor does not run all tasks in parallel and hope for the best. It runs research first (because the writer needs research results), then writing (because the fact-checker needs a draft), then fact-checking (because revision needs fact-check results), and finally revision. This is the plan-and-execute architecture from Module 4 applied to a multi-agent workflow. Each phase depends on the output of the previous phase, and the supervisor manages those dependencies explicitly.

The supervisor also implements the iterative refinement pattern from Module 6: if the fact-checker finds unverified claims, the writer revises the report. This feedback loop is what separates a naive pipeline from a quality-focused system.

Step 3: Build the Research Agent with RAG

The research agent is where Module 3 (tool use) meets Module 7 (RAG). It takes a research sub-task, searches for relevant information using both a vector store and web search, and synthesizes the findings into a structured result.

research_agent.py

This is the agentic RAG pattern from Module 7, Lesson 06. The research agent does not just retrieve documents and stuff them into a prompt. It uses the ReAct loop from Module 4 -- it reasons about what to search, executes the search, examines the results, and decides whether it needs to search again. The max_iterations safety limit prevents runaway loops, a production concern from Module 12. The confidence score is lower when no sources are found, which the fact-checker will use later to decide what needs verification.

Step 4: Build the Fact-Checker Agent

The fact-checker agent implements the adversarial debate pattern from Module 10, Lesson 06. It takes the writer's draft, extracts key claims, and challenges each one against the research evidence. This is the red team to the writer's blue team.

fact_checker.py

The fact-checker uses a deliberate adversarial stance -- its prompt says "your job is to find problems." This is not a bug; it is the debate pattern from Module 10. A cooperative verifier tends to confirm whatever it is given (the sycophancy problem from Module 13, Lesson 05 on alignment). An adversarial verifier actively looks for weaknesses. The supervisor uses the fact-checker's output to decide whether the report needs revision, closing the feedback loop.

Step 5: Build the Writer Agent

The writer agent takes raw research results and transforms them into a coherent, structured report. It also handles revisions when the fact-checker finds issues.

writer_agent.py

The writer agent uses two distinct prompts -- one for initial drafting and one for revision. The revision prompt is deliberately constrained: "Do NOT invent new information." This prevents the writer from hallucinating fixes to fact-checking failures, a critical guardrail at the agent level (Module 12) that complements the system-level guardrails at the API boundary.

Step 6: Add Quality Evaluation

Before the report reaches the user, the LLM-as-Judge pattern from Module 11 scores it. If the score is below a threshold, the supervisor loops back for another round of revision.

evaluator.py

The evaluator uses a multi-dimensional rubric, not a single score. This is the calibrated evaluation approach from Module 11, Lesson 04. Scoring on five dimensions -- completeness, accuracy, clarity, source quality, and objectivity -- gives the supervisor actionable feedback. If accuracy is low but clarity is high, the system knows to re-research rather than re-write. The quality_threshold of 7.0 is configurable -- you might set it higher for medical or legal research where accuracy matters more, or lower for exploratory brainstorming where speed matters more.

Step 7: Wire Everything Together with Guardrails and Monitoring

The final step assembles all the components behind the production layer from Module 12. This is the orchestration glue that turns individual agents into a system.

app.py

Study the three phases in the /research endpoint. Phase 1 (planning) decomposes the question. Phase 2 (execution) runs research, writing, and fact-checking through the supervisor. Phase 3 (evaluation) loops until the quality threshold is met or revision rounds are exhausted. The guardrails bracket the entire pipeline -- input guardrails before Phase 1, output guardrails after Phase 3. The monitoring (not shown here to keep the code focused, but identical to Module 12 Lesson 07) wraps all three phases.

This is the complete system. Every module contributes:

graph LR
  subgraph Foundations["Modules 1-2"]
      A1["Agent Loop"] --> A2["Prompting"]
  end
  subgraph Capabilities["Modules 3-6"]
      A2 --> B1["Tool Use"]
      B1 --> B2["Architectures"]
      B2 --> B3["Patterns"]
      B3 --> B4["Memory & RAG"]
  end
  subgraph Scale["Modules 7-9"]
      B4 --> C1["Frameworks"]
      C1 --> C2["Multi-Modal"]
      C2 --> C3["Multi-Agent"]
  end
  subgraph Quality["Modules 10-12"]
      C3 --> D1["Evaluation"]
      D1 --> D2["Production"]
      D2 --> D3["Capstone"]
  end
  style Foundations fill:#fed7d7,stroke:#c53030
  style Capabilities fill:#ebf8ff,stroke:#3182ce
  style Scale fill:#fefcbf,stroke:#b7791f
  style Quality fill:#c6f6d5,stroke:#38a169
  style D3 fill:#bee3f8,stroke:#2b6cb0,stroke-width:3px

The curriculum flows from left to right: foundations give you the mental models, capabilities give you the building blocks, scale teaches you to compose them, and quality teaches you to ship them. The capstone (highlighted) is where all four layers converge into a single working system.

Extending the Capstone

The skeleton above is a starting point. Here are concrete extensions you can build to deepen each skill area:

  • Add conversation memory (Module 7) -- let users ask follow-up questions that reference the previous report, using sliding-window context management
  • Add multi-modal research (Module 9) -- extend the research agent to analyze charts, diagrams, and images found during web search
  • Add A2A protocol (Module 10) -- replace direct function calls between agents with the Agent-to-Agent protocol, so agents can be deployed as independent services
  • Add tracing (Module 11) -- integrate OpenTelemetry to trace the full request path across all agents, capturing latency, token usage, and decision points at each step
  • Add circuit breakers (Module 12) -- wrap each agent's LLM calls in circuit breakers so that one failing agent does not take down the entire system
  • Add LLM-based guardrails (Module 12) -- replace the regex input guardrails with a classifier model that detects subtle prompt injection and topic drift

Each extension takes 30-60 minutes and reinforces the specific module it draws from.

What You Have Learned

You started this academy with a question: What is an LLM agent? Now you can answer it with precision and build one from scratch. Here is the full map of your journey:

Module 1 -- Foundations: You learned that an agent is an LLM embedded in a loop of perception, reasoning, and action. Every agent in the capstone -- the supervisor, the researcher, the writer, the fact-checker -- follows this loop.

Module 2 -- Prompting & Reasoning: You learned that how you instruct the LLM determines what it can do. Each agent in the capstone has a carefully crafted system prompt that shapes its behavior, and the supervisor uses chain-of-thought to decompose complex questions.

Module 3 -- Tool Use: You learned that agents become useful when they can interact with the world. The research agent uses tool calling to search vector stores and the web, transforming a language model into an information-gathering system.

Module 4 -- Architectures: You learned the difference between ReAct, plan-and-execute, and other architectures. The research agent uses ReAct for flexible exploration, while the supervisor uses plan-and-execute for structured coordination.

Module 6 -- Design Patterns: You learned routing, reflection, and composition patterns. The supervisor routes tasks to specialized agents, the fact-checker reflects on the writer's output, and the whole system is a composition of focused components.

Module 7 -- Memory & Knowledge: You learned that agents need both short-term context and long-term knowledge. The RAG pipeline grounds the research agent in factual information, and conversation memory lets the system handle follow-up questions.

Module 8 -- Frameworks: You learned when to use a framework and when to build from scratch. The capstone uses the raw Anthropic SDK to show the mechanics, but you could rebuild it in LangGraph, CrewAI, or AutoGen using the patterns you learned.

Module 9 -- Multi-Modal: You learned that agents can perceive images, audio, and video. The capstone's research agent can be extended to analyze visual data sources, bringing multi-modal perception into the research loop.

Module 10 -- Multi-Agent Systems: You learned orchestration, communication, and debate. The supervisor pattern, the delegation protocol, and the adversarial fact-checking loop are all multi-agent techniques you studied in depth.

Module 11 -- Evaluation: You learned that agent quality must be measured, not assumed. The LLM-as-Judge evaluator with its multi-dimensional rubric gives the system a quality gate that prevents low-quality reports from reaching users.

Module 12 -- Production & Safety: You learned that building an agent and shipping an agent are different disciplines. Guardrails, monitoring, circuit breakers, and containerization transform the capstone from a prototype into a production system.

Module 13 -- Advanced Patterns: You explored computer use, coding agents, research agents, domain-specific applications, alignment, and frontier research -- the cutting edge of what agents can do and where the field is heading.

Next Steps

You have completed the LLM Agents Academy. That is a significant achievement -- you now have a mental model for agent systems that spans from token-level prompting to production deployment, from single-agent loops to multi-agent orchestration. But the academy is a beginning, not an end.

Here is what to do next:

Build the capstone for real. Take the skeleton from this lesson and implement it end to end. Replace the placeholder tool implementations with real API calls. Deploy it with Docker. Send it real research questions. The gap between reading code and running code is where the deepest learning happens.

Pick a domain and specialize. The capstone is a general-purpose research system, but the most valuable agents are domain-specific. Take what you have learned and build an agent for your domain -- customer support, code review, data analysis, legal research, medical literature, or whatever problem you care about most.

Stay current. The field of LLM agents is evolving faster than any textbook can capture. Follow the research, try new models as they release, and revisit the benchmarks from Module 11 to track how capabilities are improving. The patterns you have learned are durable -- the specific implementations will evolve.

Share what you build. The agent community is growing rapidly and there is enormous value in sharing patterns, failures, and production lessons. Write about what worked. Write about what did not. The next generation of agent builders will learn from your experience, just as you learned from the experiences captured in this academy.

Summary

In this capstone, you designed and built a production-grade Research & Report system that integrates every concept from the LLM Agents Academy. A supervisor agent decomposes questions into sub-tasks and delegates them to specialized agents. A research agent with RAG gathers information from vector stores and web search. A writer agent synthesizes findings into structured reports. A fact-checker agent uses adversarial debate to verify claims. An LLM-as-Judge evaluator scores the output against a multi-dimensional rubric. And the entire system runs behind input and output guardrails with monitoring and observability. Every module contributed a specific, necessary piece -- from the agent loop in Module 1 to the deployment patterns in Module 12. The architecture is not theoretical; it is a skeleton you can extend into a real product. You have the knowledge. Now build.

💬Ask about this lesson...