Production Lab

Overview

Throughout Module 12, you learned why agents fail in production, how to deploy them with containers and CI/CD, how to make them reliable with circuit breakers and retries, how to protect them with guardrails, and how to monitor their cost and performance. Each lesson introduced one layer of the production stack in isolation. But a real production agent combines all of these layers into a single, deployable system -- a service that is observable, resilient, safe, and ready for real traffic.

In this hands-on lab, you will take a simple research agent and transform it into a production-ready service step by step. You will wrap it in a FastAPI application with health checks, add input and output guardrails, wire in circuit breaker and retry logic, attach monitoring middleware that exports Prometheus metrics, and finally containerize everything with Docker Compose. By the end, you will have a complete, deployable agent service -- and you will understand every layer of the production stack because you built each one yourself.

Production Agent Architecture

Here is the complete system you will build. Study this diagram before writing any code -- it shows every layer, how requests flow through them, and where each production concern lives.

graph TD
  Client["Client Request"] --> LB["Load Balancer / Reverse Proxy"]
  LB --> FW["FastAPI Service"]
  subgraph Service["Production Agent Service"]
      FW --> MW["Monitoring Middleware"]
      MW --> IG["Input Guardrails"]
      IG --> CB["Circuit Breaker"]
      CB --> RT["Retry Logic"]
      RT --> Agent["Research Agent Core"]
      Agent --> OG["Output Guardrails"]
      OG --> FW
  end
  subgraph Observability["Observability Stack"]
      MW -->|"metrics"| Prom["Prometheus"]
      MW -->|"logs"| Logs["Structured Logs"]
      Prom --> Graf["Grafana Dashboard"]
  end
  Agent -->|"API calls"| LLM["LLM Provider"]
  Agent -->|"tool calls"| Tools["External Tools"]
  style Service fill:#ebf8ff,stroke:#3182ce
  style Observability fill:#f0fff4,stroke:#38a169
  style MW fill:#fefcbf,stroke:#b7791f
  style IG fill:#fed7d7,stroke:#c53030
  style CB fill:#e9d8fd,stroke:#805ad5
  style OG fill:#fed7d7,stroke:#c53030
  style Agent fill:#c6f6d5,stroke:#38a169

The request flows through five layers before reaching the agent core, and the response passes through output guardrails on the way back. The monitoring middleware wraps the entire pipeline, capturing latency, token usage, and error rates. The guardrails sit at the boundaries -- input guardrails reject dangerous requests before they reach the agent, and output guardrails filter the agent's response before it reaches the client. The circuit breaker and retry logic protect the agent from cascading failures when the LLM provider is degraded.

This is the defense-in-depth architecture from lesson 05 combined with the observability stack from lesson 06, all running inside the containerized deployment from lesson 02.

Step 1: The Agent Core

Before adding production layers, you need the agent itself. This is a simple research agent that takes a question, calls an LLM to generate a search plan, executes searches, and synthesizes a response. In a real system, this would be the agent you have already built and tested in development.

This is deliberately simple. The agent has a single method, run, that takes a question and returns a structured AgentResponse. The structured response is critical for production -- downstream layers need to inspect the response programmatically, not parse free-form text. Notice that the agent tracks tokens_used, which the monitoring layer will need later.

Step 2: Wrap in FastAPI with Health Checks

The first production layer is the HTTP service. Wrapping the agent in FastAPI gives you a standard API that load balancers, container orchestrators, and monitoring systems know how to work with. The health check endpoints are not optional -- Kubernetes uses them to decide whether to route traffic to your pod or restart it.

A few design decisions to notice:

• Two health endpoints, not one. The /health/live endpoint tells Kubernetes "the process is running" -- it should almost never fail. The /health/ready endpoint tells Kubernetes "the service can handle traffic" -- it fails when the agent is not initialized or the error rate is too high. Kubernetes uses these differently: a failed liveness probe restarts the pod, while a failed readiness probe just stops routing traffic to it.

• Pydantic validation on the request. The QueryRequest model rejects questions shorter than 3 characters or longer than 2000 characters before the agent ever sees them. This is your first layer of input validation -- cheap and fast.

• Structured logging with context. Every request logs its token count and latency. When you are debugging a production issue at 3 AM, these structured fields let you filter and aggregate logs programmatically instead of grepping through free text.

Step 3: Add Input and Output Guardrails

Pydantic validation catches malformed requests, but it does not catch malicious requests. Input guardrails inspect the semantic content of the question -- blocking prompt injection attempts, off-topic queries, and requests for harmful content. Output guardrails inspect the agent's response before it reaches the client, filtering PII leaks, hallucinated citations, and unsafe content.

These guardrails are deliberately simple -- they use regex patterns and keyword matching rather than a secondary LLM call. In lesson 05, you learned about LLM-based guardrails that use a classifier model to evaluate inputs and outputs. Those are more powerful but add latency and cost. In practice, you layer both: fast regex guardrails catch the obvious attacks with zero latency, and LLM-based guardrails handle the subtle cases. The regex layer runs first because it is cheap -- if it rejects the request, you never pay for the LLM classifier.

Now integrate the guardrails into the service:

Notice the order: input guardrails run before the agent, output guardrails run after. If the input guardrail rejects the request, you return a 400 (client error) because the problem is with the request. If the output guardrail rejects the response, you return a 500 (server error) because the agent produced something it should not have. This distinction matters for monitoring -- a spike in 400s means clients are sending bad requests, while a spike in 500s means the agent is misbehaving.

Step 4: Add Circuit Breaker and Retry Logic

The agent depends on an external LLM provider. When that provider is slow or down, your service should not keep hammering it with requests -- that makes the outage worse and wastes money on timed-out calls. The circuit breaker pattern detects when the provider is failing and short-circuits requests, returning an error immediately instead of waiting for a timeout. The retry logic handles transient failures -- a single 429 (rate limit) or 500 (server error) from the provider does not mean the request should fail permanently.

Now wrap the agent call with both the circuit breaker and retry logic:

The retry logic and the circuit breaker work together but at different timescales. Retries handle individual transient failures -- a single 429 response that succeeds on the next attempt. The circuit breaker handles sustained outages -- when the provider is down for minutes, the circuit breaker stops all retries and returns errors immediately. Without the circuit breaker, your retry logic would keep sending requests during an outage, accumulating timeout costs and delaying your recovery.

Step 5: Add Monitoring Middleware

Every production service needs metrics. Without them, you are flying blind -- you do not know your latency distribution, your error rate, or how many tokens you are burning. The monitoring middleware wraps the entire request pipeline, capturing metrics that Prometheus can scrape and Grafana can visualize.

The histogram buckets for latency are chosen deliberately for agent workloads. Traditional web services use buckets measured in milliseconds -- 10ms, 50ms, 100ms. Agent services use buckets measured in seconds because LLM calls are inherently slow. The buckets [0.1, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0] let you track the full distribution from fast cache-hit responses to slow complex queries.

You should also instrument the guardrails so you can track rejection rates:

With these metrics, you can set up Grafana dashboards that show request rate, latency percentiles (p50, p95, p99), error rate, token burn rate, and guardrail rejection rate -- all in real time. You can also configure alerts: page someone when the error rate exceeds 5%, when latency p95 exceeds 30 seconds, or when the circuit breaker opens.

Step 6: Containerize with Docker

The final step is packaging everything into a container that any cloud provider can run. The Dockerfile uses a multi-stage build to keep the image small, and the docker-compose.yaml wires up the agent service with Prometheus and Grafana for a complete local development stack.

A few production details that matter:

• Multi-stage build. The builder stage installs dependencies, and the runtime stage copies only the installed packages. This keeps the final image small -- no compiler toolchains, no pip cache.
• Non-root user. Running as root inside a container is a security risk. The appuser has only the permissions it needs.
• HEALTHCHECK directive. Docker and orchestrators use this to monitor the container. It hits the /health/live endpoint every 15 seconds.
• Two workers. Uvicorn's --workers 2 runs two process workers behind a single port. This lets the service handle concurrent requests even though each LLM call blocks a worker for several seconds.

Now the Docker Compose file that ties everything together:

With docker compose up --build, you get three services: the agent on port 8000, Prometheus on port 9090 scraping metrics every 15 seconds, and Grafana on port 3001 ready for you to build dashboards. The deploy.resources.limits section caps the agent at 512MB of memory and one CPU core -- in production, you would tune these based on your load testing results.

The Complete Request Lifecycle

Now that every layer is in place, here is how a single request flows through the entire system:

sequenceDiagram
  participant C as Client
  participant M as Metrics Middleware
  participant IG as Input Guardrails
  participant CB as Circuit Breaker
  participant R as Retry Logic
  participant A as Agent Core
  participant OG as Output Guardrails
  participant P as Prometheus

  C->>M: POST /query
  Note right of M: Start timer, increment active_requests

  M->>IG: Validate question
  alt Injection detected
      IG-->>M: BLOCKED
      M-->>C: 400 Bad Request
      M->>P: guardrail_rejections{stage=input}
  else Clean input
      IG->>CB: Forward question
      alt Circuit open
          CB-->>M: CircuitBreakerOpen
          M-->>C: 503 Service Unavailable
      else Circuit closed
          CB->>R: Execute with retries
          loop Up to 3 retries
              R->>A: agent.run(question)
              alt Provider error
                  A-->>R: RateLimitError / 500
                  Note right of R: Exponential backoff
              else Success
                  A-->>R: AgentResponse
              end
          end
          R->>CB: Record success/failure
          R->>OG: Validate response
          alt PII detected
              OG-->>M: BLOCKED
              M-->>C: 500 Internal Server Error
              M->>P: guardrail_rejections{stage=output}
          else Clean output
              OG-->>M: QueryResponse
              M-->>C: 200 OK
          end
      end
  end

  Note right of M: Record latency, decrement active_requests
  M->>P: request_count, request_duration, tokens_total

Every request passes through every layer. The metrics middleware is the outermost layer -- it sees every request and every response, regardless of which inner layer handled it. This gives you complete visibility into your service's behavior.

Verifying the Service

Once you have all the files in place, verify the service works locally before deploying:

If the liveness probe returns "alive", the readiness probe returns "ready", the test query returns an answer, and the injection attempt is blocked -- your production agent is working. Open Prometheus at localhost:9090 to verify that it is scraping metrics from your service, and then build a Grafana dashboard to visualize your agent's latency, throughput, and error rate.

Summary

You have taken a bare agent and wrapped it in five production layers, each one addressing a failure mode you learned about in lessons 01 through 06:

• FastAPI with health checks (lesson 02) -- makes the agent deployable, monitorable, and restartable by orchestrators
• Input and output guardrails (lesson 05) -- protects against prompt injection, PII leakage, and unsafe content
• Circuit breaker and retries (lesson 04) -- prevents cascading failures when the LLM provider is degraded
• Monitoring middleware with Prometheus metrics (lesson 06) -- gives you real-time visibility into latency, errors, tokens, and guardrail rejections
• Docker and Docker Compose (lesson 02) -- packages everything into a portable, reproducible deployment

No single layer is sufficient on its own. Guardrails without monitoring means you do not know how often attacks are blocked. Circuit breakers without health checks means Kubernetes cannot tell when to restart your pod. Monitoring without guardrails means you can see failures but cannot prevent them. The layers reinforce each other -- that is why production systems are built in depth.

Your agent is production-ready -- Module 13 explores advanced patterns, specialized domains, and the cutting edge of agent research.