LLM Agents Academy
Module 7: Memory & Knowledge

Memory Lab

Hands-on: build an agent with conversation memory and RAG-powered knowledge

~15 min readIncludes code

Overview

Throughout Module 7, you learned the individual building blocks of agent memory: conversation buffers, summarization strategies, long-term storage, embeddings, vector stores, and advanced RAG patterns. Each lesson focused on one piece of the puzzle. In this lab, you will put every piece together.

You are going to build a knowledge-grounded agent -- an agent that maintains conversation memory across turns, ingests documents into a vector store, retrieves relevant knowledge before answering, and combines all of this into a single coherent system. This is the architecture behind every serious AI assistant that needs to be both conversational and factually grounded.

The build is progressive. You will start with a bare agent loop, then add conversation memory, then build a document ingestion pipeline, then wire up RAG retrieval as a tool, and finally combine everything into a complete memory-augmented agent. At each step, the agent still works -- you are layering capabilities, not doing a big-bang rewrite.

By the end of this lab, you will have built from scratch what frameworks like LangChain and LangGraph provide out of the box. That is the point. When you understand the internals, you can make informed decisions about when to use a framework and when to build your own.

The Complete Architecture

Before writing any code, let's look at the system you are going to build. Every component in this diagram corresponds to a class or function you will implement in this lab.

graph TD
  User["User"] -->|"question"| Agent["Agent Loop"]

  Agent -->|"check history"| ConvMem["Conversation Memory"]
  ConvMem -->|"recent turns + summary"| Agent

  Agent -->|"retrieve docs"| RAGTool["RAG Retrieval Tool"]
  RAGTool -->|"query"| VectorStore["Vector Store<br/>(ChromaDB)"]
  VectorStore -->|"relevant chunks"| RAGTool
  RAGTool -->|"context"| Agent

  Agent -->|"generate"| LLM["LLM API<br/>(Claude)"]
  LLM -->|"response or tool_use"| Agent

  Agent -->|"response"| User

  Ingest["Document Ingestion<br/>Pipeline"] -->|"chunks + embeddings"| VectorStore

  subgraph Memory["Memory Layer"]
      ConvMem
      VectorStore
  end

  subgraph Tools["Tool Layer"]
      RAGTool
      OtherTools["Other Tools<br/>(search, calculate, etc.)"]
  end

  Agent -->|"call"| OtherTools
  OtherTools -->|"result"| Agent

  style Memory fill:#ebf8ff,stroke:#3182ce
  style Tools fill:#f0fff4,stroke:#38a169
  style Agent fill:#fefcbf,stroke:#d69e2e
  style LLM fill:#fed7e2,stroke:#d53f8c
  style User fill:#e9d8fd,stroke:#805ad5
  style Ingest fill:#feebc8,stroke:#dd6b20

The architecture has two memory systems working in parallel. Conversation memory tracks what the user and agent have said in the current session -- it handles the "what did we just talk about?" problem. The vector store holds pre-ingested documents -- it handles the "what do you know about X?" problem. The agent consults both before generating a response, giving it continuity within a conversation and grounding in external knowledge.

Step 1: Conversation Memory

The first layer to add is conversation memory. You built the pieces in lessons 02 and 03 -- a buffer that stores recent turns and a summarizer that compresses older history. Here, you combine them into a single ConversationMemory class that the agent loop can use without thinking about the details.

conversation_memory.py

The key design decision here is the split between buffer and summary. The buffer holds recent turns in full fidelity -- the model can see exact wording, nuances, and follow-up context. The summary compresses everything older into a paragraph. This gives the agent both precision (for the current thread) and context (for the broader conversation) without exceeding the context window.

Notice that the summary is incremental. When new turns get compressed, the summarizer receives both the existing summary and the new turns, producing a single updated summary. This avoids the information loss you would get from re-summarizing a summary of a summary.

Step 2: Document Ingestion Pipeline

The second layer is the document ingestion pipeline -- the system that takes raw documents, splits them into chunks, generates embeddings, and stores them in a vector database. This is the "write path" of your RAG system.

flowchart LR
  Docs["Raw Documents<br/>(PDF, TXT, MD)"] --> Loader["Document<br/>Loader"]
  Loader --> Chunker["Text Chunker<br/>(overlap strategy)"]
  Chunker --> Embedder["Embedding<br/>Model"]
  Embedder --> Store["Vector Store<br/>(ChromaDB)"]

  subgraph Pipeline["Ingestion Pipeline"]
      Loader
      Chunker
      Embedder
  end

  style Pipeline fill:#fefcbf,stroke:#d69e2e
  style Store fill:#ebf8ff,stroke:#3182ce
  style Docs fill:#e9d8fd,stroke:#805ad5

You covered chunking strategies in lesson 05 and embedding models in lesson 05. Here, you wire them together into a pipeline that can process a directory of documents in one call.

ingestion_pipeline.py

Three design decisions worth noting. First, the chunker tries to break at sentence boundaries rather than cutting mid-sentence -- this produces chunks that are more coherent when retrieved. Second, each chunk gets a deterministic ID based on its source and content hash, which means re-ingesting the same document updates existing chunks via upsert rather than creating duplicates. Third, the pipeline stores source metadata with every chunk, which lets the agent cite where its knowledge came from.

Step 3: RAG Retrieval as a Tool

The third layer turns your vector store into an agent tool. Instead of hardcoding retrieval into the agent loop, you expose it as a tool the agent can choose to call -- or not -- based on the question. This is the agentic RAG pattern from lesson 06: the agent decides when retrieval is helpful.

rag_tool.py

The tool description is critical. It tells the model when to use retrieval ("when the user asks about topics that may be covered in the ingested documents") and how to use it ("be specific about what information you need"). A vague tool description leads to either over-retrieval (the agent searches for everything, wasting tokens) or under-retrieval (the agent guesses instead of looking things up).

Step 4: The Complete Memory-Augmented Agent

Now you combine all three components -- conversation memory, document ingestion, and RAG retrieval -- into a single agent. This is the full system from the architecture diagram at the top of this lesson.

sequenceDiagram
  participant User
  participant Agent as Agent Loop
  participant Mem as ConversationMemory
  participant LLM as Claude API
  participant RAG as search_knowledge_base
  participant VS as Vector Store

  User->>Agent: "What is RLHF?"
  Agent->>Mem: add_user_message(question)
  Agent->>Mem: get_messages()
  Mem-->>Agent: [summary] + [recent turns]

  Agent->>LLM: messages.create(tools, messages)
  LLM-->>Agent: tool_use: search_knowledge_base

  Agent->>RAG: search("RLHF reinforcement learning")
  RAG->>VS: query(embedding)
  VS-->>RAG: top-5 chunks
  RAG-->>Agent: JSON with chunks + sources

  Agent->>LLM: messages.create(tool_result)
  LLM-->>Agent: text response with citations

  Agent->>Mem: add_assistant_message(response)
  Agent-->>User: grounded answer with sources

The sequence shows how both memory systems participate in every turn. Conversation memory provides context about what has been discussed. The RAG tool provides factual grounding from documents. The LLM synthesizes both into a coherent response.

memory_agent.py

The chat method is the heart of the agent. Notice how simple the loop is -- complexity is pushed into the components. Conversation memory handles history management. The retriever handles vector search. The agent loop just orchestrates: get messages, call the LLM, handle tools, record the response.

Running the Agent

Here is how you use the complete system. The agent ingests documents once, then handles a multi-turn conversation where it draws on both conversation history and retrieved knowledge.

run_agent.py

Watch what happens across the three questions:

  • Question 1: The agent calls search_knowledge_base with "RLHF reinforcement learning from human feedback", retrieves the chunks from rlhf-overview.md, and explains the three-stage process with citations.
  • Question 2: The agent does not call the tool. "Three stages" is a reference to the previous answer, which is in conversation memory. The agent draws on the earlier turn to discuss costs -- particularly human annotation for the reward model.
  • Question 3: The agent calls the tool again, searching for "alternatives to RLHF", and retrieves the DPO and RLAIF chunks. It connects this to the earlier discussion: "As I mentioned, RLHF has high annotation costs -- DPO addresses this by..."

This is the payoff of combining conversation memory with RAG. Neither system alone could handle all three questions well. Without conversation memory, question 2 would be nonsensical ("which three stages?"). Without RAG, the answers would come from general knowledge and might contain hallucinated details about the algorithms.

How the Two Memory Systems Interact

The interaction between conversation memory and RAG retrieval is worth understanding in detail, because getting it wrong leads to subtle bugs.

graph TD
  subgraph Turn["Each Conversation Turn"]
      direction TB
      Q["User question arrives"]
      Q --> CM["ConversationMemory<br/>adds question to buffer"]
      CM --> Build["Build message list:<br/>summary + recent turns"]
      Build --> LLM1["LLM decides: answer<br/>directly or use tools?"]
      LLM1 -->|"needs facts"| RAG["RAG tool retrieves<br/>relevant chunks"]
      LLM1 -->|"has context"| Gen["Generate response<br/>from memory alone"]
      RAG --> LLM2["LLM synthesizes<br/>chunks + conversation"]
      LLM2 --> Store["ConversationMemory<br/>stores response"]
      Gen --> Store
  end

  subgraph Background["Before Conversation"]
      Docs["Documents"] --> Ingest["Ingestion Pipeline"]
      Ingest --> VS["Vector Store<br/>(persistent)"]
  end

  VS -.->|"queried by"| RAG

  style Turn fill:#f7fafc,stroke:#a0aec0
  style Background fill:#fffff0,stroke:#d69e2e
  style LLM1 fill:#fed7e2,stroke:#d53f8c
  style LLM2 fill:#fed7e2,stroke:#d53f8c
  style RAG fill:#f0fff4,stroke:#38a169
  style CM fill:#ebf8ff,stroke:#3182ce

The critical insight is when each memory system contributes. Conversation memory is always present -- every LLM call includes the summary and recent turns. RAG retrieval is conditional -- the LLM decides whether to search based on whether the question needs factual grounding. This is why the tool description matters so much: it guides the model's decision about when to retrieve.

There is also a subtle feedback loop. When the agent retrieves information and uses it in a response, that response gets stored in conversation memory. On later turns, the summarized version of that response becomes part of the context. This means RAG results gradually become part of the conversation's "common ground" -- the agent can reference them later without re-retrieving.

Adding More Tools

The architecture is designed to accommodate additional tools alongside RAG. You might want a calculator, a web search tool, or a code execution tool. Here is how you extend the agent without restructuring anything.

extra_tools.py

Adding a tool is three things: a function, a schema, and a handler entry. The agent loop does not change. The LLM decides when to use each tool based on the descriptions. This is the same extensibility pattern you saw with the Tool Registry in Module 6, but now the tools have access to persistent memory through the shared vector store.

What You Built

Take a step back and look at what the complete system does:

  • Conversation memory gives the agent continuity within a session. It remembers what was discussed, can reference earlier points, and does not repeat itself. The automatic summarization keeps the context window manageable even for long conversations.

  • Document ingestion gives the agent a knowledge base. Raw documents are chunked, embedded, and stored in a vector database. The ingestion is idempotent -- re-ingesting the same documents updates rather than duplicates.

  • RAG retrieval gives the agent factual grounding. When a question touches on topics in the knowledge base, the agent retrieves relevant chunks and cites its sources. When the knowledge base does not have relevant information, the agent says so.

  • The agent loop orchestrates everything. It is simple because complexity lives in the components. Adding new capabilities means adding new tools, not rewriting the loop.

This is roughly 300 lines of Python, not counting the sample documents. That is the entire infrastructure for a knowledge-grounded conversational agent. It is not production-ready -- you would want error handling, rate limiting, better chunking strategies, reranking, and observability -- but the architecture is sound. Every production system you will see in Module 8 follows this same shape.

What Frameworks Give You

You built this from scratch. It works. But look at the code and count how much of it is your logic versus infrastructure plumbing. The conversation memory class is mostly boilerplate for managing a list and calling a summarization API. The ingestion pipeline is mostly file I/O and ChromaDB wiring. The RAG tool is mostly JSON formatting. The agent loop is mostly the same tool-dispatch pattern you have written since Module 3.

This is exactly the problem that agent frameworks solve. They provide the plumbing so you can focus on the logic. LangChain gives you ConversationBufferMemory and ConversationSummaryMemory as one-liners. LangGraph lets you define the retrieval-generation flow as a graph with built-in state management. The Anthropic SDK handles the tool-dispatch loop for you.

But here is why you built it from scratch first: when ConversationSummaryMemory does not behave the way you expect, you now know exactly what it is doing underneath. When LangGraph's retrieval node returns unexpected results, you can reason about the chunking and embedding pipeline because you wrote one. When the framework's agent loop does not fit your use case, you know how to build your own because you have already done it.

Bridge to Module 8: You have now built every core component of an agent system by hand -- tool use (Module 3), design patterns (Module 6), and memory with RAG (this lab). Module 8 shows how frameworks like LangChain, LangGraph, the OpenAI Agents SDK, and the Anthropic Claude SDK package these same components into reusable abstractions. You will see the same patterns, the same architecture, and the same tradeoffs -- just expressed in the framework's vocabulary instead of raw Python.

Summary

In this lab, you built a complete knowledge-grounded agent with two memory systems working in parallel:

  • Conversation memory with buffer and summarization tracks dialogue across turns. Older turns are compressed into a running summary, keeping the context window manageable while preserving continuity. The summary is incremental -- each compression incorporates the previous summary.

  • A document ingestion pipeline processes raw documents into a vector store. Text is split into overlapping chunks at sentence boundaries, embedded using a sentence-transformer model, and stored in ChromaDB with source metadata for citation.

  • RAG retrieval as an agent tool lets the model decide when to search the knowledge base. The tool description guides the model's retrieval decisions. Retrieved chunks flow through the same tool-use mechanism as any other tool.

  • The complete agent combines all three components in a simple loop. Conversation memory provides every-turn context. RAG provides on-demand factual grounding. The agent synthesizes both into responses that are conversational, accurate, and citable.

You also saw how the two memory systems interact: conversation memory is always present, RAG is conditional, and retrieved information gradually becomes part of the conversation's summarized context.

This lab concludes Module 7. You have gone from understanding why agents need memory to building a fully functional memory-augmented agent. In Module 8: Agent Frameworks and SDKs, you will see how production frameworks implement these same patterns -- and you will be equipped to evaluate them critically because you have built the internals yourself.

💬Ask about this lesson...