LLM Agents Academy

Overview

In previous lessons, you explored how agents perceive images, process audio, generate visual content, and reason across modalities. But those capabilities are only as useful as the tools that connect them to the real world. An agent that can analyze images is powerful -- an agent that can capture its own screenshots, drive a browser, read a camera feed, and process arbitrary file types is transformative.

Multi-modal tool use is the practice of equipping agents with tools that capture, transform, and interact with non-text data. Instead of waiting for a user to upload an image, the agent takes a screenshot itself. Instead of asking for a URL's content, the agent launches a browser and navigates to the page. Instead of requiring pre-processed data, the agent detects a file's type and routes it to the right handler.

This lesson covers the architecture of multi-modal tool ecosystems, walks through screen capture, browser automation, camera input, and file-type-aware processing, and introduces Claude's computer use capabilities. By the end, you will have a working pattern for building agents that can see and interact with their digital environment autonomously.

The Multi-Modal Tool Ecosystem

A multi-modal agent does not rely on a single tool. It draws from an ecosystem of specialized tools, each handling a different modality or interaction pattern. The agent's reasoning layer decides which tool to invoke based on the task at hand.

graph TD
  subgraph Agent["Multi-Modal Agent"]
      LLM["LLM Reasoning Engine<br/>Decides which tool to use"]
  end

  subgraph ScreenTools["Screen & Display"]
      SC["Screenshot Capture<br/>Grab screen or window"]
      CU["Computer Use<br/>Mouse, keyboard, observe"]
  end

  subgraph BrowserTools["Browser Automation"]
      PW["Playwright / Puppeteer<br/>Navigate, click, extract"]
      WS["Web Scraper<br/>Structured data extraction"]
  end

  subgraph CameraTools["Camera & Vision"]
      WC["Webcam Capture<br/>Live camera frames"]
      VF["Video Frames<br/>Extract frames from video"]
  end

  subgraph FileTools["File Processing"]
      PDF["PDF Extractor<br/>Text, tables, images"]
      IMG["Image Analyzer<br/>OCR, classification"]
      XLS["Spreadsheet Parser<br/>Rows, charts, formulas"]
      DOC["Document Converter<br/>Format transformation"]
  end

  LLM --> SC
  LLM --> CU
  LLM --> PW
  LLM --> WS
  LLM --> WC
  LLM --> VF
  LLM --> PDF
  LLM --> IMG
  LLM --> XLS
  LLM --> DOC

  style Agent fill:#ebf8ff,stroke:#3182ce,stroke-width:2px
  style ScreenTools fill:#c6f6d5,stroke:#38a169,stroke-width:1px
  style BrowserTools fill:#fefcbf,stroke:#d69e2e,stroke-width:1px
  style CameraTools fill:#fed7e2,stroke:#d53f8c,stroke-width:1px
  style FileTools fill:#e9d8fd,stroke:#805ad5,stroke-width:1px
  style LLM fill:#e0f0ff,stroke:#3182ce,stroke-width:2px

The key insight is that each tool category serves a different purpose:

Screen and display tools let the agent see what is on a computer screen, enabling it to interact with any GUI application
Browser automation tools give the agent programmatic control over web pages, going beyond what a screenshot reveals
Camera and vision tools connect the agent to the physical world through live or recorded video
File processing tools let the agent handle arbitrary file types intelligently, extracting structured data from PDFs, images, spreadsheets, and more

Screen Capture Tools

The simplest and most immediately useful multi-modal tool is screen capture. By taking a screenshot and sending it to a vision-capable LLM, an agent can understand what is displayed on a screen without needing API access to the underlying application.

Screen capture tools typically work in three steps:

Capture -- take a screenshot of the full screen, a specific window, or a defined region
Encode -- convert the image to a format the LLM can consume (base64-encoded PNG or JPEG)
Analyze -- send the encoded image to the LLM as an image content block alongside a text prompt

screenshot_tool.py

This pattern is deceptively powerful. With just a screenshot tool and a vision-capable LLM, an agent can:

Monitor dashboards -- periodically screenshot a Grafana or Datadog dashboard and summarize the current state
Verify deployments -- capture the deployed web page and confirm it matches expectations
Read error dialogs -- identify and describe error messages from any GUI application
Extract data from legacy systems -- read information from applications that have no API

Browser Automation with Playwright

Screenshots show you what a page looks like, but browser automation lets the agent interact with it. Playwright is the most widely used library for programmatic browser control, and it pairs naturally with LLM agents.

The following sequence shows how an agent uses Playwright to navigate to a page, take a screenshot for analysis, and then decide what to click next.

sequenceDiagram
  participant User
  participant Agent as LLM Agent
  participant PW as Playwright Browser
  participant LLM as Claude Vision

  User->>Agent: "Find the pricing for Product X on example.com"
  Agent->>PW: Navigate to example.com
  PW-->>Agent: Page loaded
  Agent->>PW: Take screenshot
  PW-->>Agent: Screenshot (PNG bytes)
  Agent->>LLM: Screenshot + "Where is the pricing link?"
  LLM-->>Agent: "There is a 'Pricing' link in the top navigation bar at approximately (450, 35)"
  Agent->>PW: Click at (450, 35)
  PW-->>Agent: Pricing page loaded
  Agent->>PW: Take screenshot of pricing page
  PW-->>Agent: Screenshot (PNG bytes)
  Agent->>LLM: Screenshot + "Extract all pricing tiers and features"
  LLM-->>Agent: Structured pricing data
  Agent->>User: "Here are the pricing tiers for Product X: ..."

The Playwright tool integrates into the agent's tool ecosystem as a set of actions the LLM can invoke:

browser_tool.py

The agent uses the browser tool in its loop just like any other tool: the LLM reasons about what action to take, emits a tool_use block (e.g., navigate, click, screenshot), and your code dispatches it to the BrowserTool instance. The screenshot capability is especially important because it lets the LLM see what the page looks like after each interaction and decide what to do next.

Camera and Webcam Input

For agents that operate in the physical world -- monitoring a physical space, reading handwritten notes, or inspecting physical objects -- camera input bridges the gap between digital reasoning and physical perception.

The pattern mirrors screen capture: capture a frame, encode it, send it to the LLM. The difference is the source.

camera_tool.py

Camera tools are particularly valuable for inventory monitoring (point a camera at shelves and ask "which items are running low?"), security (analyze camera feeds for anomalies), and accessibility (describe the physical environment for a visually impaired user).

File-Type-Aware Processing

Real-world agents receive files in many formats: PDFs, images, spreadsheets, CSVs, Word documents, and more. A naive agent would try to send every file directly to the LLM. A well-designed agent uses file-type-aware processing -- detecting the file type and routing it to a specialized handler that extracts structured data before sending it to the LLM.

file_processor.py

The FileProcessor class acts as a router: it inspects the MIME type and delegates to the right handler. This keeps each handler simple and focused while giving the agent the ability to work with diverse file types. When the agent encounters a PDF, it gets structured text and extracted images. When it encounters a spreadsheet, it gets rows of data. The LLM never sees raw binary -- it always receives clean, structured input.

Claude's Computer Use Capabilities

Everything we have discussed so far -- screen capture, browser automation, file processing -- can be unified under a single, powerful abstraction: computer use. Claude's computer use feature lets the agent control a full desktop environment by taking screenshots, moving the mouse, clicking, typing, and observing the results.

Computer use works through three built-in tool types:

computer -- take screenshots, move the mouse, click, type, and scroll
text_editor -- view, create, and edit text files
bash -- execute shell commands

The agent operates in a loop: it takes a screenshot to see the current state of the screen, reasons about what action to take, executes that action (click, type, run a command), and then takes another screenshot to see the result. This is the same perception-reasoning-action loop from Module 1, but operating at the level of a full graphical desktop.

computer_use_agent.py

Computer use is the most general form of multi-modal tool use. Rather than building specialized tools for every application, the agent interacts with applications the same way a human does -- by looking at the screen, moving the mouse, and typing. This makes it possible to automate tasks in applications that have no API at all.

Computer use is a deep topic with its own patterns for safety, sandboxing, and reliability. Module 13, Lesson 01 (Computer Use Agents) covers it in detail, including how to set up sandboxed environments, handle multi-step GUI workflows, and build production-grade computer use systems.

Putting It All Together: A Multi-Modal Tool Agent

The real power emerges when you combine multiple tool types into a single agent. Here is a practical example: an agent that can take screenshots, process files, and browse the web, choosing the right tool based on the user's request.

multi_modal_agent.py

This agent seamlessly combines screen capture and browser automation in a single conversation. The LLM decides the order of operations: first it takes a screenshot to answer the question about open applications, then it browses to the status page to check service health. Each tool returns data in the format the LLM needs -- images as base64, text as strings, structured data as JSON.

Design Considerations

Building reliable multi-modal tool agents requires attention to several practical concerns:

Image size management -- high-resolution screenshots can be several megabytes. Resize or compress images before sending them to the LLM to stay within token limits and reduce latency. A 1280x720 screenshot is typically sufficient for analysis.
Sandboxing -- browser automation and computer use should run in isolated environments (Docker containers or VMs) to prevent the agent from accidentally modifying your actual system. Never give an agent unsandboxed access to a production desktop.
Error handling -- cameras disconnect, browsers time out, files are corrupted. Every tool should return structured error information that the LLM can reason about, rather than crashing the agent loop.
Privacy and security -- screenshots may capture sensitive information (passwords, private messages, financial data). Log tool inputs and outputs carefully, and consider masking or filtering sensitive regions before sending screenshots to the LLM.
Cost awareness -- image inputs consume significantly more tokens than text. A single screenshot can cost 1,000+ tokens. Agents that take frequent screenshots should be designed with token budgets in mind.

Summary

Multi-modal tool use transforms agents from passive text processors into active participants in their digital environment. By equipping an agent with screen capture tools, it can observe any application's interface. With browser automation through Playwright, it can navigate, interact with, and extract data from web pages. Camera input extends perception into the physical world. File-type-aware processing lets the agent handle PDFs, images, spreadsheets, and documents intelligently by routing each file type to a specialized handler.

Claude's computer use capability unifies these patterns under a single abstraction -- the agent controls a full desktop through screenshots, mouse, and keyboard, just as a human would. This is the most general form of multi-modal tool use, and we will explore it in depth in Module 13, Lesson 01.

The common thread across all of these tools is the same agent loop you learned in Module 3: the LLM reasons about what to do, emits a tool_use block, your code executes the action and returns a tool_result, and the cycle continues. The only difference is that the tools now operate on images, browsers, cameras, and files instead of text and APIs.

In the next lesson, you will put these concepts into practice in the Multi-Modal Lab, building an end-to-end agent that analyzes images, generates reports, and narrates its findings with speech.

Multi-Modal Tool Use