Image and Video Generation

Overview

In the previous lessons, we gave agents the ability to see (vision-language models) and hear (audio and speech). Those are perception capabilities -- the agent consumes existing media and extracts meaning. Now we flip the direction. In this lesson, agents become creators of visual content: generating images from text descriptions, producing short videos from prompts, and iteratively refining their output until it matches the user's intent.

This is not just about calling an API and returning a picture. The interesting part is what happens around the API call. An agent that generates images must engineer effective prompts for the generation model, evaluate whether the output matches the request, handle failures and content policy rejections, and decide when to refine versus when to deliver. These are the same tool-use patterns you learned in Module 3, applied to a new and uniquely challenging domain -- one where the "correctness" of output is subjective and often requires multiple iterations.

The Image Generation Pipeline

When a user asks an agent to create an image, the request passes through several stages before a final image is delivered. Understanding this pipeline is essential for building agents that produce high-quality visual content reliably.

flowchart TD
  A["User Request<br/>'Create a logo for my coffee shop'"] --> B["Prompt Engineering<br/>Expand and optimize for<br/>the generation model"]
  B --> C["Generation API Call<br/>DALL-E 3 / Stable Diffusion /<br/>Midjourney"]
  C --> D{"Content Policy<br/>Check"}
  D -->|Rejected| E["Revise Prompt<br/>Remove flagged elements"]
  E --> B
  D -->|Passed| F["Post-Processing<br/>Resize, format, metadata"]
  F --> G{"User Satisfied?"}
  G -->|No| H["Collect Feedback<br/>What to change?"]
  H --> B
  G -->|Yes| I["Deliver Final Image"]
  style A fill:#ebf8ff,stroke:#3182ce
  style B fill:#fefcbf,stroke:#d69e2e
  style C fill:#e6fffa,stroke:#38b2ac
  style D fill:#fff5f5,stroke:#fc8181
  style F fill:#f0fff4,stroke:#68d391
  style I fill:#f0fff4,stroke:#68d391

The key insight is that this is a loop, not a linear process. The agent does not fire-and-forget. It engineers the prompt, calls the API, checks the result, and refines -- potentially multiple times. This iterative refinement is what separates a useful generation agent from a thin API wrapper.

Prompt Engineering for Image Generation

The gap between what a user says and what an image generation model needs is often enormous. A user might say "a cozy coffee shop." A generation model produces its best work when given something like "a warm interior photograph of a small artisan coffee shop, morning light streaming through large windows, exposed brick walls, wooden tables with steaming ceramic cups, shallow depth of field, shot on 35mm film." Bridging this gap is the agent's first and most important job.

Prompt expansion is the technique where the agent's LLM takes a brief user request and transforms it into a rich, detailed prompt optimized for the generation model. This is where the agent's language understanding becomes a genuine advantage -- it can ask clarifying questions, infer style preferences from context, and translate abstract concepts ("make it feel professional") into concrete visual descriptors.

The expanded prompt typically includes several dimensions:

• Subject -- what is depicted (objects, people, scenes)
• Style -- artistic approach (photorealistic, watercolor, digital art, isometric)
• Composition -- framing, perspective, layout
• Lighting -- natural, studio, dramatic, soft
• Color palette -- warm tones, monochromatic, vibrant
• Technical parameters -- aspect ratio, resolution, camera lens simulation

This prompt expansion step is the bridge between human intent and model capability. Without it, users must learn the generation model's "language" themselves -- knowing that "35mm film" produces a certain look, or that "isometric" triggers a specific perspective. The agent absorbs that expertise.

Integrating DALL-E 3

DALL-E 3 is OpenAI's image generation model, accessible through their API. From an agent's perspective, it is a tool -- you send a text prompt and receive an image. But DALL-E 3 has a distinctive feature that makes it especially interesting for agents: it rewrites your prompt internally to improve generation quality, and it returns the revised prompt alongside the image. This means the agent can see what DALL-E actually generated from, which is invaluable for iterative refinement.

Notice the error handling. Content policy rejections are not bugs -- they are expected events that the agent must handle gracefully. The agent should analyze why the prompt was rejected, revise it to remove problematic elements, and retry. This is exactly the kind of error recovery we discussed in Module 3's lesson on error handling and retries.

Integrating Stable Diffusion

Stable Diffusion offers a different approach to image generation. While DALL-E is a hosted API, Stable Diffusion can run locally or through hosted endpoints like Stability AI's API. For agents, the key difference is control: Stable Diffusion exposes parameters like guidance scale, number of inference steps, and negative prompts that give the agent fine-grained control over generation.

Negative prompts are a concept unique to diffusion models -- they tell the model what to avoid in the generated image. An agent can use negative prompts strategically: "blurry, low quality, distorted faces, extra fingers" is a common baseline that improves output quality.

The choice between DALL-E and Stable Diffusion depends on the agent's requirements. DALL-E excels at understanding complex prompts and produces consistently high-quality results with minimal parameter tuning. Stable Diffusion offers more control, supports negative prompts, and can be self-hosted for cost savings or data privacy. A sophisticated agent might offer both as tools and choose based on the task.

Video Generation

Video generation extends the same principles to moving images, but introduces a critical difference for agents: time. Image generation typically completes in seconds. Video generation can take minutes. This means the agent must handle asynchronous workflows -- submitting a generation job, polling for completion, and managing the wait gracefully.

Models like Runway Gen-3, Pika, and Luma Dream Machine offer text-to-video and image-to-video capabilities. From an agent's perspective, these are asynchronous tools: you submit a request and receive a job ID, then poll until the result is ready.

The exponential backoff pattern is important. Video generation APIs have rate limits and generation times vary. An agent that polls every second wastes resources and risks hitting rate limits. The backoff pattern starts with short intervals and gradually increases, balancing responsiveness with efficiency.

Agent-Controlled Generation Architecture

The real power emerges when you combine all of these pieces into a coherent agent architecture. The agent sits at the center, orchestrating prompt engineering, model selection, generation, evaluation, and iterative refinement -- all driven by the LLM's reasoning capabilities.

graph TD
  subgraph Agent["Generation Agent"]
      LLM["LLM Controller<br/>Claude / GPT-4"]
      PE["Prompt Engineer<br/>Expand & optimize"]
      EVAL["Output Evaluator<br/>Does it match intent?"]
  end

  subgraph ImageTools["Image Generation Tools"]
      DALLE["DALL-E 3<br/>High-quality, prompt rewriting"]
      SD["Stable Diffusion<br/>Fine control, negative prompts"]
  end

  subgraph VideoTools["Video Generation Tools"]
      RUNWAY["Runway Gen-3<br/>Text & image to video"]
      PIKA["Pika<br/>Short-form video"]
  end

  subgraph PostProcess["Post-Processing"]
      RESIZE["Resize & Format"]
      META["Add Metadata"]
      STORE["Store & Deliver"]
  end

  LLM --> PE
  PE --> DALLE
  PE --> SD
  PE --> RUNWAY
  PE --> PIKA
  DALLE --> EVAL
  SD --> EVAL
  RUNWAY --> EVAL
  PIKA --> EVAL
  EVAL -->|"Needs refinement"| PE
  EVAL -->|"Approved"| RESIZE
  RESIZE --> META
  META --> STORE

  style Agent fill:#ebf8ff,stroke:#3182ce
  style ImageTools fill:#fefcbf,stroke:#d69e2e
  style VideoTools fill:#fff5f5,stroke:#fc8181
  style PostProcess fill:#f0fff4,stroke:#68d391

The architecture follows a pattern you should recognize from Module 3: the LLM controller decides which tool to use and what arguments to pass, then interprets the result to decide the next action. The generation APIs are tools in the same sense that a web search or database query is a tool -- the agent calls them, evaluates the response, and decides whether to act further.

Building the Complete Generation Agent

Let's bring everything together into a complete agent that uses image generation as a tool. This agent accepts natural language requests, engineers prompts, generates images, and iteratively refines them based on feedback.

This agent follows the same tool-use loop pattern from Module 3: the LLM reasons about the request, decides to call a tool, receives the result, and either refines or delivers. The generation-specific intelligence lives in the system prompt -- the prompt engineering workflow, the style guidelines, and the refinement limits.

Iterative Refinement in Practice

The most important capability of a generation agent is iterative refinement -- the ability to evaluate its own output and improve it. In practice, this means the agent generates an image, describes what it sees (using the vision capabilities from Lesson 02), compares that description to the user's intent, and decides whether refinement is needed.

This creates a feedback loop:

1. Generate -- call the generation API with an engineered prompt
2. Evaluate -- use vision-language capabilities to describe the result
3. Compare -- check whether the description matches the user's intent
4. Refine -- adjust the prompt to fix discrepancies and regenerate

This loop is what transforms a simple API wrapper into a genuine creative assistant. The agent does not just generate -- it evaluates and iterates, converging on the user's vision through successive refinement.

Key takeaway: The generation API is the tool. The agent's value is everything around it -- understanding the user's intent, engineering effective prompts, evaluating results, and iteratively refining until the output matches the vision.

Choosing the Right Model

Different generation models have different strengths. An agent that has access to multiple models can choose strategically based on the task:

| Capability | DALL-E 3 | Stable Diffusion XL | Runway Gen-3 | Pika | |---|---|---|---|---| | Input | Text | Text + negative prompt | Text / image | Text / image | | Output | Image | Image | Video (4-16s) | Video (3-8s) | | Latency | 5-15 seconds | 5-30 seconds | 1-5 minutes | 1-3 minutes | | Control | Moderate | High (cfg, steps) | Moderate | Moderate | | Best for | General images | Style-specific work | Cinematic video | Quick animations | | API style | Synchronous | Synchronous | Async (polling) | Async (polling) |

An intelligent agent uses this knowledge to route requests. A logo request goes to DALL-E with quality: "hd" and style: "natural". A photorealistic product shot might go to Stable Diffusion with a high cfg_scale and specific negative prompts. A product demo clip goes to Runway. The routing decision is itself an act of reasoning -- exactly what LLM-based agents excel at.

Summary

Image and video generation transforms agents from consumers of visual content into creators of it. The key principles are:

• Prompt engineering is the critical first step -- the agent's LLM bridges the gap between casual user requests and the detailed, structured prompts that generation models need.
• DALL-E 3 offers high-quality generation with automatic prompt rewriting, making it the simplest integration path. Stable Diffusion provides fine-grained control through parameters like guidance scale, inference steps, and negative prompts.
• Video generation introduces asynchronous workflows -- the agent must submit jobs, poll for completion with exponential backoff, and handle longer processing times gracefully.
• Iterative refinement is what separates a generation agent from an API wrapper. The agent evaluates output, compares it to the user's intent, and refines the prompt -- creating a feedback loop that converges on the desired result.
• Generation APIs are tools in the same sense we explored in Module 3 -- the agent calls them through function calling, interprets the results, and decides the next action based on its reasoning.

In the next lesson, we will explore cross-modal reasoning -- agents that combine multiple modalities in a single task, such as seeing a chart, generating a description, and producing an audio narration of the findings.