Multimodal Agents

Multimodal agents extend the agent loop to perception — reasoning over images, audio and screens, not just text. This unlocks tasks like understanding scanned documents end to end and operating interfaces that have no API, at the cost of a wider attack surface and, for computer-use agents, real-world control that demands serious guardrails.

Perception in the loop#

A vision model call describes one image. A multimodal agent uses vision as one capability among tools, inside a reasoning loop:

Document-understanding agents#

The most immediately useful multimodal agent: reading multi-page scanned documents and acting on them.

public ProcessingResult process(List<byte[]> scannedPages) {
    var agent = agentBuilder
            .defaultSystem("""
                    You process scanned business documents. Read each page,
                    extract the required fields, and use tools to validate and
                    file them. If a field is unreadable, flag it for human
                    review rather than guessing.
                    """)
            .defaultTools(validationTools, filingTools)   // real actions
            .build();
 
    // The agent looks at pages, extracts, validates against systems, and files
    // — a multi-step task combining vision and tools.
    return agent.processDocument(scannedPages);
}

Combine with structured output so extraction is typed, and the "flag unreadable fields rather than guess" instruction so failures are visible. The AgenticHR project uses document understanding for résumé processing.

Audio agents#

Agents that take audio input — a support call, a voice command — transcribe and act:

// Transcribe, then let the agent reason and act on the content, closing the
// loop from spoken input to real action.
String transcript = transcriber.transcribe(audioBytes);
return supportAgent.handle(transcript);

For pure transcription a dedicated speech-to-text service is cheaper and more accurate than a general model; use the agent loop when the audio content drives multi-step action.

Computer-use agents#

The frontier and the riskiest: agents that perceive a screen and operate an interface by issuing clicks and keystrokes, for applications with no API.

public Result operate(String task) {
    for (int step = 0; step < MAX_STEPS; step++) {
        byte[] screenshot = screen.capture();
        // The model perceives the screen and decides the next UI action.
        UiAction action = agent.decideAction(task, screenshot);
 
        if (action.isDone()) return Result.completed();
 
        // Every action operates a REAL interface — hence the guardrails below.
        controller.perform(action);   // click, type, scroll
    }
    return Result.incomplete("step limit reached");
}

The wider injection surface#

Multimodal input widens the prompt-injection attack surface:

• An image containing text — a screenshot, a photographed note — can carry instructions the vision model reads and follows.
• A document an agent processes can embed instructions in its content.
• A screen a computer-use agent perceives can display adversarial instructions.

The defense is the same discipline as for text agents, applied wider: treat all perceived content as untrusted, validate outputs, and gate consequential actions in code. Do not let what an agent sees directly authorize what it does.

Cost considerations#

Multimodal input is token-expensive — images cost far more tokens than text, and a computer-use agent capturing a screenshot every step multiplies that across the loop. Budget accordingly: resize images, cap loop length, and track cost per run with a hard budget.

Where to start#

Start with document-understanding agents: high value, contained risk (they read and file, they do not control interfaces), and they build directly on the multimodal and agent foundations you already have. Move to computer-use only with the sandboxing and oversight it demands, and a clear reason an API-based approach will not do.

Next#

• Code generation & AI-assisted development
• AI for DevOps & SRE
• Enterprise AI agents