Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Multimodal AI with Spring Boot

Send images and audio to vision models from Spring Boot with Spring AI: the Media API, image analysis, document extraction from scans, and handling multimodal input safely.

Intermediate4 min readUpdated
On this page

Vision models let a Spring Boot application reason about images the way it reasons about text — describe a photo, extract fields from a scanned invoice, check whether an uploaded image matches a policy. Spring AI exposes this through the same ChatClient you already use, with a Media object attached to the message.

Key Takeaways

  • Attach images with the Media API: .media(mimeType, resource) on the user message.
  • The most valuable pattern is structured extraction from scans — combine vision with .entity().
  • Images are expensive in tokens; resize before sending.
  • Validate and constrain uploads — a vision endpoint is a file-upload endpoint with all that implies.

Sending an image

ImageAnalysisService.java
@Service
public class ImageAnalysisService {
 
    private final ChatClient chatClient;
 
    public ImageAnalysisService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }
 
    public String describe(byte[] imageBytes, MimeType mimeType) {
        var media = new Media(mimeType, new ByteArrayResource(imageBytes));
 
        return chatClient.prompt()
                .user(u -> u
                        .text("Describe what is shown in this image in two sentences.")
                        .media(media))
                .call()
                .content();
    }
}

The image and the text prompt arrive together; the model reasons over both.

From an uploaded file

Controller with MultipartFile
@PostMapping("/api/analyse-image")
public AnalysisResponse analyse(@RequestParam MultipartFile file) throws IOException {
 
    // Validate before spending a model call. A vision endpoint is a file-upload
    // endpoint — treat it with the same suspicion.
    if (file.getSize() > 10 * 1024 * 1024) {
        throw new PayloadTooLargeException("image exceeds 10MB");
    }
    MimeType mimeType = MimeTypeUtils.parseMimeType(
            Objects.requireNonNull(file.getContentType()));
    if (!ALLOWED_IMAGE_TYPES.contains(mimeType)) {
        throw new UnsupportedMediaTypeException("only PNG and JPEG are accepted");
    }
 
    String description = analysisService.describe(file.getBytes(), mimeType);
    return new AnalysisResponse(description);
}
 
private static final Set<MimeType> ALLOWED_IMAGE_TYPES =
        Set.of(MimeTypeUtils.IMAGE_PNG, MimeTypeUtils.IMAGE_JPEG);

The high-value pattern: structured extraction from scans

Combining vision with structured output replaces a whole OCR-plus-parsing pipeline. Send a scanned receipt, get back a typed record:

ReceiptExtractor.java
public record Receipt(
        String merchant,
        LocalDate date,
        List<LineItem> items,
        BigDecimal total,
        String currency) {
 
    public record LineItem(String name, BigDecimal price) {}
}
 
@Service
public class ReceiptExtractor {
 
    private final ChatClient chatClient;
 
    public ReceiptExtractor(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }
 
    public Receipt extract(byte[] scan, MimeType mimeType) {
        var media = new Media(mimeType, new ByteArrayResource(scan));
 
        return chatClient.prompt()
                .user(u -> u
                        .text("""
                                Extract the receipt details from this image.
                                Use the exact amounts shown. If a field is not
                                visible, leave it null rather than guessing.
                                """)
                        .media(media))
                .options(ChatOptions.builder().temperature(0.0).build())
                .call()
                .entity(Receipt.class);
    }
}

Multiple images and comparison

Attach several images to compare or combine them:

public String compare(byte[] before, byte[] after, MimeType type) {
    return chatClient.prompt()
            .user(u -> u
                    .text("Describe what changed between the first and second image.")
                    .media(new Media(type, new ByteArrayResource(before)))
                    .media(new Media(type, new ByteArrayResource(after))))
            .call()
            .content();
}

Cost: resize before you send

Images are tokenised by resolution. A full-resolution phone photo can cost as much as a long document. Resize to the smallest size that preserves the detail you actually need:

// Downscale to a max dimension before sending. For most document and photo
// analysis, 1024px on the long edge is plenty and cuts token cost sharply.
Bufferedimage scaled = Thumbnails.of(original).size(1024, 1024).asBufferedImage();

Audio input

The same Media mechanism carries audio for transcription-capable models:

public String transcribe(byte[] audioBytes) {
    var media = new Media(MimeTypeUtils.parseMimeType("audio/mp3"),
            new ByteArrayResource(audioBytes));
    return chatClient.prompt()
            .user(u -> u.text("Transcribe this audio.").media(media))
            .call()
            .content();
}

Some providers offer a dedicated transcription API that is cheaper and more accurate than a general vision model for pure speech-to-text; prefer it when transcription is all you need.

Checklist for a vision endpoint

  • Validate file size and MIME type before the model call
  • Resize to the minimum useful resolution
  • Use structured output for extraction, with "leave unknowns null"
  • Treat extracted text as untrusted input
  • Monitor token cost — images dominate it

Next

Frequently Asked Questions

Which models support image input in Spring AI?
Vision-capable models such as GPT-4o, Claude with vision, and Gemini accept image input through the same Spring AI Media API. You attach a Media object to the user message and the model receives the image alongside your text prompt. Check that your configured model supports vision before sending images.
How do I send an image to a model in Spring Boot?
Create a Media object from the image bytes and MIME type, then attach it to the user message with .media(). Spring AI encodes and transmits it to the provider. The image can come from an uploaded MultipartFile, a URL, or a classpath resource.
Is multimodal input more expensive?
Yes. Images are tokenised into a significant number of tokens depending on resolution, often equivalent to hundreds or thousands of text tokens per image. Resize images to the smallest resolution that preserves the detail you need before sending, and be deliberate about how many images you attach.
Can I extract structured data from a scanned document?
Yes, and it is one of the most useful multimodal patterns. Send the scan as an image with a prompt describing the fields to extract, and combine it with structured output so the response comes back as a typed Java record. This handles receipts, invoices and forms without a separate OCR pipeline.

Related tutorials