Spring AI with Ollama (Local LLMs)

Local models are not just a privacy feature — they change your development loop. No API key, no per-call cost, no network dependency in tests. This tutorial covers running Ollama with Spring AI, and is honest about the quality trade-off.

Setting up Ollama#

Install Ollama, then pull a model:

# Install from ollama.com, then:
ollama pull llama3.2          # a small, fast general model
ollama pull nomic-embed-text  # a local embedding model
ollama serve                  # serves on http://localhost:11434

Wiring it into Spring AI#

<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-model-ollama</artifactId>
</dependency>

spring:
  ai:
    ollama:
      base-url: http://localhost:11434
      chat:
        options:
          model: llama3.2
          temperature: 0.2
      embedding:
        options:
          model: nomic-embed-text

Your service code is identical to the OpenAI version — that is the whole point of the ChatModel abstraction:

@Service
public class LocalChatService {
 
    private final ChatClient chatClient;
 
    public LocalChatService(ChatClient.Builder builder) {
        this.chatClient = builder.build();   // no idea it is talking to Ollama
    }
 
    public String ask(String question) {
        return chatClient.prompt().user(question).call().content();
    }
}

Profiles for local vs hosted#

spring:
  ai:
    ollama:
      chat:
        options:
          model: llama3.2

spring:
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      chat:
        options:
          model: gpt-4o-mini

Run locally with --spring.profiles.active=local; deploy with prod. One codebase, two providers.

Choosing a local model#

Start small. A 3B model handles a surprising amount of routine work — intent classification, simple extraction, first-line RAG answering — at zero marginal cost.

Local embeddings: the easy win#

Even if you use a hosted chat model, local embeddings are worth it for development. Spring AI supports in-process ONNX models that need no server:

spring:
  ai:
    embedding:
      transformer:
        onnx:
          model-uri: <all-MiniLM-L6-v2 onnx model>

This removes cost and network latency from your ingestion pipeline entirely, which makes iterating on chunking and retrieval far faster.

When a local model is the right call#

• Privacy and compliance — data that cannot legally or contractually leave your infrastructure.
• High-volume simple tasks — classifying millions of items where hosted per-call cost adds up and a small model is good enough.
• Development and testing — free, offline, fast.
• Air-gapped environments — where there is no external network at all.

When it is not#

• Frontier reasoning — the hardest tasks still need the best hosted models.
• Tiny teams without ops capacity — running and scaling model inference is real operational work; a hosted API removes it.
• Bursty low-volume traffic — you pay for idle hardware; a hosted API bills per use.

The pragmatic pattern many teams land on: local models for the easy, high-volume, or sensitive 80%, and a hosted model for the hard 20%. Routing between them is covered in small language models.

Next#

• Spring AI observability & monitoring
• GenAI on AWS, Azure & GCP — the hosted side