Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

LangChain4j Chat Models

Configure chat models in LangChain4j: OpenAI, Anthropic Claude, Google Gemini, Mistral and Ollama — with streaming, timeouts, retries and how to swap providers without touching your code.

Beginner3 min readUpdated
On this page

The ChatModel is the seam that makes provider choice a one-line decision. This tutorial shows how to configure each major provider, add streaming and resilience, and swap between them without touching a line of your application code.

Key Takeaways

  • Every provider is a ChatModel implementation — swap the constructor, keep everything else.
  • Always set a timeout and retries; a model call with neither is a latent outage.
  • StreamingChatModel for interactive UIs, ChatModel for everything else.
  • Configure once in a @Bean; inject the interface everywhere.

The providers

OpenAI
ChatModel openai = OpenAiChatModel.builder()
        .apiKey(System.getenv("OPENAI_API_KEY"))
        .modelName("gpt-4o-mini")
        .temperature(0.2)
        .timeout(Duration.ofSeconds(60))
        .maxRetries(3)
        .build();
Anthropic Claude
ChatModel claude = AnthropicChatModel.builder()
        .apiKey(System.getenv("ANTHROPIC_API_KEY"))
        .modelName("claude-sonnet-4-latest")
        .temperature(0.2)
        .timeout(Duration.ofSeconds(60))
        .build();
Google Gemini
ChatModel gemini = GoogleAiGeminiChatModel.builder()
        .apiKey(System.getenv("GOOGLE_AI_API_KEY"))
        .modelName("gemini-2.0-flash")
        .build();
Ollama (local)
ChatModel local = OllamaChatModel.builder()
        .baseUrl("http://localhost:11434")
        .modelName("llama3.2")
        .timeout(Duration.ofSeconds(120))   // local models can be slower
        .build();

Each needs its own Maven module (langchain4j-open-ai, langchain4j-anthropic, and so on). The interface they all satisfy is the same.

Swapping providers is a construction change

// Your service depends on ChatModel, not on any provider.
public class Assistant {
    private final ChatModel model;
    public Assistant(ChatModel model) { this.model = model; }
    // ... uses model
}
 
// Wire whichever provider you want. Switching is a one-line edit here.
Assistant assistant = new Assistant(openai);   // or claude, or local

Streaming

For interactive UIs, use StreamingChatModel and handle tokens as they arrive:

Streaming with a handler
StreamingChatModel streaming = OpenAiStreamingChatModel.builder()
        .apiKey(System.getenv("OPENAI_API_KEY"))
        .modelName("gpt-4o-mini")
        .build();
 
streaming.chat("Explain virtual threads", new StreamingChatResponseHandler() {
    @Override
    public void onPartialResponse(String token) {
        // Forward each token to the client (e.g. over SSE) as it arrives.
        emitter.send(token);
    }
 
    @Override
    public void onCompleteResponse(ChatResponse response) {
        emitter.complete();
    }
 
    @Override
    public void onError(Throwable error) {
        emitter.completeWithError(error);
    }
});

The declarative AiServices style can also return a TokenStream from an interface method, which is usually cleaner — covered in the architecture tutorial.

Resilience: timeouts and retries

ChatModel resilient = OpenAiChatModel.builder()
        .apiKey(System.getenv("OPENAI_API_KEY"))
        .modelName("gpt-4o-mini")
        // A hung provider must not hold a thread forever.
        .timeout(Duration.ofSeconds(60))
        // Retries transient 429/5xx with backoff; not 4xx client errors.
        .maxRetries(3)
        .build();

Model options that matter

OpenAiChatModel.builder()
        .modelName("gpt-4o-mini")
        .temperature(0.0)          // deterministic for extraction/classification
        .maxTokens(1000)           // caps response length and cost
        .topP(1.0)
        .build();

Temperature is the option you will tune most. Near zero for structured, deterministic tasks; higher only where variety is the goal.

Logging requests during development

OpenAiChatModel.builder()
        .apiKey(System.getenv("OPENAI_API_KEY"))
        .modelName("gpt-4o-mini")
        .logRequests(true)     // dev only — requests may contain personal data
        .logResponses(true)
        .build();

Turn these off in production; request logs carry prompt content, which is often personal data. See security in AI-powered Spring applications.

Choosing a model

  • Development / high volume / simple tasks — a small hosted model or local Ollama.
  • General production — a mid-tier model like gpt-4o-mini.
  • Hard reasoning / code analysis — a frontier model, and measure whether the quality gain justifies the cost on your inputs.

Route between them per task rather than picking one for everything — see agent frameworks compared and small language models.

Next

Frequently Asked Questions

How do I switch chat model providers in LangChain4j?
Build a different ChatModel implementation — OpenAiChatModel, AnthropicChatModel, GoogleAiGeminiChatModel, OllamaChatModel — and pass it where you built the old one. Because your AI Services and application code depend on the ChatModel interface, nothing else changes. Provider choice is a single construction site.
What is the difference between ChatModel and StreamingChatModel?
ChatModel returns the complete response in one call. StreamingChatModel emits tokens as they are generated, through a handler, for interactive UIs. In LangChain4j 1.0 these are the current names; older code uses ChatLanguageModel and StreamingChatLanguageModel.
How do I set a timeout and retries in LangChain4j?
Both are builder options on the model: .timeout(Duration) and .maxRetries(int). Always set a timeout — a model call with no timeout can hang a request thread indefinitely. Retries handle transient failures with backoff.
Can I use Azure OpenAI or Amazon Bedrock with LangChain4j?
Yes. LangChain4j has dedicated modules for Azure OpenAI, Amazon Bedrock, Google Vertex AI and many others. Each provides a ChatModel implementation configured with that provider's credentials, and your downstream code is unaffected by the choice.

Related tutorials