Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

LangChain4j Chat Memory

Add conversation memory to LangChain4j AI Services: message and token windows, per-user memory with @MemoryId, persistent stores, and why unbounded memory breaks in production.

Intermediate4 min readUpdated
On this page

A language model has no memory of your last message. Every call is the first call unless you send the history yourself. LangChain4j manages that for you through ChatMemory, but the details — bounding it, keying it per user, persisting it — are where production correctness lives.

Key Takeaways

  • The model is stateless; memory means resending prior messages each call.
  • Always bound memory — a window by messages or tokens. Unbounded memory breaks and overcharges.
  • Key memory per user with @MemoryId, derived from the authenticated principal.
  • The default store is in-memory — use a persistent store for anything replicated.

Basic memory

A remembered conversation
interface Assistant {
    String chat(String message);
}
 
Assistant assistant = AiServices.builder(Assistant.class)
        .chatModel(chatModel)
        // Keep the last 20 messages. The model now "remembers" within the window.
        .chatMemory(MessageWindowChatMemory.withMaxMessages(20))
        .build();
 
assistant.chat("My name is Sam.");
assistant.chat("What is my name?");   // "Your name is Sam." — because of memory

Without the memory, the second call would have no idea. With it, LangChain4j prepends the prior messages before each call.

Why bounding is mandatory

Because every call resends the whole conversation, an unbounded memory has two failure modes that both worsen over time:

  1. The prompt grows until it exceeds the context window and requests start failing — for the users with the longest conversations, which are often your most engaged ones.
  2. Cost grows on every turn, because you pay for the entire accumulated history on each call, not just the new message.

Message window vs token window

// Message window: keep the last N messages. Simple, predictable.
ChatMemory byMessages = MessageWindowChatMemory.withMaxMessages(20);
 
// Token window: keep as many recent messages as fit in a token budget.
// More precise control over cost and context, needs a tokenizer.
ChatMemory byTokens = TokenWindowChatMemory.withMaxTokens(2000, new OpenAiTokenizer());

Use a message window by default. Switch to a token window when message lengths vary widely — a conversation mixing one-word replies with pasted documents is better bounded by tokens than by count.

Per-user memory

A shared memory would mix everyone's conversations together. @MemoryId keeps them separate:

Per-user memory
interface Assistant {
    // The @MemoryId argument selects which conversation's memory to use.
    String chat(@MemoryId String userId, @UserMessage String message);
}
 
Assistant assistant = AiServices.builder(Assistant.class)
        .chatModel(chatModel)
        // A separate bounded memory per user.
        .chatMemoryProvider(userId -> MessageWindowChatMemory.withMaxMessages(20))
        .build();
 
assistant.chat("user-42", "Hello");   // user-42's memory
assistant.chat("user-99", "Hi");      // user-99's memory, entirely separate

Persistent memory

The default ChatMemoryStore is in-memory: lost on restart, not shared across instances. In a replicated deployment, a user's second message may land on a pod that never heard their first. Back memory with a persistent store:

Persistent memory store
// Implement ChatMemoryStore against your database or Redis.
class JdbcChatMemoryStore implements ChatMemoryStore {
 
    @Override
    public List<ChatMessage> getMessages(Object memoryId) {
        return repository.loadMessages(memoryId.toString());
    }
 
    @Override
    public void updateMessages(Object memoryId, List<ChatMessage> messages) {
        repository.saveMessages(memoryId.toString(), messages);
    }
 
    @Override
    public void deleteMessages(Object memoryId) {
        repository.deleteMessages(memoryId.toString());
    }
}
ChatMemoryProvider provider = userId -> MessageWindowChatMemory.builder()
        .id(userId)
        .maxMessages(20)
        .chatMemoryStore(new JdbcChatMemoryStore(repository))
        .build();

Summarising older messages

When you need long-range memory without an unbounded prompt, summarise the messages that fall out of the window instead of dropping them:

// Conceptually: as messages age out of the window, replace them with a running
// summary, so the agent retains the gist of a long conversation cheaply.
// This trades some fidelity for a bounded, affordable context.

This "summarisation memory" is the standard technique for assistants that must recall a long history — covered further in memory systems for agents.

Memory and RAG are different things

A common confusion: chat memory holds the conversation; RAG retrieves knowledge. Memory answers "what did we just discuss?"; retrieval answers "what does the handbook say?". A capable assistant uses both — see LangChain4j agents and tools for combining them.

Next

Frequently Asked Questions

How do I add memory to a LangChain4j AI Service?
Provide a ChatMemory or a ChatMemoryProvider when building the AI Service. A single ChatMemory suits a single conversation; a ChatMemoryProvider keyed by a @MemoryId parameter gives each user their own bounded memory. The memory loads prior messages before each call and stores the exchange afterwards, automatically.
Why must chat memory be bounded?
A language model is stateless, so every call resends the whole conversation. Unbounded memory means the prompt grows on every turn until it exceeds the context window and starts failing — and you pay for the entire history on every single call. A message or token window caps this, keeping cost and context predictable.
What is the difference between a message window and a token window?
A message window keeps the last N messages regardless of length — simple and predictable. A token window keeps as many recent messages as fit within a token budget, which controls cost and context usage more precisely but needs a tokenizer. Use a message window for simplicity, a token window when message lengths vary a lot.
How do I persist chat memory across restarts?
Back the memory with a persistent ChatMemoryStore — a JDBC or Redis implementation — instead of the default in-memory one. The in-memory store is lost on restart and not shared across instances, so a replicated deployment needs a shared persistent store or users lose context mid-conversation.

Related tutorials