Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Memory Systems for Agents

How agent memory works beyond a chat window: working, episodic and semantic memory, vector-based recall, memory consolidation, and implementing persistent agent memory in Java.

Advanced4 min readUpdated
On this page

Chat memory — a window of recent messages — is only the working memory of an agent. A genuinely capable agent also remembers past sessions and accumulates knowledge, which needs a richer memory architecture. This tutorial covers the three memory types and how to implement long-term memory in Java.

Key Takeaways

  • Working memory = the bounded chat window (current task context).
  • Episodic memory = a recallable record of past interactions and events.
  • Semantic memory = learned facts, retrieved by relevance via vector search.
  • Consolidation keeps long-term memory bounded and surfaces what matters.

The three memory types

Working, episodic and semantic memory — the immediate context, the recallable past, and accumulated knowledge.

Working memory is the chat window you already know. Episodic and semantic memory are the additions that make an agent feel like it remembers.

Semantic memory: facts by relevance

Store facts as embedded records; retrieve the relevant ones for the current context. This is RAG pointed at the agent's own knowledge:

Semantic memory
class SemanticMemory {
 
    private final EmbeddingStore<TextSegment> store;
    private final EmbeddingModel embeddings;
 
    // Remember a fact the agent learned.
    public void remember(String fact, Map<String, Object> metadata) {
        var segment = TextSegment.from(fact, Metadata.from(metadata));
        store.add(embeddings.embed(segment).content(), segment);
    }
 
    // Recall facts relevant to the current situation.
    public List<String> recall(String context, int limit) {
        return store.search(EmbeddingSearchRequest.builder()
                        .queryEmbedding(embeddings.embed(context).content())
                        .maxResults(limit)
                        .minScore(0.7)
                        .build())
                .matches().stream()
                .map(m -> m.embedded().text())
                .toList();
    }
}

Episodic memory: recalling past events

Episodic memory records what happened — past conversations, decisions, outcomes — so the agent can learn from experience:

Episodic memory
record Episode(String summary, Instant when, String outcome, Map<String, Object> context) {}
 
class EpisodicMemory {
 
    // After a session, store a summary of what happened.
    public void record(Episode episode) {
        String text = "%s (outcome: %s)".formatted(episode.summary(), episode.outcome());
        var segment = TextSegment.from(text, Metadata.from(Map.of(
                "when", episode.when().toString(),
                "outcome", episode.outcome())));
        store.add(embeddings.embed(segment).content(), segment);
    }
 
    // Before a new task, recall similar past episodes.
    public List<Episode> recallSimilar(String currentTask, int limit) {
        // Retrieve past episodes resembling the current situation, so the agent
        // can apply what worked (or avoid what did not) last time.
        return search(currentTask, limit);
    }
}

Assembling context from memory

At the start of a task, the agent gathers relevant memories and composes its context:

Context assembly
public String buildContext(String task, String conversationId) {
    // 1. Working memory: the current conversation.
    List<ChatMessage> recent = workingMemory.get(conversationId);
 
    // 2. Semantic memory: relevant facts.
    List<String> facts = semanticMemory.recall(task, 5);
 
    // 3. Episodic memory: similar past experiences.
    List<Episode> episodes = episodicMemory.recallSimilar(task, 3);
 
    // Compose into the prompt, clearly labelled by type so the model knows what
    // is current, what is known, and what is precedent.
    return composePrompt(task, recent, facts, episodes);
}

Consolidation: keeping memory bounded

Without maintenance, memory grows until retrieval degrades and cost climbs. Consolidation periodically distils detailed memories into compact ones:

// Periodically: cluster related episodes, summarise each cluster into a single
// durable memory, and archive the originals. Keeps the store lean and surfaces
// patterns over individual events.
public void consolidate() {
    List<List<Episode>> clusters = clusterBySimilarity(allRecentEpisodes());
    for (var cluster : clusters) {
        String lesson = summariser.distil(cluster);   // "X approach works for Y tasks"
        semanticMemory.remember(lesson, Map.of("type", "consolidated"));
        episodicMemory.archive(cluster);
    }
}

Privacy and memory

Long-term memory stores what users told the agent, sometimes indefinitely. That is personal data with all the obligations that implies: a retention policy, deletion on request, and access controls. Do not build a memory system that silently accumulates personal data forever — see ethical AI & responsible agent design and AI regulations & compliance.

Next

Frequently Asked Questions

What are the types of agent memory?
Working memory holds the current task context — the conversation and recent tool results. Episodic memory records past interactions and events the agent can recall later. Semantic memory holds learned facts and knowledge, usually retrieved through vector search. A capable agent combines all three: the immediate context, a history it can look back on, and a knowledge base it can query.
How is agent memory different from chat memory?
Chat memory is a bounded window of recent messages — the working memory. Agent memory systems add long-term components: episodic memory of past sessions and semantic memory of facts, both typically stored in a vector database and retrieved by relevance rather than recency. Chat memory answers "what did we just say"; long-term memory answers "what happened last week" and "what do I know".
How do I give an agent long-term memory in Java?
Store memories as embedded records in a vector store, then retrieve the most relevant ones for the current context and inject them into the prompt — the same mechanism as RAG, applied to the agent's own history. Consolidate or summarise old memories periodically so the store does not grow without bound.
What is memory consolidation?
Periodically summarising or distilling detailed memories into more compact, durable ones — much as human memory consolidates experiences during sleep. It keeps the memory store manageable and surfaces the important patterns, at the cost of losing fine detail. Without consolidation, an agent's memory grows until retrieval quality and cost both suffer.

Related tutorials