AI Caching Strategies

Model calls are the dominant cost and latency in an AI application, and a lot of traffic is repetitive. Caching — from simple exact-match to embedding-based semantic caching — can cut both dramatically. This tutorial covers the strategies, with the honest caveat that semantic caching trades a little accuracy for the savings.

Exact-match caching#

The simplest and safest: cache the response for an identical request. Same prompt, same parameters, same answer.

public String ask(String question, ChatOptions options) {
    // Key on the full request — prompt and parameters — so a cached answer is
    // genuinely for the same call.
    String key = cacheKey(question, options);
    String cached = redis.get(key);
    if (cached != null) {
        return cached;   // no model call, no cost, sub-millisecond
    }
    String answer = chatClient.prompt().user(question).options(options).call().content();
    redis.set(key, answer, Duration.ofHours(24));
    return answer;
}

Exact-match is safe because an identical request genuinely has the same answer (at temperature 0). It only misses when wording differs — which is where semantic caching comes in.

Semantic caching#

Users phrase the same question many ways. Semantic caching matches by meaning:

public String askCached(String question) {
    float[] queryVector = embeddingModel.embed(question);
 
    // Look for a cached query similar enough to reuse its answer.
    var hit = semanticCache.search(SearchRequest.builder()
            .query(question)
            .topK(1)
            // Conservative threshold — a near-miss returns a wrong answer.
            .similarityThreshold(0.95)
            .build());
 
    if (!hit.isEmpty()) {
        return hit.get(0).getMetadata().get("answer").toString();   // cache hit
    }
 
    String answer = chatClient.prompt().user(question).call().content();
    // Store the question (embedded) and its answer for future similar queries.
    semanticCache.add(question, answer);
    return answer;
}

Provider prompt caching#

Distinct from caching responses: some providers cache the processing of a repeated prompt prefix. If every request shares a long system prompt or a large fixed context, structuring the prompt so the stable part comes first lets the provider reuse its processing:

// Put the stable, reusable content first (system prompt, fixed context), and
// the variable content last. Providers that support prefix caching then make
// repeated calls cheaper and faster on the shared prefix.
String prompt = STABLE_SYSTEM_PROMPT + STABLE_CONTEXT + variableUserQuery;

This is especially valuable for RAG or agents with a large fixed instruction set repeated on every call. It reduces cost without any risk of a wrong answer, because it caches computation, not results.

What to cache and what not to#

Do not cache:

• Personalized responses — a cache keyed only on the question serves one user's answer to another. Include the user or tenant in the key, or do not cache.
• Time-sensitive answers — "what is today's status?" cached for a day is wrong for 23 hours.
• Anything with tenant-scoped data — a cross-tenant cache hit is a data leak.

Invalidation#

The hard part of caching, doubly so here. When the underlying knowledge changes, cached answers based on it are stale:

// When source documents change, invalidate cached answers derived from them.
// Tag cache entries with the sources they used, and invalidate by source.
public void onDocumentUpdated(String sourceId) {
    cache.invalidateByTag("source:" + sourceId);
}

For RAG answers, tag each cached answer with the sources it drew on, so updating a document invalidates exactly the answers that depended on it.

Measuring cache effectiveness#

Track hit rate and savings, and watch for quality problems:

// Hit rate tells you the savings; sampled quality checks on cache hits tell you
// whether semantic caching is serving near-misses.
metrics.counter("ai.cache", "result", hit ? "hit" : "miss").increment();

A high hit rate is only good if the hits are correct. For semantic caching, sample cache hits and verify the returned answer actually fits the query — a hit rate that comes from serving near-misses is worse than no cache. See LLM evaluation & benchmarks.

Next#

• Low-latency LLM serving
• Tokenization & context windows
• Multi-tenant AI architectures