Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Multi-Tenant AI Architectures

Build multi-tenant AI systems in Java: strict tenant isolation in retrieval, per-tenant quotas and rate limits, cost allocation, and data residency — keeping tenants apart safely at scale.

Expert5 min readUpdated
On this page

Serving multiple tenants from one AI system concentrates the risks: a retrieval bug leaks one tenant's data to another, one tenant's usage exhausts shared capacity, and costs blur across tenants. This tutorial covers the isolation, quota and cost-allocation patterns that make multi-tenant AI safe at scale.

Key Takeaways

  • Isolation is structural, in code — filter every access by an authenticated tenant ID.
  • Never rely on the model or a prompt to keep tenants apart.
  • Per-tenant quotas and rate limits prevent the noisy-neighbour problem.
  • Tag cost by tenant — you cannot bill or spot abuse without it.

Tenant isolation: the non-negotiable

Every data access is scoped by tenant, and the tenant comes from the authenticated session — never from the request:

Enforced isolation
public RagAnswer answer(String question) {
    // Tenant ID from the security context. If a client could supply it, a
    // client could read another tenant's data.
    String tenantId = SecurityContextHolder.getContext().getAuthentication()
            .getTenantId();
 
    List<Document> context = vectorStore.similaritySearch(SearchRequest.builder()
            .query(question)
            .topK(5)
            // The filter is what makes isolation structural — another tenant's
            // vectors are never even retrieved, so they cannot leak.
            .filterExpression("tenantId == '%s'".formatted(tenantId))
            .build());
 
    return generate(question, context);
}

Shared vs isolated stores

Shared store with filtering scales and simplifies operations; separate stores give stronger isolation at higher cost.
ApproachIsolationOperationsBest for
Shared store, metadata filterStrong if filtering is rigorousSimpleMost cases, many tenants
Store per tenantStrongestMore complexHigh-security, data residency
Hybrid (shared + isolated for premium)TieredModerateMixed requirements

The shared approach scales to many tenants and is simpler to operate — provided the filtering is enforced without exception. The isolated approach suits strict security or data-residency requirements.

Per-tenant quotas and rate limits

Without limits, one tenant's spike consumes the shared model rate limit and budget, degrading everyone — the noisy-neighbour problem:

Per-tenant limits
public Response handle(Request request, String tenantId) {
    // Enforce the tenant's rate limit before the expensive call.
    if (!rateLimiters.forTenant(tenantId).tryAcquire()) {
        return Response.rateLimited("tenant rate limit exceeded");
    }
    // And their quota (e.g. monthly token budget).
    if (quotas.forTenant(tenantId).isExhausted()) {
        return Response.quotaExceeded("tenant quota exhausted");
    }
    return process(request, tenantId);
}

Limits also contain abuse: a compromised or malicious tenant account cannot run up an unbounded bill or starve others.

Cost allocation

Tag every cost metric with the tenant, so cost rolls up per tenant:

// Every token counter is tagged by tenant. Now "what does tenant Acme cost?"
// and "which tenant is driving the bill?" both have answers.
registry.counter("ai.cost.usd", "tenant", tenantId, "model", model)
        .increment(cost);

This enables usage-based billing, surfaces the tenants driving your costs, and flags anomalies (a tenant whose usage suddenly spikes may be abusing the system or hitting a bug). See Spring AI observability.

Per-tenant configuration

Tenants often need different behaviour — a different model tier, a different system prompt, different data sources:

public ChatClient forTenant(String tenantId) {
    TenantConfig config = tenantConfigs.get(tenantId);
    // Premium tenants might get a stronger model; each gets its own prompt and
    // retrieval scope. Cached per tenant to avoid rebuilding on every request.
    return clientCache.computeIfAbsent(tenantId, id -> builder
            .defaultSystem(config.systemPrompt())
            .defaultOptions(ChatOptions.builder().model(config.model()).build())
            .build());
}

Data residency

Some tenants require their data to stay in a specific region — a legal requirement, not a preference. This shapes architecture: region-specific stores and model endpoints, with routing by tenant.

// Route a tenant's requests to their required region's model endpoint and
// store, so their data never leaves the jurisdiction they require.
Region region = tenantConfigs.get(tenantId).dataRegion();
return regionalClients.get(region).process(request);

See GenAI on AWS, Azure & GCP and AI regulations & compliance.

The multi-tenant checklist

  • Tenant ID from the authenticated session, never the request
  • Every data access filtered by tenant, enforced in code
  • Isolation tested — a test that asserts tenant A cannot retrieve tenant B's data
  • Per-tenant rate limits and quotas before the model call
  • Cost tagged and tracked per tenant
  • Per-tenant configuration where needed
  • Data residency honored where required

Next

Frequently Asked Questions

How do I keep tenants isolated in a shared AI system?
Enforce tenant scope in code at every data access — filter every vector search and query by a tenant ID derived from the authenticated principal, never from a request parameter. Never rely on the model or a prompt instruction to keep tenants apart. The isolation must be structural: another tenant's data is never even retrieved, so it cannot leak into a prompt or response.
How do I allocate AI costs across tenants?
Tag every token metric with the tenant ID, so cost rolls up per tenant. Since model calls are the dominant cost and vary hugely by tenant usage, per-tenant cost tracking is essential for billing, for spotting abuse, and for enforcing quotas. Without it, you cannot tell which tenant is driving your bill.
Should each tenant have a separate vector store or a shared one?
A shared store with strict metadata filtering scales better and is simpler to operate for most cases, provided the filtering is enforced rigorously in code. Separate stores per tenant give stronger isolation and suit high-security or data-residency requirements, at higher operational cost. Choose based on your isolation requirements and tenant count.
How do I stop one tenant exhausting shared AI capacity?
Per-tenant rate limits and quotas, enforced before the model call. Without them, one heavy or abusive tenant can consume the shared rate limit and budget, degrading service for everyone — the noisy-neighbour problem. Bound each tenant's consumption so their usage cannot starve others.

Related tutorials