Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Low-Latency LLM Serving

Serve LLMs with low latency: time to first token, streaming, continuous batching, vLLM and TGI, speculative decoding, and the latency levers available whether you self-host or use an API.

Expert4 min readUpdated
On this page

Latency shapes whether an AI feature feels responsive or broken. The levers differ depending on whether you self-host models (where continuous batching and engines like vLLM matter) or call an API (where streaming, model choice and caching are your tools). This tutorial covers both, and the metric that matters most: time to first token.

Key Takeaways

  • Time to first token (TTFT) matters more than total time for interactive features — streaming hides the rest.
  • Self-hosting: continuous batching and engines like vLLM are what make serving efficient.
  • Hosted API: stream, choose a faster model, cache, shorten prompts, run concurrently.
  • Optimise perceived latency first — it is often the biggest, cheapest win.

Time to first token is the metric

For anything a user watches, TTFT — how long until the first token appears — dominates perceived responsiveness. A three-second total generation feels instant if the first token arrives in 300ms and streams; it feels broken if the user stares at a spinner for three seconds then gets everything at once.

// Streaming turns total latency into TTFT. The user reads as it generates.
return chatClient.prompt().user(question).stream().content();

Levers when you use a hosted API

You cannot change the provider's serving, but you have several tools:

  • Stream — hide latency behind TTFT (above).
  • Smaller/faster model — a faster model where quality allows; route hard cases to a slower one. See small language models.
  • Cache — repetitive responses served from a cache skip the model entirely.
  • Shorten prompts — less input to process means faster TTFT; trim context to what is needed.
  • Cap output length — a bounded maxTokens finishes sooner.
  • Concurrency — run independent calls in parallel on virtual threads, not sequentially.
Concurrent calls hide sequential latency
// Three independent calls sequentially = sum of latencies. Concurrently = the
// slowest one. On Java 21, virtual threads make this cheap.
try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
    var a = executor.submit(() -> callA());
    var b = executor.submit(() -> callB());
    var c = executor.submit(() -> callC());
    return combine(a.get(), b.get(), c.get());
}

Levers when you self-host

Self-hosting gives you control over serving, where the big throughput wins live.

Continuous batching

Serving requests one at a time wastes the GPU. Continuous batching processes many together, adding and removing requests dynamically as they arrive and finish:

Continuous batching keeps the GPU busy by dynamically batching concurrent requests, rather than serving one at a time.

This is not something you implement — it is why you run a serving engine.

Serving engines: vLLM and TGI

Do not serve a raw model. Run an engine built for it:

  • vLLM — high throughput via continuous batching and PagedAttention (efficient KV-cache memory). The common choice for self-hosted open-weight models.
  • TGI (Text Generation Inference) — Hugging Face's production serving engine, similar goals.
# Point Spring AI at a vLLM server via its OpenAI-compatible endpoint.
spring:
  ai:
    openai:
      base-url: http://vllm-server:8000
      chat:
        options:
          model: your-served-model

Because these expose OpenAI-compatible APIs, your Spring AI code is unchanged — self-hosting behind vLLM is a base-URL change. See Spring AI with Ollama.

Speculative decoding

An advanced technique where a small fast model drafts several tokens and the large model verifies them in one pass, accepting the correct ones — reducing the number of expensive large-model steps. Engines support it; it can meaningfully cut latency for the same quality. It is a serving-engine feature, not something you build.

Quantization for speed

A quantized model is smaller and faster to run, trading a little accuracy for lower latency and higher throughput. On your own hardware, this is a direct latency lever.

Measuring latency properly

Track the right numbers, split correctly:

// TTFT and total generation, separately. And split retrieval from generation
// for RAG — a slow total could be either.
metrics.timer("ai.ttft").record(timeToFirstToken);
metrics.timer("ai.generation").record(totalGenerationTime);
metrics.timer("ai.retrieval").record(retrievalTime);

P95 and P99 matter more than the average — the tail is what frustrates users. See AI observability & LLM tracing.

A latency optimization order

  1. Stream — hide latency behind TTFT. Do this first, always.
  2. Cache repetitive requests.
  3. Right-size the model — fastest that meets quality.
  4. Trim prompts and cap output.
  5. Parallelise independent calls.
  6. Self-host behind vLLM/TGI — only if you have the ops capacity and a reason to self-host.

Most teams get most of the win from the first five, without ever self-hosting.

Next

You have completed Phase 5 — the patterns for shipping AI in real systems.

Frequently Asked Questions

What is time to first token and why does it matter?
Time to first token (TTFT) is how long the model takes to emit its first token. For interactive features it matters more than total generation time, because streaming lets the user start reading immediately — a fast TTFT feels responsive even if the full answer takes several seconds. Optimising TTFT is often the biggest perceived-latency win.
What is continuous batching?
A serving technique where the model processes multiple requests together and dynamically adds and removes requests from the batch as they arrive and finish, rather than waiting for a fixed batch. It dramatically improves throughput and GPU utilisation when self-hosting, letting one GPU serve many concurrent requests efficiently. vLLM popularised it.
What is vLLM?
A high-throughput LLM serving engine that uses continuous batching and efficient memory management (PagedAttention) to serve open-weight models with high throughput and low latency. If you self-host models, vLLM and similar engines like TGI are what you run in front of them, rather than serving the raw model, because they handle batching and memory far better.
How do I reduce LLM latency if I use a hosted API?
You cannot change the provider's serving, but you can stream to hide latency, use a smaller or faster model where quality allows, cache repetitive responses, shorten prompts and cap output length, and run calls concurrently rather than sequentially. Perceived latency — especially via streaming — is often what you can most improve.

Related tutorials