Small Language Models (SLMs)

The instinct to reach for the biggest, most capable model is often wrong. For a large share of real tasks — classification, routing, extraction, straightforward answering — a small model is more than good enough, and dramatically cheaper and faster. This tutorial makes the case for small models, and for routing between small and large based on what each request actually needs.

The case against always going big#

Frontier models are expensive and slower. A lot of production AI work does not need them:

• Classification — "is this billing, technical, or sales?" A small model nails it.
• Routing — deciding which handler or tool applies.
• Simple extraction — pulling clear fields from text.
• Straightforward RAG answering — when the retrieved context makes the answer obvious.

Paying frontier prices and latency for these is waste. A small model does them at a fraction of the cost, often faster, and — for high-volume tasks — the savings are enormous.

Model routing#

Route each request to the cheapest model that can handle it:

public String handle(Request request) {
    // A cheap classifier decides the difficulty. Often a small model, or even
    // rules, can make this call.
    Difficulty difficulty = router.classify(request);
 
    ChatModel model = switch (difficulty) {
        case SIMPLE -> smallModel;    // classification, routing, easy extraction
        case MODERATE -> midModel;    // general tasks
        case HARD -> largeModel;      // deep reasoning, complex analysis
    };
 
    return process(request, model);
}

The classifier's cost must be less than the savings — often it is a small model or even simple heuristics on the request. The chains tutorial covers the routing mechanism.

Running small models#

Small models run locally via Ollama, removing per-call cost and network latency entirely:

spring:
  ai:
    ollama:
      chat:
        options:
          model: phi3      # or gemma2, or a small Llama/Mistral

For the highest volume, a small model served on your own hardware behind vLLM is both cheaper and faster than hosted frontier calls.

On-device and edge AI#

Quantized small models run on phones and edge hardware, enabling AI with properties a hosted API cannot offer:

• No network — works offline.
• No per-call cost — free after the model ships.
• Strong privacy — data never leaves the device, which matters for sensitive personal use.
• Low latency — no round trip.

The trade-off is capability and battery, but for focused on-device tasks — a local classifier, a simple assistant, on-device extraction — it is increasingly practical and sidesteps entire categories of cost and privacy concern.

The quality question#

The honest caveat: small models have a lower ceiling. They struggle with multi-step reasoning, nuanced instruction-following and hard problems. The engineering discipline is to know which of your tasks fall below that ceiling — and to measure, not assume:

// Do not guess whether a small model is good enough — measure it on your task.
// Run your evaluation set through both and compare accuracy against cost.
double smallAccuracy = evaluate(smallModel, taskSuite);
double largeAccuracy = evaluate(largeModel, taskSuite);
// If small is 95% as good at 10% of the cost, the routing decision is clear.

See LLM evaluation & benchmarks.

A pragmatic architecture#

The pattern many mature systems converge on:

1. A cheap classifier or router decides difficulty.
2. Small models handle the high-volume easy majority — often on your own hardware.
3. A large model handles the genuinely hard minority.
4. Measurement continuously validates that the small models are good enough for what they are given.

This delivers most of the quality of a frontier-everywhere approach at a fraction of the cost and latency — and it is invisible to users, who just get fast, accurate responses.

Next#

• Enterprise AI agents
• Open source vs proprietary AI strategy
• Low-latency LLM serving