Tokenization & Context Windows

Tokens are the currency of LLMs — the unit models read, generate and bill for. Understanding tokenization and the context window explains cost, informs how you budget a prompt, and prevents a class of production failures. This tutorial covers both for a working engineer.

What a token is#

Tokenizers split text into sub-word units using an algorithm like Byte-Pair Encoding (BPE), which learns common character sequences from training data. Frequent words become single tokens; rare words split into pieces.

"tokenization"  → ["token", "ization"]     (2 tokens)
"the"           → ["the"]                    (1 token)
"antidisestablishmentarianism" → many tokens (rare, fragments heavily)

A rough rule for English prose: 1 token ≈ 0.75 words, or about 4 characters. But it is only a rule of thumb — the real count depends on the specific text and tokenizer.

Why code costs more tokens#

Tokenizers are trained predominantly on natural language, so code fragments more:

// This innocuous line is more tokens than its length suggests:
Map<String, List<Integer>> resultsByCategory = new HashMap<>();
// The generics, camelCase, and punctuation each split into multiple tokens.

Counting tokens accurately#

Do not guess when it matters — count with the model's tokenizer:

Tokenizer tokenizer = new OpenAiTokenizer("gpt-4o-mini");
int promptTokens = tokenizer.estimateTokenCountInText(prompt);
 
// Use this to keep prompts within budget and to predict cost before the call.
if (promptTokens > MAX_PROMPT_TOKENS) {
    prompt = truncateToTokens(prompt, MAX_PROMPT_TOKENS, tokenizer);
}

The context window is a shared budget#

The context window is the total tokens the model can consider at once, and everything shares it:

If their sum exceeds the window, you get truncation or an error. Budget each component:

• System prompt — fixed; keep it focused.
• History — bound the chat memory.
• Retrieved documents — limit top-k and chunk size.
• Response — reserve space with a max-tokens cap.

Long context is not a dumping ground#

Models now offer very large context windows, which tempts you to stuff everything in. Resist it, for three reasons: cost and latency both rise with context length, and models attend less reliably to information buried in the middle of a very long prompt (the "lost in the middle" effect). Retrieving the relevant few thousand tokens beats sending a hundred thousand — see transformer architecture for why.

The cost math#

Cost is per token, with input and output priced separately:

// Prices per 1M tokens, input and output differ. Keep them in config.
double cost = (promptTokens / 1_000_000.0) * inputPricePerM
            + (completionTokens / 1_000_000.0) * outputPricePerM;

A key insight: input tokens often dominate. A long system prompt, accumulated history, and retrieved context are sent on every call, while the response is usually shorter. This is why unbounded chat memory is a cost problem, not just a context one — you pay for the whole history every turn. Track token cost per feature to see where the money goes; see Spring AI observability.

Practical token discipline#

• Count tokens with the real tokenizer when budgets are tight.
• Bound history, retrieval and response length so their sum fits comfortably.
• Prefer focused retrieval over long context — cheaper, faster, more reliable.
• Track cost per feature; input tokens are usually the larger line.
• Remember code and non-English text cost more than their length suggests.

Next#

• LLM evaluation & benchmarks
• AI caching strategies
• Low-latency LLM serving