Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Transformer Architecture Explained

The transformer architecture explained for engineers, not researchers: self-attention, multi-head attention, positional encoding and why it explains context limits, token cost and hallucination.

Advanced4 min readUpdated
On this page

You do not need to implement a transformer to build with one, but understanding the architecture explains behaviours that otherwise seem arbitrary — why context is limited, why tokens cost what they do, why the model hallucinates. This tutorial explains it at the level an engineer needs.

Key Takeaways

  • A transformer predicts the next token, repeatedly — that objective explains most of its behaviour.
  • Self-attention lets each token weigh every other; it is the core mechanism.
  • Attention is quadratic in sequence length — the reason context windows are bounded and long context is expensive.
  • Hallucination falls out of the "predict the plausible next token" objective, not a bug.

The one-sentence model

A transformer takes a sequence of tokens and predicts the next one. Generation is that, repeated: pick the next token, append it, predict again. Everything else is machinery to make that prediction good.

A transformer: tokens become embeddings, pass through attention and feed-forward layers, and the model predicts the next token — then repeats.

Tokens and embeddings

Text is first split into tokens — roughly word-pieces. Each token becomes a vector (an embedding) that encodes its meaning. Because the model processes all tokens at once with no inherent notion of order, a positional encoding is added so the model knows each token's place in the sequence — otherwise "dog bites man" and "man bites dog" would look identical.

Self-attention: the core idea

Self-attention is how each token gathers relevant information from the rest of the sequence. For every token, the model computes how much every other token matters to it, then builds a new representation as a weighted blend.

Concretely, each token produces three vectors — a query (what it is looking for), a key (what it offers), and a value (what it contributes). A token's attention to another is how well its query matches the other's key; the result is a weighted sum of values.

This is what lets the model connect "it" to the noun it refers to three sentences back, or a question to the fact that answers it earlier in the prompt. The connections are learned, not programmed.

Multi-head attention

Rather than one attention computation, transformers run several in parallel — "heads" — each learning to attend to different relationships. One head might track grammatical subject-verb links, another long-range topical connections. Their outputs combine, giving the model multiple simultaneous views of how tokens relate.

Why context length is limited

Self-attention compares every token to every other token. For a sequence of n tokens, that is n² comparisons. Double the context and you quadruple the attention cost. This quadratic scaling, plus the memory it needs, is the practical reason context windows are bounded — and why a model offering very long context (hundreds of thousands or millions of tokens) is doing real architectural work to make it affordable.

The KV cache

During generation, the keys and values for tokens already processed do not change, so they are cached rather than recomputed each step. This KV cache is why generation speeds up after the first token, and why memory use grows with context length — an important consideration for low-latency serving.

Why hallucination is intrinsic

The model is trained to predict the most probable next token given everything so far. It is not trained to only say true things — it has no separate notion of truth. When it lacks the relevant information, it does not stop; it produces the most plausible continuation, which is often fluent and wrong.

This is why hallucination cannot be fully solved by prompting. The fixes work around the objective: grounding the model in retrieved facts so the plausible continuation is also the correct one, and validating output so wrong answers are caught. See building a RAG pipeline and guardrails & safety systems.

What this explains

Understanding the architecture demystifies a lot:

  • Token cost — you pay per token because the model processes per token.
  • Context limits — quadratic attention cost.
  • Why order matters in prompts — attention weights position.
  • Hallucination — the next-token objective.
  • Why generation streams — it is genuinely produced one token at a time.

Next

Frequently Asked Questions

How does a transformer work in simple terms?
A transformer processes a sequence of tokens by letting each token attend to every other token — weighing how much each one matters for understanding the current one — through a mechanism called self-attention. Stacked layers of this build up increasingly abstract representations, and the final layer predicts the next token. Generation repeats this one token at a time.
What is self-attention?
The core mechanism where each token computes how relevant every other token is to it, then builds its representation as a weighted blend of them. It is what lets a model connect "it" to the noun it refers to, or a question to the relevant fact earlier in the prompt. Attention over all pairs of tokens is also why context length is computationally expensive.
Why do LLMs have a context length limit?
Self-attention compares every token to every other, so cost grows with the square of the sequence length. That quadratic cost, together with memory for the attention computation, is the practical reason context windows are bounded. Longer contexts need architectural tricks to stay affordable, which is why very long context is a notable feature when a model offers it.
Why do transformers hallucinate?
A transformer is trained to predict the most plausible next token, not to state truths. When it lacks the relevant information, it still produces the most probable continuation, which can be fluent and wrong. Hallucination is a consequence of the objective the model optimises, which is why grounding with retrieval and validating output are the fixes, not better prompting alone.

Related tutorials