Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Model Distillation & Quantization

Make models smaller and faster: quantization (GGUF, GPTQ, AWQ), knowledge distillation, the accuracy-vs-efficiency trade-off, and when self-hosting a compressed model makes sense.

Expert4 min readUpdated
On this page

Distillation and quantization make models smaller, cheaper and faster — the techniques behind running capable models on modest hardware. They matter most when you self-host. This tutorial explains them and the accuracy-versus-efficiency trade-off, honestly noting when a hosted API makes the whole topic moot.

Key Takeaways

  • Quantization stores weights at lower precision — big memory savings, small accuracy cost.
  • Distillation trains a small student to mimic a large teacher — capability in a smaller package.
  • GGUF, GPTQ, AWQ are quantization formats/methods; the right one depends on your runtime.
  • These matter when you self-host; a hosted API removes the need for most teams.

The efficiency problem

Full-precision large models need substantial GPU memory and are expensive to run. If you self-host — for privacy, cost or offline operation — compression is what makes it feasible. If you use a hosted API, the provider handles all of this and you can largely skip this topic.

Quantization

A model's weights are numbers, normally stored at 16-bit precision. Quantization stores them at lower precision — 8-bit or 4-bit — trading a little accuracy for large savings:

PrecisionRelative memoryQuality
16-bit (full)100%Baseline
8-bit~50%Near-identical
4-bit~25%Small, usually acceptable, degradation

A 4-bit quantized model uses roughly a quarter of the memory, which is often the difference between fitting on your hardware and not. The quality cost is usually modest and task-dependent — measure it on your evaluation set rather than assuming.

The formats

  • GGUF — a widely-used format for running quantized models on CPU and consumer hardware, standard with llama.cpp and Ollama. If you run models locally, you will meet GGUF.
  • GPTQ — a quantization method that carefully chooses how to compress weights to preserve accuracy, common for GPU inference.
  • AWQ (Activation-aware Weight Quantization) — protects the weights that matter most to the model's activations, often preserving quality well at low bit-widths.

Which to use depends on your runtime and hardware. For CPU or mixed hardware, GGUF; for GPU serving, GPTQ or AWQ are common. See low-latency LLM serving.

Knowledge distillation

Distillation transfers capability from a large "teacher" model into a smaller "student" by training the student to mimic the teacher's outputs. The student learns not just the right answers but the teacher's output distribution, which packs more of the teacher's behaviour into fewer parameters than training the small model on raw data would.

Distillation is how some notably small models achieve surprising capability. They rarely fully match the teacher on the hardest tasks, but for many uses a distilled small model is good enough at a fraction of the cost — the basis of small language models.

The accuracy-efficiency trade-off

Every compression technique trades quality for efficiency. The engineering question is where on that curve your task sits:

Compression trades quality for efficiency — the right point on the curve depends on your task's tolerance.

When this matters for you

  • Self-hosting for privacy or offline — quantization makes it fit your hardware.
  • High-volume, cost-sensitive workloads — a quantized or distilled small model can serve the easy majority cheaply, routing hard cases to a bigger model.
  • Edge and on-device — small quantized models run where a hosted API cannot reach.

And when it does not: if a hosted API meets your latency, cost and privacy needs, quantization is operational complexity you do not have to take on. Most application teams are in this category.

Next

Frequently Asked Questions

What is model quantization?
Storing a model's weights at lower numeric precision — 8-bit or 4-bit instead of 16-bit — to reduce memory use and speed up inference, with a small accuracy cost. A 4-bit quantized model uses roughly a quarter of the memory of the full-precision version, which is what lets large models run on modest hardware. GGUF, GPTQ and AWQ are common quantization formats.
What is knowledge distillation?
Training a smaller student model to mimic a larger teacher model, transferring much of the teacher's capability into a model that is cheaper and faster to run. The student learns from the teacher's outputs rather than from raw data alone. It is how some small models punch above their size, though they rarely fully match the teacher on hard tasks.
What is the difference between GGUF, GPTQ and AWQ?
They are different quantization formats and methods. GGUF is a widely-used format for running quantized models on CPU and consumer hardware, common with Ollama and llama.cpp. GPTQ and AWQ are quantization methods that optimise which weights to compress and how, often preserving accuracy better than naive quantization. Which to use depends on your runtime and hardware.
Should I quantize a model or just use a hosted API?
Use a hosted API unless you have a specific reason to self-host — privacy, cost at high volume, or offline operation. Quantization matters when you run models yourself, where it makes larger models fit on your hardware. If a hosted model meets your needs, quantization is a complexity you do not need to take on.

Related tutorials