Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Vector Databases Deep Dive

How vector databases work under the hood: the HNSW index, approximate nearest-neighbour search, cosine vs Euclidean distance, product quantization and metadata filtering — for Java developers.

Advanced4 min readUpdated
On this page

You can use a vector database effectively without understanding its internals, but knowing how approximate nearest-neighbour search works explains its trade-offs — why it is fast, why it is approximate, and how to tune it. This tutorial goes under the hood of the stores you met in Spring AI embeddings.

Key Takeaways

  • Vector search is approximate — it trades a little recall for an enormous speed-up.
  • HNSW is the dominant index: a navigable graph you traverse toward the query.
  • Match the distance metric to your embedding model — cosine for normalised vectors.
  • Quantization trades a little accuracy for large capacity at scale.

The problem: exact search does not scale

Finding the most similar vectors to a query by comparing against every stored vector is exact but linear — a million vectors means a million comparisons per query. That is far too slow for interactive use. Vector databases solve this with approximate nearest-neighbour (ANN) search: accept a tiny chance of missing the true best match in exchange for searching in milliseconds.

HNSW: the dominant index

Hierarchical Navigable Small World builds a multi-layer graph of vectors:

HNSW: search starts in the sparse top layer for fast navigation and descends to dense lower layers for accuracy.

Search starts at the top layer, where sparse long-range links let it jump quickly toward the query's region, then descends through layers with progressively denser local connections for accuracy. At each layer it greedily moves to the neighbour closest to the query. The result is search that scales logarithmically rather than linearly.

Tuning HNSW

Two parameters trade quality against speed and memory:

  • m — connections per node. Higher means better recall and more memory.
  • ef (search) — how many candidates to consider during search. Higher means better recall and slower queries.
pgvector HNSW tuning
spring:
  ai:
    vectorstore:
      pgvector:
        index-type: HNSW
        # Higher m and ef_construction improve recall at the cost of build
        # time and memory. Defaults are sensible; tune only if recall is low.

Distance metrics

The metric defines what "similar" means, and it must match your embedding model:

MetricMeasuresUse when
CosineAngle between vectorsNormalised embeddings (most text models)
Dot productMagnitude and directionSome models, when magnitude carries meaning
Euclidean (L2)Straight-line distanceEmbeddings trained for it

Product quantization and scale

At very large scale, storing millions of full-precision vectors strains memory. Product quantization compresses vectors by splitting each into sub-vectors and encoding each with a compact code from a learned codebook. This can shrink memory by an order of magnitude, at a small accuracy cost — the codes are approximations of the originals.

It is one of several capacity techniques (alongside disk-based indexes and sharding) that let vector databases scale beyond what in-memory full-precision storage allows. For most applications you will not need it; it becomes relevant at the hundreds-of-millions-of-vectors scale.

Metadata filtering and the index

Filtering by metadata ("only this tenant's documents") interacts with the ANN index in ways that affect performance. A very selective filter can force the database to search more of the graph to find enough matching results, slowing queries. Databases handle this differently — some filter during traversal, some after. At scale, test filtered query performance, not just unfiltered, and index the metadata fields you filter on heavily. See multi-tenant AI architectures.

What this means for you

Understanding the internals guides practical decisions:

  • Recall vs speed is a dial (m, ef) — raise it if retrieval misses relevant passages, lower it if queries are too slow.
  • The distance metric must match the embedding model — the most common silent misconfiguration.
  • Scale techniques (quantization, sharding) matter only at large scale — do not adopt their complexity early.
  • Filtered queries need their own performance testing.

Next

Frequently Asked Questions

How does a vector database find similar vectors so fast?
It uses an approximate nearest-neighbour index, most commonly HNSW, which builds a navigable graph of vectors so a search can hop toward the query region instead of comparing against every stored vector. This trades a small amount of recall for an enormous speed-up — searching millions of vectors in milliseconds rather than scanning them all.
What is HNSW?
Hierarchical Navigable Small World, a graph-based index for approximate nearest-neighbour search. It builds layers of connections between vectors, with sparse long-range links at the top for fast navigation and dense local links at the bottom for accuracy. Search starts at the top and descends, greedily moving toward the query. It is the default index in most vector databases.
Cosine similarity or Euclidean distance — which should I use?
It depends on your embedding model. Most modern text embedding models produce normalised vectors, for which cosine similarity is the natural choice and is what libraries default to. Euclidean distance suits some other embeddings. The key rule is to match the distance metric to what your embedding model was trained for — a mismatch degrades retrieval quality.
What is product quantization?
A compression technique that splits each vector into sub-vectors and represents each with a compact code, drastically reducing memory at a small accuracy cost. It lets a vector database hold far more vectors in memory than storing them at full precision would allow, which matters at very large scale. It is one of several ways vector databases trade a little accuracy for a lot of capacity.

Related tutorials