What a vector index actually stores

“Vector database” sounds like a new kind of machine. It isn’t. Open one up and you find two things: a big array of floats, and a data structure that lets a search skip most of that array. Once you know what each costs, you can reason about any of them: pgvector, FAISS, a hosted service, or the 30 lines of Python at the end of this post.

Text becomes a list of numbers

Start one layer down. An embedding model reads a piece of text and returns a fixed-length list of floats, a vector. Common sizes are 384, 768, 1,024 and 1,536 numbers. The model is trained so that texts with similar meaning get vectors that point in similar directions.

“How do I reset my password?” and “I forgot my login” share almost no words. Their vectors still end up close together. That is the only property retrieval needs.

So the first thing a vector store holds is just this:

doc_id   vector (1,536 float32 values)
0        [ 0.0123, -0.0441,  0.0907, ... ]
1        [-0.0310,  0.0228,  0.0015, ... ]
...

Do the arithmetic. A float32 is 4 bytes. 1,536 of them is 6,144 bytes, about 6 KB per document. A million documents is about 6.1 GB of raw vectors, before any index. That number decides whether your index fits in RAM, and it is the first thing to compute for any project.

Similarity is a dot product

“Close together” needs a number. The usual one is cosine similarity: the dot product of two vectors divided by the product of their lengths.

cos(a, b) = (a · b) / (|a| × |b|)

It is 1 when two vectors point the same way, 0 when they are unrelated, and -1 when they point opposite ways. Many stores normalise every vector to length 1 when it is inserted. Then the denominators are all 1 and cosine similarity is just the dot product: multiply pairs, add them up.

The baseline: look at everything

The simplest search is exact. Compare the query with every stored vector, keep the k best.

import math

def cosine(a, b):
  dot = sum(x * y for x, y in zip(a, b))
  na = math.sqrt(sum(x * x for x in a))
  nb = math.sqrt(sum(x * x for x in b))
  return 0.0 if na == 0 or nb == 0 else dot / (na * nb)

docs = {
  "reset your password":       [0.9, 0.1, 0.0],
  "forgot my login":           [0.7, 0.3, 0.2],
  "track my order":            [0.1, 0.9, 0.2],
  "where is my parcel":        [0.0, 0.8, 0.3],
  "change billing address":    [0.3, 0.2, 0.9],
}
query = [0.85, 0.15, 0.05]               # "can't sign in"

scored = sorted(docs, key=lambda d: -cosine(query, docs[d]))
for d in scored[:2]:
  print(f"{cosine(query, docs[d]):.3f}  {d}")

Three dimensions instead of 1,536, but the algorithm is identical. It is always correct: it returns the true top k, by definition.

The cost is the problem. Each query does n × d multiply-adds. With a million documents at 1,536 dimensions, that is about 1.5 billion per query, and it reads all 6.1 GB. A modern CPU with SIMD can do it in somewhere around a tenth of a second to a second. That is fine for a demo and for small collections. Up to about 100,000 vectors, brute force is often the right answer and needs no index at all. It is not fine for a search box serving many people over 100 million documents.

What an index adds

To go faster you must look at fewer vectors. There are two common ways to decide which ones, and they store different things.

IVF (inverted file). At build time, cluster the vectors, typically with k-means, into a few thousand groups. Store the cluster centres, and for each centre a list of the document ids filed under it. At query time, compare the query with the centres only, pick the nearest few lists (nprobe of them), and brute-force just those. On disk: the vectors, plus centroids, plus one id list per centroid.

HNSW (hierarchical navigable small world). At build time, link every vector to a few dozen of its near neighbours, in a graph with a few sparse layers on top for long jumps. At query time, start at an entry point and walk greedily towards the query, always moving to the neighbour that is closer. On disk: the vectors, plus a neighbour list for every vector. With 32 links per node at 4 bytes per id, that is about 128 bytes more per document, per layer it appears in.

Both are often combined with quantisation: storing a compressed code instead of the full vector. Product quantisation can shrink 6 KB down to around 100 bytes per document, so a billion of them fit in memory. The search then compares against approximations of the vectors, not the vectors themselves.

Why every fast index approximates

Here is the catch. An index skips vectors based on a guess about where the answer is. IVF guesses that the nearest documents sit in the nearest clusters. Usually they do. But a document right on the border of two clusters can be filed in the one you did not probe, and the search will never see it. HNSW’s greedy walk can stop in a good neighbourhood that is not the best one. Quantisation can rank two close vectors in the wrong order.

So the result is a set of good matches, not the guaranteed best ones. The standard measure is recall@k: of the true top k from brute force, what fraction did the index return? You can watch it trade against work with a tiny IVF:

import math
import random

random.seed(7)
DIM, N, LISTS, K = 16, 2000, 20, 10

def normalize(v):
  n = math.sqrt(sum(x * x for x in v))
  return [x / n for x in v]

def dot(a, b):
  return sum(x * y for x, y in zip(a, b))

# 40 topics; each document is a noisy copy of one topic's direction
topics = [[random.gauss(0, 1) for _ in range(DIM)] for _ in range(40)]
docs = [normalize([t + random.gauss(0, 0.6) for t in random.choice(topics)]) for _ in range(N)]
queries = [normalize([t + random.gauss(0, 0.6) for t in random.choice(topics)]) for _ in range(50)]

def brute_top_k(q, k):
  return sorted(range(N), key=lambda i: -dot(q, docs[i]))[:k]

# Build an IVF index: pick centroids, then file every doc under its nearest one
centroids = random.sample(docs, LISTS)
lists = {c: [] for c in range(LISTS)}
for i, d in enumerate(docs):
  best = max(range(LISTS), key=lambda c: dot(d, centroids[c]))
  lists[best].append(i)

def ivf_top_k(q, k, nprobe):
  near = sorted(range(LISTS), key=lambda c: -dot(q, centroids[c]))[:nprobe]
  candidates = [i for c in near for i in lists[c]]
  return sorted(candidates, key=lambda i: -dot(q, docs[i]))[:k], len(candidates)

truth = [set(brute_top_k(q, K)) for q in queries]
print(f"brute force: {N} vectors scanned per query, recall 1.00")
for nprobe in (1, 2, 4, 8):
  hits, scanned = 0, 0
  for q, t in zip(queries, truth):
      found, n = ivf_top_k(q, K, nprobe)
      hits += len(t & set(found))
      scanned += n
  print(f"nprobe={nprobe}: ~{scanned // len(queries)} vectors scanned, recall@{K} {hits / (K * len(queries)):.2f}")

With nprobe=1, the search looks at about 110 of 2,000 vectors, roughly 5%, and finds about 71% of the true top 10. With nprobe=8 it looks at about 40% and finds 99%. That curve is the whole design space. Every vector index has a knob like this: nprobe in IVF, efSearch in HNSW. Turning it up buys recall with time.

What to take away

When someone says “we put it in a vector database”, you can now ask the useful questions:

1. How many vectors, how many dimensions, what type? Multiply. That is your memory floor.
2. Exact or approximate? Below about 100,000 vectors, exact is often fast enough and never wrong.
3. What recall@k do you get, at what latency? Measure it against brute force on your own queries. A demo that “looks right” is not a measurement.

The index is a shortcut. Brute force is the truth you check it against, so build that first.