🔍 Embeddings & Retrieval

A search query rarely matches the exact wording of its answer. “How can I recover access?” might need a document titled “Reset your password.” An embedding model learns vectors that make useful matches score highly. The important word is learns: nearby vectors reflect the training objective and data, not a guarantee of semantic equivalence or relevance.

Before you start

Linear Algebra explains dot products, norms, and projections. ML Fundamentals and Probability & Statistics prepare you to evaluate retrieved results. Transformers explains the encoders used in many text embedding models; Information Theory supports contrastive losses.

The main route is a small ranking example → representation training → indexing → filtering and reranking → evaluation and updates. By the end, you should distinguish relevance errors from approximate-search errors and design a retrieval experiment that measures both.

1. A vector score is a modeling choice

Let a query vector be q=[1,0] and candidate document vectors be a=[2,2], b=[1,0].

Score	Document a	Document b	Higher-ranked document
Dot product q·d	2	1	a
Cosine q·d/(‖q‖₂‖d‖₂)	1/√2≈.707	1	b

The dot product includes magnitude; cosine compares direction. Magnitude might reflect something learned, but it is not automatically popularity or confidence. Use the metric the model was trained and validated with. Cosine itself divides by norms, so it can be computed on nonnormalized nonzero vectors. Pre-normalizing lets a dot-product index reproduce cosine rankings; a zero vector needs a defined handling policy.

For unit-length vectors, squared Euclidean distance is ‖q−d‖₂²=2−2(q·d). Thus maximizing cosine/dot product and minimizing Euclidean distance produce the same ranking under normalization. Without unit norms, that equivalence fails. Some libraries return squared L2 distance rather than distance; check whether larger or smaller scores mean a better match.

Check yourself: normalize a above. Its squared distance from q is 2−√2≈.586; b has distance zero. Why did a originally win the dot product? Solution: its greater magnitude outweighed its worse direction alignment.

2. Decide what representation is trained to preserve

Static word embeddings such as Word2Vec/GloVe assign one vector per vocabulary entry. Contextual token states vary with surrounding text. A sentence/document representation pools or otherwise combines token states, but taking a mean or a class token from any language model does not automatically create a useful retrieval embedding. Padding masks, pooling, task instructions, and representation training matter.

A bi-encoder computes query and document representations independently:

query encoder: q = f(query)            # D entries
document encoder: d_i = g(document_i)  # D entries
score_i = q·d_i                       # one scalar per candidate

The two encoders can share weights or use different weights trained into a compatible space. Store document vectors ahead of time, encode the query online, then search. The independence makes indexing possible while restricting query–document interaction to the score function. Static, contextual, sentence, and multimodal embeddings all remain useful in appropriate systems; a model's geometry is only as relevant as its training and evaluation support.

Contrastive training and its labels

For one positive document d+ among candidates d_i, a common loss is:

L = −log(exp(sim(q,d+)/τ) / Σ_i exp(sim(q,d_i)/τ)), with positive temperature τ.

With similarities [1,0], first candidate positive, and τ=.5, the logits are [2,0], positive probability≈.8808, and loss≈.1269. This encourages the positive to outrank the sampled alternatives; the denominator and temperature define the competition. Compute it with stable log-sum-exp or a logits loss API.

In-batch negatives reuse other examples' documents, but another query's positive may also answer this query. Hard negatives are plausible-looking documents judged irrelevant, such as a password-reset document for the wrong product. Mine with lexical search, an earlier embedding model, or a teacher model, then check labels. A hard negative that is actually relevant teaches the wrong distinction. Mix difficulties and monitor label noise rather than assuming harder always means better.

Clicks, purchases, or watched videos are implicit feedback, affected by exposure, position, availability and user intent. A skipped result is not automatically irrelevant. Synthetic question–passage pairs need quality checks and contamination controls. Fine-tuning is justified by a measured domain gap, with held-out queries/documents or temporal splits suitable to the intended generalization. No fixed number of pairs or epochs guarantees a gain.

More than one vector per text

A cross-encoder jointly processes a query and candidate document, enabling token interactions before producing a relevance score. It can improve shortlist ranking, but each pair needs query-time computation; improvement is empirical, not automatic. Ordinary cross-encoder scores cannot be precomputed as one query-independent document vector.

Late interaction, as in ColBERT-style models, separately encodes query/document tokens and then combines their similarities. A common MaxSim score is Σ_query_tokens max_document_tokens(q_i·d_j). For query token vectors [1,0] and [0,1], a document containing both gets score 2 with normalized exact matches; a document containing only [1,0] gets score 1 under this toy scoring. Real models add tokenization, masking, learned projections, and indexing choices. Storage depends on document token count, token width, compression, and retained metadata; it is not universally a fixed multiple of a single-vector index.

Matryoshka representation learning applies objectives to multiple embedding prefixes during training. A model trained that way can support smaller prefixes for candidate retrieval and larger representations for later scoring. Renormalize a prefix if required by its metric. Arbitrarily truncating an unrelated embedding model does not have the same guarantee, and quality at each dimension still needs measurement. A full-width later stage requires storing or recomputing that representation; shorter search vectors do not necessarily reduce the encoder's own compute.

Multimodal contrastive models align suitable image/text or other modality encoders into compatible spaces. A shared width alone does not align two independently trained models. See Multimodal VLMs.

3. Build the system as a sequence of measurable stages

Documents → parse/chunk → encode → versioned vectors + metadata + index

Query → understand explicit constraints → encode
      → dense candidates + lexical candidates
      → fuse/deduplicate → rerank → enforce eligibility → results

Chunking determines what a “document” in retrieval means. Preserve headings, identifiers, source offsets, version, and access metadata. Very small chunks may lose context; very large chunks may dilute a match or exceed encoder input limits. Overlap can duplicate results, so evaluate at the level users need—passage, document, product, or answer—and deduplicate accordingly.

Candidate retrieval aims to include useful items within a feasible budget. Reranking reorders that set using richer features or a cross-encoder; it cannot recover a relevant document that never reached the shortlist. Candidate count is a tunable parameter rather than a universal 100→10 recipe. Measure embedding, search, fusion, reranking, and end-to-end latency separately.

Hybrid lexical and dense retrieval

Lexical retrieval such as BM25 scores term matches with corpus and length statistics. It can help with identifiers, rare names, and exact terms. Dense retrieval can connect paraphrases and differently worded concepts, but either can fail depending on training and query structure. Exact identifiers may also justify explicit lookup or filters.

Reciprocal rank fusion combines ranked lists without making their raw scores comparable:

RRF(d)=Σ_lists 1/(c+rank_list(d)).

Ranks start at one; a document absent from a retrieved list contributes zero. The positive constant c controls how heavily to favor the very top positions. If dense returns [A,B] and lexical returns [B,C], with c=60, B gets 1/62+1/61≈.03252; A gets .01639 and C .01613. B ranks first because both systems found it. Weighted fusion and learned reranking are alternatives. RRF's common parameter choices are starting points, not evidence that hybrid search always improves quality.

4. Approximate search trades work for missed neighbors

Exact scoring over N vectors of width D costs O(ND) per query before top-k selection. With normalized vectors, a matrix product can serve as an excellent exact baseline, especially for smaller corpora, batches, or accelerators. “Exact” means exact relative to the stored vectors and chosen metric; those nearest neighbors may still be irrelevant.

An approximate nearest-neighbor (ANN) index avoids some comparisons or compresses representations. Measure ANN recall@k against an exact top-k set for the same vectors, eligible universe, and metric. This differs from relevance recall@k, which uses judged relevant documents. A perfect index cannot rescue a bad embedding ranking.

HNSW: navigate a graph

HNSW constructs a hierarchy of proximity graphs. Sparse upper layers guide a query toward promising regions; a denser bottom layer supports a bounded candidate search. Its parameters commonly include M for graph connectivity, efConstruction for build search effort, and efSearch for query search effort. More effort often improves approximate recall at memory/build/latency costs, but tuning and implementation matter.

It does not require a separate centroid-training phase, though constructing the graph is significant work. Dynamic insertion, deletion behavior, disk storage, distance support and filtered traversal vary by implementation. There is no universal O(log N) worst-case query guarantee, sub-millisecond latency, or fixed memory multiplier. Measure the actual dimension, graph storage, precision, and recall/latency curve.

IVF and PQ: partition and compress

An inverted-file index partitions vectors using a coarse quantizer, often k-means. At query time, nprobe controls how many lists are scanned. More probes increase coverage and work; coarse assignment can omit a useful vector if its list is not searched.

Product quantization splits a D-entry vector into m sub-vectors and stores a codebook entry for each. With 256 codewords per subspace, each code takes one byte; m=64 uses 64 bytes per vector. A query can use lookup tables to approximate distances to codes. IVF-PQ combines list selection with this compressed scoring. Optional reranking with full vectors can recover some quantization-order errors, but needs those full vectors accessible.

For one billion 1024-dimensional FP32 vectors, raw vector storage is 10^9×1024×4 = 4.096 TB decimal. Sixty-four-byte codes use 64 GB; eight-byte IDs alone add 8 GB, before codebooks, lists, documents, metadata, replicas, and any retained full vectors. This is an illustrative storage calculation, not a promise that a billion-vector database fits in 72 GB.

Variants such as optimized PQ, disk-oriented graphs, and combinations of coarse and graph indexes offer other operating points. ScaNN's anisotropic quantization accounts for how error affects inner-product scoring rather than simply preserving a fixed set of “important dimensions.” Compare implementations on a common dataset/metric/recall target; no algorithm is the universal billion-scale default.

5. Filtering, freshness, and model versions are part of correctness

Filters can be hard eligibility rules—tenant access, product availability, delivery region—or uncertain intent predictions. An uncertain category guess should not silently eliminate all correct candidates. Enforce actual eligibility before exposing results, regardless of candidate-generation method.

• Pre-filter/exact subset search: search only eligible items; for a small subset, a flat scan may be efficient.
• Filtering during ANN traversal: the index coordinates search with eligibility, with implementation-specific recall and work.
• Post-filter: retrieve globally then discard ineligible items; it can return fewer than k valid results, even if many eligible neighbors exist elsewhere.

Pre-filtering does not guarantee k results: fewer than k eligible items may exist, and approximate traversal may still miss some. Highly restrictive filters can challenge a graph or make a small exact scan attractive. Over-retrieval is one mitigation, not a guarantee; evaluate selectivity slices and end-to-end permission correctness. Avoid using “high selectivity” ambiguously—state the fraction of items that pass.

For updates, measure ingestion-to-search delay and deletion propagation. A hot recent index plus a compact main index is one option, but merges need deduplication, version handling and score compatibility. Metadata such as availability can change much faster than semantic vectors. Keep it fresh and recheck at the decision point.

A query encoder and stored document vectors must belong to a compatible trained space, including pooling, normalization, prompt conventions, dimensions and version. Different query/document encoders can be compatible when jointly trained. The same dimension does not make a new model compatible with old vectors. Re-embedding and rebuilding often accompany an upgrade; use versioned indexes, shadow evaluation, coordinated cutover and rollback rather than mixing spaces accidentally.

Vector databases add operational responsibilities beyond ANN: metadata/filter semantics, transactions, updates, authorization, replication, backups and recovery. A relational extension such as pgvector, a dedicated engine such as Qdrant, or a managed service such as Pinecone may suit different requirements. Evaluate current supported features and benchmark your workload instead of choosing by a fixed corpus-size cutoff or assuming managed means no operational responsibility.

6. Evaluate the retrieval funnel with a clear denominator

If eight documents are judged relevant and three appear among five results, relevance recall@5 is 3/8=.375, precision@5 is 3/5=.6, and hit-rate@5 is 1. Hit rate asks whether there was at least one hit; it equals recall only in a single-relevant-item setting. If the first relevant result is at rank two, reciprocal rank is .5; MRR averages that quantity over queries.

NDCG@k compares discounted graded relevance to an ideal ordering; specify gain and discount conventions. Average precision/MAP also need consistent cutoff and denominator conventions. Queries with no known relevant labels and responses shorter than k need documented handling. Incomplete judgments make “all relevant documents” an estimate, so review unjudged high-ranked results rather than automatically calling them negatives.

Evaluate each stage and the final experience: ANN recall against exact search, relevance candidate recall, final ranking, latency percentiles under realistic concurrency, zero-result rates, freshness and eligibility. Break out language, query length, rare identifiers, new items, and restrictive filters. Use query-level uncertainty estimates and an untouched holdout when comparing changes. Online clicks require care about position/exposure bias and should be paired with task success or satisfaction measures.

Final check: ANN recall is .99, but users still find results irrelevant. Must the index be the problem? Solution: no. It closely reproduces the stored embedding's exact ranking. Investigate model/task mismatch, chunking, labels, query interpretation, eligibility, and ranking objectives before spending more on ANN effort.

Sources and next lessons

• Sentence-BERT — independently encoded sentence representations and similarity search.
• Dense Passage Retrieval — dual encoders, training negatives, and retrieval evaluation.
• Hierarchical Navigable Small World graphs — graph hierarchy and approximate search.
• Faiss documentation — exact/vector indexes, quantization, and implementation tradeoffs.

Continue with RAG for connecting retrieval to generation and Evaluation & Benchmarking for comparing systems on held-out tasks. Check Classical ML for ranking metrics and calibration.