🚀 Model Serving & Inference Optimization

The LLM Inference Challenge

Serving large language models is fundamentally different from serving classical ML models. Inference is autoregressive: each output token depends on all previous tokens, so generation cannot be parallelized along the sequence dimension. A single request that produces 500 tokens requires 500 sequential forward passes through the model.

Critically, decoding is memory-bandwidth bound, not compute-bound. For each token, the GPU must load the entire model's weights from HBM into compute units, but only performs a small amount of math per byte loaded. A modern GPU like the H100 can do ~3 TB/s of memory bandwidth, so a 70B model in FP16 (140 GB) reads in ~47 ms per token even with perfect utilization. This sets a hard floor on per-token latency that no amount of compute can lower.

KV Cache

During the autoregressive loop, every previously generated token produces Key and Value tensors in every attention layer that the next token must attend to. Recomputing them each step would make generation O(n²) in sequence length, so instead we cache them, turning each decode step back into O(n). The KV cache is the single most important optimization in LLM serving — and, because of how it scales, the primary obstacle to high concurrency.

Size formula: 2 (K + V) × n_layers × n_kv_heads × head_dim × bytes_per_element × seq_len per request.

For Llama-3 70B (80 layers, 8 KV heads with GQA, 128 head_dim, FP16): each token costs 2 × 80 × 8 × 128 × 2 = 320 KB in the cache (the leading 2 is for storing both K and V). A single 8K-context conversation eats ~2.5 GB; 100 concurrent users at 8K push the cache to ~250 GB, well past the ~130 GB of model weights themselves. Two things make this brutal: the cache grows linearly with context length and linearly with concurrency, with no sharing between requests by default. So the cache — not the weights — usually sets the ceiling on how many users a GPU can hold at once.

Shrinking the cache: MHA → MQA → GQA → MLA

The dominant term in that formula is n_kv_heads, and a line of attention variants attacks exactly that:

• MHA (Multi-Head Attention) — the original: every query head has its own key/value head, so n_kv_heads = n_heads. Maximal expressiveness, maximal cache. This is why the KV cache became a bottleneck in the first place.
• MQA (Multi-Query Attention) — all query heads share a single key/value head (n_kv_heads = 1). Shrinks the cache by n_heads× (often 32–64×), but collapsing to one KV head can hurt quality and training stability.
• GQA (Grouped-Query Attention) — the middle ground that won: query heads are split into groups, each sharing one KV head (e.g. 64 query heads → 8 KV heads = 8× smaller cache). Near-MHA quality at a fraction of the memory, which is why Llama-3, Mistral, Qwen, and most modern models use it.
• MLA (Multi-head Latent Attention) — DeepSeek's (V2/V3) approach, and the current frontier. Instead of caching per-head K and V, it projects them down to a single low-rank latent vector (plus a small decoupled RoPE key) that is cached once per token per layer and reconstructed per-head on the fly at compute time. Because the cache no longer scales with head count at all, it captures most of MHA's expressiveness while caching dramatically less — MLA's advantage over MQA is that it preserves quality, not merely that it's small.

A useful way to hold it: MQA and GQA reduce how many K/V heads you store; MLA changes what you store (a compressed latent instead of raw K/V).

Quantizing the cache

Orthogonal to the variant choice, you can also shrink each cached element by storing it at lower precision — FP8 or INT8 KV cache roughly halves or quarters the footprint, directly buying more concurrency or longer context. Keys tolerate low precision better than values (attention is driven by relative scores, so small key errors rarely flip the argmax), so methods like KIVI and KVQuant quantize asymmetrically and protect outlier channels at higher precision. See Quantization for the full treatment. Note this is a separate lever from weight quantization — INT4 weights still leave the KV cache at FP16 unless you quantize it too.

Reusing and evicting the cache

• Prefix / prompt caching — when many requests share a prefix (a long system prompt, a few-shot preamble, a document being repeatedly queried), its KV blocks are computed once and reused across all of them, slashing TTFT. vLLM's PagedAttention and SGLang's RadixAttention both do this via block sharing (see PagedAttention below).
• Eviction & offload — when the cache fills mid-serving, a scheduler must choose: preempt a request and recompute its cache later, or offload it to CPU/host memory and page it back (cheaper than recompute for long contexts, but bandwidth-bound). This is the failure mode behind most "why did my request stall" incidents.
• Sliding-window / eviction policies — models like Mistral cap attention to a fixed window, so the cache stops growing past that window; more general policies (H2O, attention-sink) keep only the tokens that actually get attended to.

PagedAttention and vLLM

Naïve serving allocates a contiguous KV buffer per request, sized to the maximum sequence length. This wastes 60 to 80% of memory through internal fragmentation (unused tail) and external fragmentation (free gaps too small for new requests).

PagedAttention (vLLM) borrows from virtual memory: the logical KV sequence is split into fixed-size blocks (typically 16 tokens). Each request keeps a block table mapping logical positions to physical block IDs. Blocks are allocated on demand and can live anywhere in the GPU memory pool. Memory waste drops to <4%, enabling 2 to 4× more concurrent requests than naive systems. Blocks can also be shared (copy-on-write) across requests with a common prefix, a free 2 to 10× speedup for system-prompt-heavy workloads.

Continuous Batching vs Static Batching

Static batching waits for a batch of requests, runs them together, and returns when the slowest finishes. Short requests are stuck waiting for long ones, and new arrivals queue indefinitely. GPU utilization is poor.

Continuous batching (a.k.a. iteration-level scheduling, in-flight batching) schedules at the granularity of a single decoding step. After each forward pass, finished requests are evicted and new requests are admitted into freed KV slots. Throughput improvements of 5 to 10× over static batching are typical for LLM workloads, with no latency penalty for short requests. Pioneered by ORCA and now standard in vLLM, TGI, and TensorRT-LLM.

Speculative Decoding

A small draft model proposes k candidate tokens cheaply; the large target model verifies them all in a single forward pass. Accepted tokens are kept; the first rejected token is replaced by the target's sample. With acceptance rates around 70 to 80%, you get 2 to 3× wall-clock speedup with no quality change (rejection sampling guarantees identical distribution to the target).

Variants: Medusa (extra decoding heads on the target model, no draft model needed), EAGLE (feature-level drafting, higher acceptance), lookahead decoding (n-gram based). Works best when the draft model is well-aligned with the target and per-step latency dominates throughput concerns. Helps little under heavy batching, because the target forward pass is already amortized across many requests.

Quantized Inference

Reducing weight precision is the single biggest lever for memory bandwidth (and therefore latency).

• INT8 / W8A8: Roughly lossless for 7B+ models. SmoothQuant enables it on activations.
• INT4 (GPTQ, AWQ): 1 to 3% quality drop, 4× weight memory reduction; the default for cost-sensitive serving.
• FP8 (E4M3 / E5M2): Hopper/Blackwell native support; near-lossless and supports activations naturally. Increasingly the production default on H100/B200.
• NF4: 4-bit "NormalFloat" used by QLoRA; convenient for training-adjacent workloads.

Quantization can also shrink the KV cache itself (FP8 / INT8 KV cache), recovering significant headroom for long contexts.

Serving Frameworks (May 2026)

These frameworks are serving harnesses: they wrap raw model weights with a scheduler, KV-cache manager, tokenizer, and API surface, turning a checkpoint into a usable system. The choice is essentially a harness choice; see Harness Engineering for how this layer fits alongside training, eval, and agent harnesses.

Key Metrics

• TTFT (Time to First Token): Dominated by prefill (a single large parallel forward pass over the prompt). Scales with prompt length and model size. Typical: 100 to 500 ms for 1K-token prompts on a 70B model.
• TPOT / ITL (Time Per Output Token, Inter-Token Latency): Pure decoding latency. Memory-bandwidth bound. Typical: 20 to 80 ms/token for 70B models.
• TPS (Tokens Per Second): Per-request is 1 / TPOT. System-wide is the aggregate across all concurrent requests.
• Throughput vs Latency: Bigger batches give higher aggregate TPS but higher per-token latency. The Pareto frontier is set by KV memory and the SLA you must hit.

Scaling Across GPUs

When a model doesn't fit on one GPU (or you want more throughput):

• Tensor Parallelism (TP): Shards each weight matrix across GPUs within a node. Every layer requires an all-reduce, so it demands high-bandwidth interconnect (NVLink). Standard for serving models that exceed a single GPU.
• Pipeline Parallelism (PP): Different layers live on different GPUs; requests flow through as a pipeline. Lower communication, but introduces bubbles unless many micro-batches are in flight. Useful across nodes where NVLink isn't available.
• Expert Parallelism (EP): For MoE models, different experts live on different GPUs. Each token is routed only to the GPUs hosting its top-k experts. Communication is an all-to-all over selected tokens, a different bandwidth profile than TP.
• Data Parallelism (DP): Replicas of the full model; route different requests to different replicas. Pure throughput scaling, zero per-request speedup.

Production setups commonly combine these (e.g., TP=8 within a node, DP across nodes; or TP × EP for MoE).