ⓘ Information Theory for ML

A model assigns the correct class probability 0.9. Another assigns it 0.01. Both might choose the same wrong class on some examples, yet the second can be far more confidently wrong. A loss based only on right or wrong misses that distinction.

Information theory gives us a language for surprise, uncertainty and the cost of predicting with the wrong distribution. Start with probability and statistics. You only need logarithms: log₂(8)=3 because 2³=8. Calculus helps with the optional gradient connection.

What you should leave with: calculate a small entropy and cross-entropy, explain the direction of a KL divergence, and know when two perplexity numbers can be compared.

1. Surprise adds when probabilities multiply

Define the information content, or surprisal, of an outcome x as I(x)=−log₂ p(x), measured in bits. An event with probability 1/2 has 1 bit of surprise; probability 1/8 has 3 bits. A certain event has zero surprise.

For independent events with probabilities p and q, the joint probability is pq and −log₂(pq)=−log₂p−log₂q. This additive property is useful for sequences and coding. For dependent sequences, use conditional probabilities instead of assuming independence.

You will also see natural logarithms, ln, producing nats. Divide nats by ln(2) to get bits. Do not compare losses measured with different log bases as if their numbers used the same units.

2. Entropy is expected surprise

For a discrete distribution p, H(p)=−Σ p(x) log₂ p(x). We take 0 log 0 as zero by its limiting value.

Imagine a source emits A half the time, B a quarter and C a quarter:

Symbol	p(x)	−log₂ p(x)	Contribution to H(p)
A	0.50	1 bit	0.50
B	0.25	2 bits	0.50
C	0.25	2 bits	0.50

Entropy is 1.5 bits per symbol. A code A→0, B→10, C→11 is prefix-free and has exactly that expected length. In general entropy is a lower bound on expected lossless code length under the relevant source/coding assumptions; a finite single-symbol code need not achieve it exactly.

A fair binary source has entropy 1 bit. A binary source with probability 0.99 for one outcome has entropy about 0.081 bits. Lower entropy means less uncertainty under that distribution; it does not mean a model is more accurate. A model can be confidently wrong.

Check: which has greater entropy, a fair four-sided source or a fair coin?

The four-sided source: −4×0.25×log₂(0.25)=2 bits, compared with 1 bit for the coin. Among distributions on the same finite set, uniform probability maximizes entropy.

3. Cross-entropy: predict p using q

The real source follows p, while your model predicts q. Their cross-entropy is H(p,q)=−Σ p(x) log₂ q(x). The averaging uses the real distribution p; the logarithm uses the model q.

If p=[0.5,0.25,0.25] and q=[0.25,0.5,0.25], then:

H(p,q) = 0.5×2 + 0.25×1 + 0.25×2 = 1.75 bits

The model pays 0.25 extra bits per symbol compared with knowing p. If q assigns zero probability to an outcome that p can produce, its cross-entropy is infinite in theory. Floating-point implementations need stable logits-based calculations, not a casual claim that log(0) is harmless.

For an observed class label y, the target is one-hot and its loss is −ln q(y). If q(y)=0.9, loss is about 0.105 nats; if q(y)=0.01, loss is about 4.605 nats. Confident mistakes are penalized strongly.

On a dataset, average this loss over examples. Minimizing it is maximum likelihood when the joint data likelihood factors in the assumed way. Token-level language-model loss similarly uses the probability of each observed token conditioned on its prefix. Tokens within a sequence are not assumed independent.

4. KL divergence: the excess cost

D_KL(p ∥ q)=Σ p(x) log₂[p(x)/q(x)] = H(p,q)−H(p).

Our example gives 1.75−1.5 = 0.25 bits. For normalized discrete distributions, KL is nonnegative, and it is zero exactly when they agree. It is not a distance metric: it is asymmetric and does not satisfy the triangle inequality.

Take p=[0.8,0.2] and q=[0.5,0.5]:

• D_KL(p ∥ q) ≈ 0.278 bits.
• D_KL(q ∥ p) ≈ 0.322 bits.

The direction tells you which distribution supplies the samples and weights the errors. Minimizing D_KL(p_data ∥ q_model) strongly penalizes missing data-supported outcomes. In restricted model families, the reverse direction can prefer concentrating on one supported mode instead of covering low-density gaps. Those are tendencies under assumptions, not universal promises about all optimization runs.

KL in RLHF and DPO measures policy changes relative to a reference. KL in generative models also appears between latent-variable distributions. Always name the two distributions, their support, and the direction.

5. Perplexity: entropy on another scale

Language-model perplexity is exp(mean token negative log-likelihood) when loss uses natural logs, or 2^(mean loss in bits). A mean loss of ln(4) gives perplexity 4.

For a two-token sequence where the correct token gets probabilities 0.5 and 0.25, the sequence probability is 0.125. Mean negative log-likelihood is −ln(0.125)/2 ≈ 1.040 nats, so perplexity is √8 ≈ 2.828. It reflects the geometric mean of inverse correct-token probabilities, not their arithmetic mean and not “2.828 wrong guesses.”

Compare perplexity only with compatible tokenization, evaluation data, context handling, masking and normalization. A tokenizer that changes the number of tokens changes the unit of token-level loss. Lower perplexity on a corpus does not by itself establish better instruction following, factuality or task utility.

Check: a model gives 0.5 probability to every correct token. What is its perplexity?

Each token contributes −ln(0.5)=ln(2). The average is ln(2), so perplexity is 2 regardless of sequence length. This says nothing about whether the most likely token was actually correct.

6. Optional depth: mutual information

Mutual information measures how much learning X reduces uncertainty about Y:

I(X;Y) = H(Y) − H(Y given X) = D_KL(p(X,Y) ∥ p(X)p(Y))

If X and Y are the same fair bit, knowing X removes all uncertainty in Y: I(X;Y)=1 bit. If they are independent fair bits, it is zero. It detects dependence beyond linear correlation, but estimating it from finite, high-dimensional data is difficult.

For a representation Z=f(X), the data processing inequality gives I(Z;Y)≤I(X;Y): processing X alone cannot add information about target Y. More generally this holds when Y→X→Z is a Markov chain, so Z receives no additional information about Y beyond X. It can make existing information easier for a constrained predictor to use. This is why an embedding can improve a simple classifier without inventing new label information.

These discrete intuitions need care for continuous variables: differential entropy can be negative and depends on units. KL and mutual information retain nonnegativity under their appropriate density definitions. Do not substitute differential entropy into a discrete coding argument without checking assumptions.

7. Use the right objective, then inspect its limits

With softmax probabilities q and a one-hot target y, the cross-entropy gradient with respect to logits is q−y. Increasing the correct logit reduces loss; incorrect logits receive pressure proportional to their predicted probabilities. Numerical computing explains why fused log-softmax implementations are safer than softmax followed by log.

Label smoothing replaces the one-hot target with a mixture such as (1−ε)y+εu, where u is uniform. This changes the objective and the preferred probabilities. It can help some generalization problems but is not a universal calibration fix. Entropy regularization encourages uncertainty or exploration depending on its sign; it does not establish honesty, correctness or safety.

Check: can a classifier reduce cross-entropy while accuracy stays unchanged?

Yes. Changing the correct-class probability from 0.6 to 0.8 improves its loss while keeping the same top class. Conversely, one very confident error can worsen cross-entropy even while more examples become correctly classified. Report metrics matched to the decision.

Continue to numerical computing, then ML Fundamentals. Return here when reading language-model losses, variational objectives or reference-policy penalties.

Sources and further practice

• Information Theory, Inference, and Learning Algorithms — MacKay's free textbook, with coding and inference exercises.
• Dive into Deep Learning: information theory — entropy, KL and mutual information examples.
• Hugging Face: perplexity of fixed-length models — tokenizer and context-window evaluation caveats.