⚄ Probability and Statistics for ML

A fraud detector catches 90% of fraud. That sounds promising. But when it flags a payment, how often is it right? You cannot answer until you know how common fraud is and how many ordinary payments it flags.

This is what probability is for: making the missing assumptions visible. Statistics takes the next step—learning about a population from a sample without pretending the sample is the whole world.

Before you start: arithmetic, fractions and basic algebra are enough. Linear algebra helps with the optional covariance section. By the end, you should be able to calculate a conditional probability, interpret an uncertainty interval, and design a fair comparison between two models.

1. Start with outcomes and counts

An event is a set of possible outcomes. P(A) is the probability that event A occurs. Probabilities are between 0 and 1; probabilities of mutually exclusive, exhaustive outcomes sum to 1.

Among 10,000 payments, suppose 100 are fraudulent. The detector flags 90 frauds and 198 ordinary payments:

	Fraud	Ordinary	Total
Flagged	90	198	288
Not flagged	10	9,702	9,712
Total	100	9,900	10,000

Conditional probability changes which outcomes count as the denominator: P(A given B) = P(A and B) / P(B), provided P(B) > 0.

• P(flagged given fraud) = 90/100 = 90%: recall or sensitivity.
• P(flagged given ordinary) = 198/9,900 = 2%: false-positive rate.
• P(fraud given flagged) = 90/288 = 31.25%: precision.

The same 90 sits in two different fractions. Reversing the condition changes the question.

Events are independent if P(A and B) = P(A)P(B). Disjoint events are different: they cannot both happen. Two nonzero-probability disjoint events are not independent.

Check: what is the chance a payment is flagged?

288/10,000 = 2.88%. Equivalently, 0.90 × 0.01 + 0.02 × 0.99 = 0.0288. We sum across the fraud and ordinary groups because they partition all payments.

2. Bayes: turn the conditional around

Bayes' rule follows by writing the same joint probability in two ways:

P(A given B) = P(B given A) P(A) / P(B)

In the payment example, the prior probability of fraud is 0.01. The likelihood of a flag given fraud is 0.90. The evidence, P(flagged), is 0.0288. The posterior, P(fraud given flagged), is 0.009/0.0288 = 0.3125.

The flag increases fraud probability from 1% to 31.25%. It is useful evidence even though most flags are false alarms. Whether to block a payment depends on the cost of a miss, the cost of a false alarm, and what other checks are available.

If prevalence falls to 0.1% while the two conditional error rates stay fixed, precision falls to 0.0009 / (0.0009 + 0.01998) ≈ 4.31%. That assumption about fixed error rates may fail under distribution shift; it is a scenario calculation, not a production guarantee.

Conditional independence means independence after conditioning on another variable. Naive Bayes assumes features are conditionally independent given the class. It does not assume every feature is independent in the overall dataset. See classical ML for model choices built on these assumptions.

3. Random variables: average and spread

A random variable assigns a number to an outcome. Suppose X is the number of support tickets arriving in a minute:

x	P(X = x)	x P(X = x)	(x − 1)² P(X = x)
0	0.25	0	0.25
1	0.50	0.50	0
2	0.25	0.50	0.25

The expectation is E[X] = Σ x P(X = x) = 1 ticket. It need not be the next observed value. The variance is E[(X − E[X])²] = 0.5 tickets². Standard deviation is √0.5 ≈ 0.707 tickets, in the original units.

Linearity of expectation, E[X + Y] = E[X] + E[Y], does not require independence. Adding variances does: in general, Var(X + Y) = Var(X) + Var(Y) + 2 Cov(X,Y).

Covariance is E[(X − E[X])(Y − E[Y])]. Positive covariance means the variables tend to move together relative to their means. Zero covariance does not generally imply independence: for symmetric X, X and X² can be uncorrelated yet clearly dependent. Correlation divides covariance by the two standard deviations and is undefined if either standard deviation is zero.

4. Choose a distribution for a reason

A discrete PMF gives probability at each value. A continuous PDF gives probability through area over an interval; density at a point can exceed 1. For a continuous variable, the probability of an exact individual value is zero. The CDF, F(x) = P(X ≤ x), works for both kinds.

Distribution	What it models	Assumption to check
Bernoulli(p)	One binary outcome; mean p, variance p(1 − p)	Define what counts as success
Binomial(n,p)	Successes in n trials	Independent trials, common success probability
Categorical	One of several classes	Probabilities cover the whole class set
Normal(μ,σ²)	A continuous quantity around a mean	Symmetry/tails are a suitable approximation
Poisson(λ)	Counts in a fixed exposure; mean and variance λ	Constant rate and independent increments for the process model
Beta(a,b)	Uncertainty about a probability	Prior shape is a modeling choice

Clicks from the same person are often dependent. Ticket rates change through the day. A distribution name does not remove those issues.

The central limit theorem says that under conditions such as independent, identically distributed samples with finite variance, a suitably standardized sample mean approaches a normal distribution. It does not say the raw observations become normal. Strong dependence, extreme tails and tiny samples can make the approximation poor.

5. Estimate a parameter, then admit uncertainty

Let p be an unknown success probability. In 100 independent trials, you observe 60 successes. The likelihood as a function of p is proportional to p⁶⁰(1 − p)⁴⁰. Maximum likelihood chooses p̂ = 60/100 = 0.6. A likelihood compares parameter values for the observed data; it is not automatically a probability distribution over parameters.

With a Beta(1,1) prior, the posterior is Beta(61,41), and the posterior mean is 61/102 ≈ 0.598. This is a small smoothing effect. With three successes from only three trials, the same prior gives posterior mean 4/5 = 0.8 instead of the MLE of 1. The prior matters more when data are scarce.

For a sample mean of n independent observations with population standard deviation σ, standard error is σ/√n. For a proportion, an approximate plug-in standard error is √(p̂(1 − p̂)/n). At p̂ = 0.6, n = 100, it is about 0.049. A rough normal 95% interval is 0.6 ± 1.96 × 0.049, or [0.504, 0.696].

That simple interval is only an approximation. Wilson or exact binomial methods behave better near 0 or 1 and with small samples. Resample users rather than individual clicks when users are the independent units.

Check: how many independent samples halve the standard error?

Four times as many, with the same variance. Standard error scales as 1/√n, so moving from 100 to 400 samples halves it. Duplicating the same observations does not create 400 independent samples.

6. Intervals, tests and fair comparisons

A frequentist 95% confidence interval comes from a procedure that covers the fixed population parameter in 95% of repeated samples under its assumptions. It is not a 95% probability statement about that fixed parameter after this interval is observed. A Bayesian 95% credible interval contains 95% of the posterior probability under the chosen prior and likelihood.

A p-value is the probability, assuming a null hypothesis and test model, of a statistic at least as extreme as the observed one. It is not P(null is true), the chance the result is a fluke, or the size of the effect.

Suppose model A gets 80/100 items right and B gets 82/100 right on the same test set. The 2-point gain alone is insufficient. Did B fix two errors and break none, or fix 22 and break 20? Paired outcomes matter. Use a paired test or paired bootstrap; preserve item pairs and any user/document clusters. An ordinary bootstrap samples units with replacement, recomputes the statistic, and uses its variation to estimate sampling uncertainty. It cannot repair a biased test set.

For an A/B experiment, define the unit of randomization, primary metric, minimum useful effect, sample-size plan, and stopping rule before looking at results. Random assignment helps isolate a causal effect; an observational correlation usually does not. Repeatedly checking an ordinary fixed-horizon p-value and stopping when it crosses 0.05 inflates false positives. Use the planned end point or a method designed for sequential testing.

7. Carry this into ML

Before trusting a score, ask: what population does this sample represent, what is independent, how large is the effect, and how uncertain is it? A narrow interval around a biased estimate is still misleading. Training on tomorrow's information, repeatedly tuning on a test set, or testing only easy cases breaks the comparison before the statistics begin.

Check: a model predicts 0.8 on 100 examples, but only 60 are positive. What can you conclude?

In this sample and prediction bin it is overconfident: the observed rate is 0.6. Inspect uncertainty, sample selection and other bins before making a population-wide claim. Calibration asks whether stated probabilities match observed frequencies; ranking accuracy asks a different question.

Continue to calculus and optimization to learn how parameters change, then information theory for probability-based losses. ML Fundamentals uses these ideas to split data, choose models and diagnose generalization. Evaluation develops the model-comparison workflow.

Sources and further practice

• Harvard Stat 110 — probability lectures and exercises, including conditioning and expectation.
• Seeing Theory — interactive probability and inference explanations.
• NIST: confidence intervals — repeated-sampling interpretation and assumptions.
• An Introduction to Statistical Learning — resampling, model assessment and practical exercises.