🤖 Classical ML

A fraud detector can improve accuracy while catching less fraud. A more flexible model can fit the training set better while making worse decisions tomorrow. Classical ML gives us a manageable place to learn both lessons: define the decision, fit a model, and test the assumptions behind its apparent success.

Before you start

Read ML Fundamentals for train/validation/test splits and Probability & Statistics for conditional probabilities and sampling uncertainty. Linear Algebra explains vectors and matrix products. You can follow the numerical examples before studying derivatives; Calculus & Optimization prepares you for the optional optimization sections.

By the end, you should be able to:

• Compute the errors at an operating threshold and connect them to a decision cost.
• Explain what a linear model, kernel, and tree ensemble can represent, rather than choosing by reputation.
• Distinguish ranking, probability calibration, and performance at one threshold.
• Make a clustering or embedding result useful without mistaking geometric similarity for ground truth.

The main route is evaluation → linear models → trees → validation. The kernel derivation and embedding objectives add depth after that route makes sense.

1. Start with the decision, not the algorithm

Suppose 10 of 1,000 transactions are fraudulent. At one threshold, a detector produces:

Actual outcome	Flagged	Not flagged	Total
Fraud	8 true positives (TP)	2 false negatives (FN)	10
Legitimate	18 false positives (FP)	972 true negatives (TN)	990

Its accuracy is (8 + 972) / 1000 = 98%. A detector that flags nothing gets 99% accuracy, but catches no fraud. Accuracy is not mathematically wrong; it gives each error the same weight, which may not match this decision.

Read the other metrics as questions:

• Precision: of the 26 flags, how many were fraud? 8 / 26 ≈ 30.8%.
• Recall: of the 10 fraud cases, how many were found? 8 / 10 = 80%.
• False-positive rate: of the 990 legitimate cases, how many were flagged? 18 / 990 ≈ 1.82%.
• F1: the harmonic mean of precision P and recall R, 2PR / (P + R) = 2TP / (2TP + FP + FN) ≈ 44.4%.

The positive class must be defined explicitly. If no examples are flagged, precision has a zero denominator; document the evaluation convention instead of silently treating the result as an ordinary measured precision.

A threshold is a policy choice

Suppose a false flag costs 5 units and missed fraud costs 100. This detector costs 18×5 + 2×100 = 290, compared with 10×100 = 1000 for flagging nothing. Those are illustrative costs, not a recommendation for a payments system.

For a calibrated probability p, predicting positive has expected error cost (1−p)C_FP; predicting negative costs pC_FN. Under this simplified two-action model, choose positive when p > C_FP / (C_FP + C_FN). With these costs, the threshold is about 0.0476. Real systems also have review capacity, transaction-specific loss, intervention effects, and uncertainty; include them in the policy rather than assuming 0.5 is special.

Fβ = (1+β²)PR / (β²P + R) weights recall more as β increases. It is useful for a specified evaluation objective, but β is not a direct conversion from a false-negative/false-positive cost ratio. Optimize actual expected cost or a constraint such as recall at a feasible review volume when that is the business requirement.

Ranking, calibration, and one operating point are different

ROC-AUC summarizes true-positive rate against false-positive rate across thresholds. It also equals the probability that a random positive outranks a random negative, counting ties as one half. Changing only class prevalence while preserving the two class-conditional score distributions does not change ROC-AUC. It does change precision: a small false-positive rate may generate many false alarms when negatives are numerous.

Precision–recall curves expose that burden. For an uninformative ranking, the population precision baseline is the positive prevalence. Average precision summarizes precision at recall increases; trapezoidal PR-AUC is a different calculation. Name the implementation. Report PR behavior and operational counts for rare positives, while retaining ROC metrics when their question is relevant. There is no universal prevalence cutoff at which ROC-AUC becomes invalid.

Calibration asks whether outcomes match predicted probabilities. Among comparable cases assigned p≈0.8, about 80% should be positive. This is not the same as top-label classification accuracy. A reliability plot compares mean probability with observed positive frequency in bins. The Brier score, mean((p−y)²), measures overall probability error, including both calibration and discrimination; a low aggregate Brier score can hide a poor subgroup. Binned expected calibration error depends on binning and sample size.

Fit Platt scaling, isotonic regression, or temperature scaling on data not used to train the predictor; then evaluate on another untouched set. Platt scaling fits a slope and intercept to a score. Temperature scaling divides logits by a positive scalar. Neither fixes missing features, distribution shift, or inadequate labels by itself.

Metrics beyond binary classification

For multiclass metrics, macro averaging gives each class equal weight, weighted averaging weights by class support, and micro averaging pools counts. Micro precision/recall/F1 equal accuracy for ordinary single-label classification over all classes; that identity does not extend to every multilabel setting.

For retrieval, first define the relevance judgments and candidate universe. If two of the top three results are relevant and four relevant items are known, precision@3 is 2/3, recall@3 is 2/4, and hit-rate@3 is 1. If the first relevant item is second, reciprocal rank is 1/2.

• MRR averages reciprocal first-relevant ranks across queries.
• Average precision rewards relevant items appearing early; MAP averages it across queries. Truncation and denominator conventions for MAP@k must be stated.
• NDCG@k divides discounted graded relevance by the best possible gain for the judged query. A common gain is 2^relevance−1, discounted by log₂(rank+1).

For regression, MSE/RMSE emphasize large residuals; MAE gives linear error cost; Huber is quadratic near zero and linear farther away. MAPE is problematic for targets near zero. RMSLE compares log(1+prediction) with log(1+target) for appropriate nonnegative data; it emphasizes relative differences and is not automatically the right loss for every skewed target.

Check yourself: if prevalence falls but class-conditional score distributions stay fixed, must ROC-AUC fall? Solution: no. Its pairwise ranking probability is unchanged. Precision at the old threshold usually falls because more negatives compete with each positive.

2. Fit a model whose predictions you can inspect

A linear regressor starts with ŷ = w·x + b. Here x contains p features, w contains p learned coefficients, and b is an intercept. For n examples stored as rows, X has shape n×p, w has shape p, and Xw + b has shape n.

One prediction, one update

Take x = [1, 2], w = [2, −1], b = 1, and target y = 3. The prediction is 2×1 − 1×2 + 1 = 1. For loss L = (ŷ−y)²/2, the residual is −2, the weight gradient is −2x = [−2, −4], and the intercept gradient is −2.

With learning rate 0.1, the new values are w = [2.2, −0.6] and b = 1.2. The same example now predicts 2.2 and its loss falls from 2 to 0.32. This illustrates a local update; one favorable training step does not establish generalization.

A stack of linear transformations still defines a linear transformation. However, linear in the coefficients does not require linear in the original measurements: adding a feature such as age² or an interaction gives a linear model a curved boundary in the original input space.

Solving and regularizing least squares

Ordinary least squares minimizes ‖Xw−y‖₂². If X has full column rank, the normal equations are XᵀXw = Xᵀy. The inverse expression (XᵀX)⁻¹Xᵀy is a useful derivation, not the preferred instruction to a numerical library. QR or SVD-based least squares avoids explicitly inverting the Gram matrix; forming XᵀX squares the condition number. SVD also handles rank deficiency through a pseudoinverse.

Dense direct solves require substantial memory and work as n and p grow. Iterative methods use matrix-vector products or minibatches, and online methods support streaming data. Choose using sparsity, conditioning, memory, and accuracy requirements—not a fixed feature-count cutoff. Iteration does not make ill-conditioning disappear.

Regularization changes the objective:

Penalty	What it encourages	Important boundary
Ridge: λ‖w‖₂²	Smaller coefficients; stability with correlated features	Usually does not make coefficients exactly zero
Lasso: λ‖w‖₁	Sparse coefficients	Selection among correlated features can be unstable
Elastic net: L1 + L2	Sparsity with additional shrinkage	Both strength and mixing need validation

Scale features appropriately before interpreting or penalizing coefficient magnitudes. The intercept is often excluded from the penalty. Tune preprocessing and regularization inside training folds to prevent validation leakage.

Least squares does not require normally distributed residuals just to compute a predictor. Assumptions such as a correctly specified conditional mean and exogenous errors matter for coefficient interpretation; independence and variance assumptions matter for conventional standard errors. Gaussian errors support exact small-sample inference under the usual model. Heteroscedasticity can make conventional standard errors wrong; robust errors do not repair omitted-variable bias or a bad prediction function.

Logistic regression produces a probability model

For a binary target, form a logit z = w·x + b, then p = σ(z) = 1/(1+exp(−z)). A logit of 0 gives p=0.5; a logit of ln(3) gives p=0.75. Train by minimizing binary cross-entropy, −[y log p + (1−y) log(1−p)], evaluated with a stable logits-based implementation.

The gradient with respect to z is p−y. For p=0.1 and y=1, it is −0.9: the update should increase the logit. With squared error through a sigmoid, an additional factor p(1−p) attenuates this gradient when the prediction saturates.

The coefficient relation is log(p/(1−p)) = w·x+b. Holding other features fixed, increasing feature j by one unit multiplies the modeled odds by exp(w_j). It does not multiply the probability by that amount and does not establish a causal effect. Correlated features and feature transformations complicate interpretation.

Log-loss is a proper scoring rule, but a restricted, regularized, finite-data model can still be miscalibrated. Class weighting, resampling, and deployment shift also change the probability interpretation. Check calibration rather than granting it to a model family.

Multinomial logistic regression uses a jointly normalized softmax for mutually exclusive classes. One-vs-rest uses separate binary models whose raw probabilities need not sum to one. More generally, generalized linear models specify a response distribution and a link from its mean to a linear predictor: Bernoulli/logit for binary data, Poisson/log for counts, and Gamma/log for positive skewed outcomes are useful examples. Check assumptions such as Poisson mean–variance equality before adopting a likelihood.

Check yourself: a coefficient is ln(2). If the old probability is 0.2, what is the new probability after increasing that feature by one? Solution: old odds are 0.2/0.8 = 0.25; new odds are 0.5; new probability is 0.5/1.5 = 1/3, not 0.4.

3. Let trees discover conditional rules

A decision tree divides feature space using rules such as “amount > 100” and then “account age < 7 days.” Each leaf predicts a value or a class distribution. The interactions are explicit: the age rule applies only on one side of the amount split.

For classification, Gini impurity is 1−Σ p_k². A node containing four positives and four negatives has impurity 0.5. Suppose a split makes two four-example children with positive fractions 3/4 and 1/4. Each child has impurity 1−(3/4)²−(1/4)² = 0.375; weighted impurity decreases by 0.125. Compare this with other candidate splits, then apply stopping or pruning constraints. Entropy is another classification criterion; squared-error reduction is common for regression.

An unconstrained tree can fit noise. Depth, minimum leaf size, feature sampling, and pruning restrict that flexibility. A model cannot distinguish two identical feature vectors with conflicting targets perfectly, however deep the tree.

Averaging trees: bagging and random forests

Bagging fits trees to bootstrap samples and averages their predictions. Random forests additionally sample candidate features at each split, reducing correlation between trees. If M predictors each have variance σ² and pairwise correlation ρ, their average has variance σ²[ρ + (1−ρ)/M]. More trees reduce the independent part; correlation creates a floor. “Bagging reduces variance” is useful intuition, not a promise of unchanged bias for every construction.

A training row is omitted from a size-n bootstrap sample with probability (1−1/n)^n, approaching e⁻¹ ≈ 36.8%. Predictions from trees that omitted it give an out-of-bag estimate. OOB estimation does not fix time leakage, repeated entities, or preprocessing that already used all labels.

Impurity-based feature importance favors features with many split opportunities. Held-out permutation importance is often easier to interpret, but correlated features can share or mask importance. Neither is a causal explanation.

Adding trees: gradient boosting

Boosting builds an additive predictor, F_(m+1)(x) = F_m(x) + ηh_m(x). The next tree fits a direction that reduces the current loss. For half squared error, that direction is the residual y−F_m(x); for logistic loss on logits it is y−p, not a binary error flag.

For targets [2, 4] and current predictions [1, 3], residuals are [1, 1]. A constant tree h=1 with η=0.5 moves predictions to [1.5, 3.5]. The summed half squared error drops from 1 to 0.25. Real trees approximate many residuals with a limited set of leaves. Tune learning rate together with the number and complexity of trees, and stop using a representative validation set.

Optional depth: what the main boosting libraries change

XGBoost uses a regularized second-order approximation. Let g_i and h_i be the loss gradient and Hessian with respect to the current scalar prediction. In leaf j, let G_j = Σg_i and H_j = Σh_i. With L2 leaf penalty λ, the approximate optimal leaf value is −G_j/(H_j+λ). For a proposed split, the gain is:

0.5[G_L²/(H_L+λ) + G_R²/(H_R+λ) − G²/(H+λ)] − γ

Here γ is a per-leaf complexity cost. For a leaf with G=−6, H=3, λ=1, the value is 1.5 before any learning-rate shrinkage. The approximation, valid Hessians, and solver constraints matter; second-order information does not guarantee fewer trees on every dataset. Sparsity-aware missing-value directions and efficient exact/approximate/histogram tree methods address engineering costs.

LightGBM is associated with histogram-based splitting and best-first leaf growth. Histogramming reduces split candidates after bin assignment. Its published techniques include gradient-based one-side sampling (GOSS) and exclusive feature bundling (EFB). Which are active depends on configuration. Limit leaves and minimum leaf data when narrow branches overfit; there is no universal speed multiple relative to another library.

CatBoost develops ordered target statistics and ordered boosting to reduce target leakage and prediction shift. A training row's target must not leak into its own categorical encoding. Ordered schemes use preceding rows in a permutation with suitable priors; prediction-time encodings are built from training data, not new targets. Symmetric trees use the same split across a level. Library modes differ, so inspect the chosen configuration rather than assuming every training run uses the same procedure.

Trees are strong candidates for heterogeneous tabular inputs, but missing-value and categorical support depends on the implementation. They do not inherently understand timestamps as sequences, raw images as spatial signals, or arbitrary IDs as shared semantic features. Compare them with a regularized linear baseline and, when appropriate, learned representations on the same honest split.

4. Optional depth: a kernel changes the geometry

An SVM scores f(x)=w·x+b and trades margin size against violations. For labels y_i∈{−1,+1}, its soft-margin objective can be written:

0.5‖w‖₂² + C Σ max(0, 1−y_i f(x_i))

The hinge term is zero for a correctly signed score with margin at least 1. A point with y=+1 and f(x)=0.2 still contributes 0.8; correct classification alone does not mean sufficient margin. C controls the penalty on violations relative to the weight penalty. Its numerical meaning also depends on whether the loss is summed or averaged.

The dual depends on dot products between training examples. Replacing x_i·x_j by a valid positive-semidefinite kernel K(x_i,x_j)=φ(x_i)·φ(x_j) lets the model use an implicit feature map φ. The resulting score is a weighted sum over support vectors with nonzero dual coefficients. Convexity gives global optimal objective values under the problem's assumptions; it does not guarantee unique dual coefficients or superior test accuracy.

A linear kernel uses ordinary dot products. A polynomial kernel adds polynomial interactions. The RBF kernel exp(−γ‖x−x′‖₂²), for γ>0, measures proximity in a feature space that can be infinite-dimensional. With γ=1, identical points have similarity 1 and points at distance 2 have similarity exp(−4) ≈ 0.0183. Feature scaling therefore changes what “nearby” means. A sigmoid-shaped similarity is not a valid positive-semidefinite kernel for arbitrary parameter choices.

Large γ produces localized influence; large C emphasizes fitting margin violations. Tune both on a leakage-safe validation scheme, usually on logarithmic grids or equivalent searches. Linear solvers can exploit sparse features. Kernel methods may require quadratic storage/work in sample count and can be expensive when many support vectors remain; exact costs depend on solver and kernel. Random features or kernel approximations offer intermediate choices.

A one-class SVM models a boundary around training data for novelty detection. Its ν parameter bounds the training error fraction from above and the support-vector fraction from below under the standard formulation. It is not a guaranteed deployment anomaly rate. Compare against Isolation Forest and reconstruction-based models using meaningful anomalies or operational review outcomes.

5. Find groups without inventing labels

Clustering begins by choosing features and a distance that reflect the purpose of grouping. A customer cluster is not automatically a stable personality type or a useful marketing segment.

k-means minimizes summed squared distance to k centroids. On one-dimensional points [0, 2, 8, 10], initialize centroids at 0 and 10. Assignment gives {0,2} and {8,10}; recomputing means gives 1 and 9. The squared-distance sum falls from 8 to 4. Lloyd's assign/update steps do not increase this objective, but can converge to different local solutions. k-means++ improves initialization; its guarantee concerns expected objective quality, not finding meaningful real-world categories.

DBSCAN connects dense neighborhoods using a radius ε and a minimum neighborhood count. Core points can expand clusters; border points need not. It can represent nonconvex clusters and leave noise points unassigned, but one global density scale may fail when densities differ. HDBSCAN explores a density hierarchy; it does not remove the need for a suitable metric or validation.

Agglomerative clustering repeatedly merges clusters according to single, complete, average, or another linkage. Ward linkage chooses merges that increase within-cluster squared error least and is tied to Euclidean geometry. The dendrogram makes granularity visible; memory and computation can limit large datasets.

For large data, minibatch k-means is a candidate, not an automatic answer. Standardization, meaningful categorical representations, sampling, and optional dimensionality reduction come first. PCA may remove noise but can also discard low-variance signals useful to the task. Compare stability across seeds and samples, inspect cluster profiles, and measure whether using the segments improves the intended decision. Silhouette and inertia evaluate geometry, not business usefulness.

6. A bridge to learned word representations

A static embedding assigns a vector to a vocabulary word. Instead of treating “cat” and “kitten” as unrelated one-hot IDs, training can place them near each other when their corpus contexts support that relationship. This is learned statistical structure, not a definition of meaning.

Word2Vec includes CBOW, which predicts a center word from nearby words, and skip-gram, which predicts contexts from a center word. Full vocabulary softmax can be costly. Skip-gram negative sampling instead maximizes, for a positive pair and k sampled negatives:

log σ(v_center·u_context) + Σ log σ(−v_center·u_negative)

The minimized loss is the negative of that expression. For a positive dot product of zero, σ=0.5: increasing the dot product improves the positive term. For a negative pair at zero, decreasing it improves the negative term. Center and context vectors are separate learned tables. This is a different objective from the full normalized softmax, not an exact inexpensive evaluation of it. A unigram distribution raised to 3/4 flattens the sampling distribution relative to raw unigram counts.

Hierarchical softmax factorizes a word probability along a vocabulary tree. GloVe fits vector dot products plus biases to log co-occurrence counts using weighted least squares. FastText shares character n-gram vectors across words, allowing a vector to be constructed for an unseen word from subword features; that does not guarantee a good representation for an arbitrary unseen string.

Static vectors cannot distinguish the occurrence of “bank” beside “river” from the same word beside “loan.” A contextual encoder changes token representations with the surrounding text. Pooling and retrieval training are additional choices: a contextual token encoder is not automatically a good sentence retriever. Static lookup remains useful under tight compute budgets; corpus size, vocabulary coverage, and measured task quality decide whether it suffices.

Check yourself: can a stable, well-separated clustering or a convincing word analogy prove that the representation is suitable for a credit decision? Solution: no. Neither evaluates that decision, protects against leakage, establishes causality, or measures error on the deployment population.

Put the pieces together

A defensible experiment starts with a simple baseline and a deployment-shaped split. Fit imputers, scalers, feature selectors, encoders, and calibration maps without using held-out targets. Report error counts, uncertainty, and important slices alongside aggregate metrics. Compare quality, latency, memory, maintainability, and explanation requirements; a more complex model must earn its added cost.

Continue to Deep Learning Basics to learn representations through layers, or Embeddings & Retrieval to build and evaluate a search pipeline. Revisit Information Theory for entropy and cross-entropy, and Numerical Computing for stable linear solves and probability calculations.

Sources and further reading

• scikit-learn: metrics and scoring — definitions, averaging conventions, probability and ranking metrics.
• scikit-learn: supervised learning guide — linear models, SVMs, trees, ensembles, and calibration.
• scikit-learn: clustering guide — objectives, assumptions, evaluation, and algorithm boundaries.
• Mikolov et al.: Distributed Representations of Words and Phrases — skip-gram negative sampling and distributed word representations.