↗ Linear Algebra for ML

You open an attention tutorial. The first line is QKᵀ. Everyone seems comfortable with it. You are still wondering which way the matrices go.

This lesson starts there: what the numbers represent, which operations make sense, and how to check your answer. You only need arithmetic and basic algebra.

What you will learn: name tensor axes, multiply matrices, explain lost directions, and solve a small projection problem. Work through sections 1–5 first. Sections 6–8 are optional bridges to PCA, attention and LoRA; you can return when those models appear. Keep a scrap of paper nearby—the examples fit on it.

1. Vectors: a list with a job

A scalar is one number. A vector is an ordered list of numbers. A matrix is a rectangular table of numbers. A tensor generalizes these to any number of axes.

Suppose one training example has two features: hours studied and practice questions attempted. We could write it as x = [2, 5]. The order matters: swapping the entries changes what the example means.

An embedding is also a vector, though its coordinates are learned features rather than named measurements. A batch stacks examples into a matrix:

Object	Shape	Meaning
One example x	2 coordinates	Two features
Dataset X	100 × 2	100 examples, two features each
Token embeddings H	12 × 64	12 tokens, 64 features per token
Batched token embeddings	8 × 12 × 64	Eight sequences of those tokens

Throughout this lesson, rows are examples or tokens; columns are features. Other sources use column vectors. Both conventions work; check the shapes before borrowing a formula.

Adding vectors adds matching coordinates: [2, 5] + [1, −2] = [3, 3]. Scaling multiplies every coordinate: 2[2, 5] = [4, 10]. These two operations form a linear combination. Mixing directions this way is the basic move behind a linear layer.

2. Dot products: multiply, then add

The dot product of two equally sized vectors is the sum of their coordinate-wise products:

x · w = x₁w₁ + x₂w₂ + … + x_dw_d

Take x = [2, 5] and w = [3, −1]. Then:

x · w = 2 × 3 + 5 × (−1) = 1

With a bias b = 4, a linear model predicts x · w + b = 5. Each weight tells us how one feature contributes, holding the others fixed. The bias shifts the result. Adding a bias makes the map affine, though ML libraries usually call the whole operation a linear layer.

The same operation can measure alignment. Two vectors pointing in similar directions have a positive dot product; perpendicular vectors have dot product zero. But magnitude matters too.

Length, distance and cosine similarity

The L2 norm is a vector's length: ||x||₂ = √(x₁² + … + x_d²). For [3, 4], it is 5. The Euclidean distance between x and y is ||x − y||₂.

Cosine similarity removes length from the dot product:

cos(x, y) = (x · y) / (||x||₂ ||y||₂)

For x = [1, 0] and y = [3, 4], the dot product is 3 and cosine similarity is 3/5 = 0.6. Doubling y doubles the dot product to 6 but leaves cosine similarity at 0.6. A zero vector has no direction, so its cosine similarity is undefined; a retrieval system needs an explicit zero-vector policy.

Why you care: for unit-normalized embeddings, dot product and cosine similarity are equal. Their squared Euclidean distance is 2 − 2 cos(x, y), so all three give the same nearest-neighbor ranking, with distance sorted ascending. For unnormalized vectors, the rankings can differ. Use the metric your embedding model was trained for.

Try it: are [1, 2] and [2, −1] perpendicular?

Yes. Their dot product is 1 × 2 + 2 × (−1) = 0. Neither vector is zero, so they form a right angle. This does not mean their corresponding features are statistically independent.

3. Matrices: several dot products at once

If A has shape m × d and B has shape d × n, then AB has shape m × n. The inner dimensions must match. Each output entry is one row of A dotted with one column of B.

Let's transform two examples, each with two features:

X = [[2, 1], [0, 3]] W = [[1, −1], [2, 0]]

Output entry	Row × column	Result
First row, first column	2 × 1 + 1 × 2	4
First row, second column	2 × (−1) + 1 × 0	−2
Second row, first column	0 × 1 + 3 × 2	6
Second row, second column	0 × (−1) + 3 × 0	0

So XW = [[4, −2], [6, 0]]. W creates two new features for each example. Matrix multiplication is different from element-wise multiplication, which would multiply matching cells without summing.

The transpose swaps rows and columns: if X is 100 × 2, Xᵀ is 2 × 100. Transposing does not undo a transformation. An inverse, when it exists, does: W⁻¹W = I, where I is the identity matrix. Only square, full-rank matrices have an ordinary inverse.

Order matters. AB and BA can have different shapes, different values, or one may not exist at all. A reliable debugging habit is to write the shape next to every intermediate result.

A dense layer, with the shapes left in

X: [batch, d_in]
W: [d_in, d_out] b: [d_out]
Y = XW + b: [batch, d_out]

The bias is added to each row. A framework may store the weights transposed; PyTorch's Linear does. Its computation is equivalent to this convention.

Try it: X is 32 × 128 and W is 128 × 64. What shape is XW?

32 × 64. Each of 32 examples gets 64 output features. WX is not defined for these shapes: its inner dimensions would be 64 and 32.

4. Basis and rank: how many directions survive?

The span of some vectors is every linear combination you can make from them. A basis is a set of independent vectors that spans a space. Independent means no member can be built by combining the others.

[1, 0] and [0, 1] form a basis for the 2D plane. [1, 2] and [2, 4] do not: the second is just twice the first. Their span is a line.

The rank of a matrix is the number of independent columns (also the number of independent rows). It cannot exceed its smaller dimension.

A = [[1, 2], [2, 4]]

A has rank 1. Its second column repeats the first at twice the scale. With column-vector convention, A[2, −1]ᵀ = [0, 0]ᵀ: a nonzero input is lost. Such inputs form its null space. There is no inverse that can recover every original input.

Solve a system before reaching for an inverse

Let A=[[2,1],[1,−1]] and solve Az=[5,1]ᵀ. The equations are 2z₁+z₂=5 and z₁−z₂=1. Adding them gives 3z₁=6, so z=[2,1]ᵀ. Multiplying A by that solution reconstructs [5,1]ᵀ. The columns are independent, so this square system has one solution for every right-hand side.

Contrast A=[[1,2],[2,4]]. Right-hand side [3,6]ᵀ has infinitely many solutions (z₁+2z₂=3); [3,7]ᵀ has none because the second row demands twice the first. Least squares still finds a closest representable output when exact agreement is impossible.

Check: does full column rank guarantee every rectangular system has an exact solution?

No. A tall matrix with independent columns has a unique least-squares solution, but a target outside its column space cannot be matched exactly. Full column rank concerns uniqueness of weights, not whether every target is representable.

Why you care: duplicated features create rank deficiency in regression. Low-rank layers deliberately restrict a transformation to fewer directions. Neither the number of parameters nor the matrix's shape alone tells you its rank.

5. Projection: the closest answer you can represent

Suppose the only outputs your model can produce lie on the line in direction u = [1, 1]. Your target is y = [3, 1]. Which point on that line comes closest?

The orthogonal projection is:

ŷ = ((y · u) / (u · u)) u = (4 / 2)[1, 1] = [2, 2]

The residual is y − ŷ = [1, −1]. It is perpendicular to u because their dot product is zero. Moving along the line cannot reduce that remaining error.

Least-squares regression does the same thing with more directions. It chooses w to minimize ||Xw − y||₂²; the best prediction is the projection of y onto the column space of X. At an optimum, Xᵀ(Xw − y) = 0.

If X has independent columns, the solution can be written w = (XᵀX)⁻¹Xᵀy. For computation, solve the least-squares problem with QR or SVD instead of explicitly forming this inverse. Nearly dependent columns make the problem sensitive to small changes; forming XᵀX worsens the condition number. Regularization can help, but its strength is a modeling choice.

If X is rank-deficient, several weight vectors may make the same best prediction. The pseudoinverse chooses the solution with the smallest L2 norm. More data only helps identify those weights if it adds useful independent information.

6. Optional depth: eigenvectors, SVD and PCA

This section is a bridge to dimensionality reduction. The earlier dot-product and shape checks are the essentials to get comfortable with first.

An eigenvector v of a square matrix C is a nonzero direction satisfying Cv = λv. The matrix scales that direction by its eigenvalue λ. A negative eigenvalue also flips the direction. Not every real square matrix has a full basis of real eigenvectors.

For C = [[2, 1], [1, 2]]:

• C[1, 1]ᵀ = [3, 3]ᵀ: eigenvalue 3.
• C[1, −1]ᵀ = [1, −1]ᵀ: eigenvalue 1.

A covariance matrix records each feature's variance on the diagonal and how pairs of features vary together off the diagonal. If C is such a matrix, its eigenvectors describe directions of variation in the data. PCA selects orthonormal directions with the largest variance. In this example the first direction, normalized to [1/√2, 1/√2], explains 3 / (3 + 1) = 75% of total variance.

Singular value decomposition (SVD) works for rectangular matrices too:

X = UΣVᵀ

For X of shape m × n, the compact form uses k = min(m, n): U is m × k, Σ is k × k, and V is n × k. The columns of U and V are orthonormal; Σ contains nonnegative singular values. Keeping only the largest r singular values gives X_r = U_rΣ_rV_rᵀ, a best rank-at-most-r approximation under squared reconstruction error.

To compute PCA with examples in rows: subtract the training-set mean from every feature, then take the right singular vectors of the centered matrix as the principal directions. Fit any feature scaling on training data too. Apply those same transformations to validation and test data. PCA maximizes variance, so a useful low-variance signal can still be discarded; check downstream performance.

7. Optional bridge: read an attention formula

In one self-attention head, suppose there are n = 3 tokens, each query and key has d_k = 2 features, and each value has d_v = 4 features.

Operation	Shapes	What it means
QKᵀ	(3 × 2)(2 × 3) → 3 × 3	Each query dotted with every key
S = QKᵀ / √2	3 × 3	Scaled attention scores
P = softmax(S), row by row	3 × 3	One distribution over keys for each query
PV	(3 × 3)(3 × 4) → 3 × 4	A weighted mixture of value vectors per token

If one query is [1, 0] and the keys are [1, 0], [0, 1], [1, 1], its raw scores are [1, 0, 1]. Dividing by √2 and applying softmax gives approximately [0.401, 0.198, 0.401]. Those weights sum to one. The output mixes the three value vectors with those weights.

This is the unmasked case. Causal attention masks future positions before softmax. Softmax itself is nonlinear; attention is more than a single matrix multiplication.

Now the attention walkthrough has somewhere to land. You can also explore the attention weights in the attention playground.

8. Optional bridge: low rank and LoRA

Suppose a dense weight update ΔW is 4096 × 4096. Storing every entry takes 16,777,216 parameters.

Instead, learn two matrices A of shape 4096 × 8 and B of shape 8 × 4096, and set ΔW = AB. That takes 65,536 parameters, or 1/256 as many. The update's rank is at most 8.

That is the central idea behind LoRA: keep the original weight matrix frozen and learn a low-rank update. LoRA learns the factors during training; it does not have to compute an SVD of the original weights. The base weights and activations still use memory, so 256× fewer update parameters does not mean 256× lower total training memory. Some sources name the factors in the opposite order; the shapes are what matter.

Put the pieces together

Before moving on, try explaining these without looking up the formula:

1. Why can a longer vector have a larger dot product but the same cosine similarity?
2. Why is QKᵀ an n × n table for n tokens in self-attention?
3. What information is lost when a matrix maps a plane onto a line?
4. Why is PCA fit on the training split?
5. What restriction does a rank-8 LoRA update place on ΔW?

Check your five explanations

1. A dot product scales with length; cosine divides that length out. 2. Every one of n query rows scores every one of n key rows. 3. Inputs differing along the null space become indistinguishable. 4. Fitting PCA on held-out data leaks distribution information into the representation. 5. The update changes at most eight independent directions, while the frozen base matrix may remain full rank.

Continue to probability and statistics for uncertainty, then calculus for learning weights. Numerical computing develops stable solving and conditioning. The Practice questions link gives additional worked shape and projection exercises.

Go deeper

• 3Blue1Brown: Essence of Linear Algebra — visual intuition for transformations, span and eigenvectors.
• MIT OpenCourseWare: Linear Algebra, 18.06 — lectures and problems for a fuller course.
• Mathematics for Machine Learning — the free textbook connects linear algebra to probability, optimization and ML.