∂ Calculus and Optimization

You have a prediction and a target. They disagree. Which weight should change, in which direction, and by how much? Calculus supplies the local direction; optimization chooses how to use it.

Before you start: be comfortable with functions, powers and the vector shapes in linear algebra. The stochastic-gradient section uses averages from probability. Your goal is to compute one update by hand, trace it through a small computation graph, and explain why a lower training loss is not a guarantee about new data.

1. A derivative measures local change

For a scalar function f(w), the derivative is the limiting slope:

f′(w) = lim as h → 0 of [f(w+h) − f(w)] / h

For f(w) = w², expanding (w+h)² gives a difference quotient of 2w+h. Its limit is 2w. At w=3, a small change Δw gives approximately Δf ≈ 6Δw. This is a local approximation: changing w from 3 to 4 changes f by 7, not 6.

Function	Derivative	Condition
Constant c	0	c does not depend on w
wⁿ	n wⁿ⁻¹	On the function's differentiable domain
exp(w)	exp(w)	All real w
ln(w)	1/w	w > 0
a f(w) + b g(w)	a f′(w) + b g′(w)	a,b constant

An integral accumulates small contributions. For example, a continuous probability over [a,b] is the integral of its density on that interval. Differentiation and integration are linked by the fundamental theorem of calculus under suitable continuity conditions. Most training code needs derivatives; expectations and probability densities are where integrals reappear.

2. One loss, one update

Take a one-feature model ŷ = wx. Let x=2 and target y=6. Use half squared error:

L(w) = ½(wx − y)²

Start at w=1. Prediction is 2, residual is −4, and loss is 8. The derivative is dL/dw = (wx − y)x = −8. Gradient descent updates w ← w − η dL/dw, where η is the learning rate.

With η=0.1, the new weight is 1.8, prediction 3.6, loss 2.88. The negative gradient pointed toward an improvement. With η=1, the new weight is 9, prediction 18, loss 72. A correct direction with an oversized step can make things much worse.

Check: with w=1 and η=0.25, what happens?

w becomes 1 − 0.25(−8) = 3. Prediction is 6 and loss is zero. This exact jump works for this particular quadratic and feature scale; it is not a generally safe learning rate.

3. The chain rule is backpropagation's basic move

Break the loss into nodes: z=wx, r=z−y, L=½r². Then:

dL/dw = (dL/dr)(dr/dz)(dz/dw) = r × 1 × x

Each operation knows its local derivative. Backpropagation multiplies these along a path and adds contributions when several paths meet. It is reverse-mode automatic differentiation, not a finite-difference estimate and not a symbolic simplification of the entire network.

Add a bias and a nonlinearity: z=wx+b, a=ReLU(z), L=½(a−y)². For z>0, dL/dw=(a−y)x and dL/db=a−y. For z<0, ReLU's local derivative is zero, so this example sends no gradient to w or b. At z=0 the mathematical derivative is undefined; frameworks use a chosen convention, commonly zero.

For a shared parameter used twice, both uses matter. If f(w)=w²+w, then f′(w)=2w+1. Forgetting a branch is a common manual derivation error.

4. Many parameters: gradients and shapes

The gradient of a scalar loss is a vector of partial derivatives. A partial derivative varies one coordinate while holding the others fixed. For X of shape n × d, weights w of shape d × 1, and targets y of shape n × 1:

L = (1/(2n)) ‖Xw − y‖₂²

∇w L = (1/n) Xᵀ(Xw − y) shape: d × 1

The gradient has the same shape as w. The factor 1/n matters: summing losses instead of averaging them changes the gradient scale and therefore the effective step size.

For a vector-valued function, the Jacobian contains every output's derivative with respect to every input. Reverse-mode differentiation usually computes vector–Jacobian products without materializing that whole matrix. For a dense layer Y=XW+b, if G=∂L/∂Y has shape n × k, then ∂L/∂W=XᵀG has shape d × k, ∂L/∂X=GWᵀ has shape n × d, and ∂L/∂b sums G over the n rows.

Check: X is 32 × 128, W is 128 × 64, and G is 32 × 64. What is XᵀG?

A 128 × 64 gradient for W. Multiplication (128 × 32)(32 × 64) sums each weight's contribution over the batch. If the upstream loss is already averaged, do not divide by the batch size a second time.

5. Curvature explains step-size trouble

A derivative tells you the slope; a second derivative tells you how the slope changes. For our loss L=½(2w−6)², L″=4. The error from the optimum, e=w−3, evolves as e_next=(1−4η)e. Convergence requires |1−4η|<1, or 0<η<0.5. At η=0.5 it oscillates without shrinking; beyond that it diverges unless it started at the optimum.

In many dimensions, the Hessian is the matrix of second partial derivatives. A smooth convex function has no suboptimal local minima, but convexity alone does not promise rapid convergence or a unique minimizer. Neural-network losses are generally nonconvex. A zero gradient can be a local minimum, maximum, saddle point, or flat region.

Feature scaling changes curvature. An elongated loss valley forces a basic optimizer to take small steps along the steep direction while progressing slowly along the flat direction. Standardizing features using training statistics can help. Numerical computing explains conditioning and finite precision.

6. Full batches, mini-batches and momentum

For an average loss over N examples, a uniformly sampled mini-batch gives an unbiased estimate of the full gradient under the usual sampling setup. Its noise depends on batch size and correlations. Larger batches often reduce gradient variance, but they cost memory and do not guarantee better generalization.

SGD uses the current mini-batch gradient g_t. One momentum convention is v_t=βv_(t−1)+g_t, w_(t+1)=w_t−ηv_t. Momentum carries some previous direction, reducing oscillation in some landscapes. Other implementations include a (1−β) factor; compare conventions before copying a learning rate.

Adam keeps exponential moving averages of the gradient and squared gradient, corrects their initial bias, and rescales each coordinate by a square-root second-moment estimate plus ε. That is not an exact inverse Hessian and does not remove the need to tune the learning rate. AdamW applies weight decay separately from the adaptive gradient update; adding an L2 penalty inside Adam is generally not equivalent.

Learning-rate warmup, decay, clipping and optimizer choice solve different problems. Clipping caps large finite gradients. It does not repair a wrong label, a detached graph or a NaN that has already occurred.

7. Optional depth: constraints and second-order steps

For a nonnegative weight constraint, a projected-gradient step first takes an ordinary step, then projects back into the allowed set: w ← max(0, w−ηg), coordinate by coordinate. A constraint is not the same as an L2 penalty; one restricts feasible values, the other changes their cost.

A Newton step solves HΔ = −g, then updates w ← w+Δ. For an invertible positive-definite quadratic Hessian, it reaches the unconstrained minimum in one exact step. General losses need safeguards such as damping or line search. Explicit Hessians are huge for neural networks, and indefinite Hessians can point the wrong way. Quasi-Newton methods approximate curvature information with a different memory/computation tradeoff.

8. Check a gradient before trusting a run

On a tiny deterministic example, compare automatic differentiation with the central difference:

g_i ≈ [L(w+h e_i) − L(w−h e_i)] / (2h)

Here e_i changes only coordinate i. Use float64, disable dropout and stochastic data changes, and try several moderate h values. Very large h measures nonlocal behavior; very small h loses the difference to rounding. Avoid nondifferentiable boundaries such as ReLU at zero.

Check: your gradient is zero and the loss is high. Is the model trained?

No. Check for inactive ReLUs, saturation, frozen parameters, a detached graph, a nondifferentiable operation, or a stationary point that is not a good minimum. Inspect intermediate activations and gradients on one small batch. Low gradient magnitude is a diagnostic, not a quality certificate.

Next, information theory explains the losses used for probabilities, and numerical computing explains why mathematically equivalent implementations behave differently. Deep learning basics puts the gradient into a complete training loop.

Sources and further practice

• Mathematics for Machine Learning — vector calculus and continuous optimization chapters.
• Dive into Deep Learning: optimization — worked optimizer implementations.
• PyTorch: autograd mechanics — graph execution and differentiation conventions.
• Decoupled Weight Decay Regularization — AdamW and its distinction from an L2 loss penalty.