Calculus and Optimization

DifferentiationDifferentiation and integration are usually taught as opposites — one measures instantaneous rate of change, the other accumulates area under a curve — and for computing purposes, differentiation is the one that matters most: almost every learning algorithm on this site is, underneath, a search for the input that makes some function as small (or large) as possible, and a derivative is the tool that tells the search which direction to move.

Differentiation and Integration, Briefly

AA derivative f'(x) gives the instantaneous slope of f at x — how fast the output changes for a tiny change in input, right at that specific point. An integral gives the accumulated area between a curve and the axis over some range, and by the Fundamental Theorem of Calculus, integration and differentiation are inverse operations of each other. Most of what follows on this page only needs differentiation; integration matters most in areas like probability (covered on Probability and Statistics for Computing, where a continuous probability distribution's total area must integrate to exactly 1).

Partial Derivatives and the Gradient

AA function of several variables — f(x, y), say — has a separate partial derivative with respect to each input, written ∂f/∂x, found by differentiating with respect to that one variable while treating every other variable as a constant. Collecting all of a function's partial derivatives into a single vector gives the gradient, ∇f, and the gradient has a genuinely useful geometric meaning: it points in the direction of steepest increase of f, at the point it's evaluated. Consequently, minus the gradient points in the direction of steepest decrease — which is precisely the fact gradient descent exploits: repeatedly step a little in the direction of the negative gradient, and the function's value reliably decreases, step after step, until it reaches a point where the gradient is (near) zero — a minimum, where there's no longer a downhill direction to move in.

The Jacobian and the Hessian

WhereWhere a function outputs several values at once (not a single number but a vector), the Jacobian matrix collects every output's partial derivative with respect to every input — row i, column j holds ∂(output i)/∂(input j). It's the natural multivariable generalisation of a single derivative, and it's exactly the object backpropagation is built from, chained across layers: each layer's Jacobian describes how its outputs respond to its inputs, and the chain rule composes those Jacobians together to work out how the network's final error responds to a weight buried many layers back.

The Hessian goes one derivative further — the matrix of second partial derivatives of a function that outputs a single number, ∂²f/∂xᵢ∂xⱼ — and it describes the local curvature of f: how the gradient itself is changing, not just its current direction. A Hessian that's positive-definite at a point where the gradient is zero confirms that point is a genuine local minimum, not a saddle point (where the function curves up in one direction and down in another) or a maximum — the gradient alone can't distinguish these cases, since all three have a zero gradient; the Hessian's curvature information is what tells them apart.

The Newton-Raphson Method

GradientGradient descent takes a step proportional to the gradient, scaled by a learning rate chosen somewhat by trial and error. The Newton-Raphson method instead uses curvature information (the second derivative, or the Hessian in multiple dimensions) to jump much more directly toward a root or an extremum: for finding a root of f, the update is x_new = x - f(x)/f'(x) — geometrically, follow the tangent line at the current point down to where it crosses zero, and jump there directly rather than taking a small step.

Finding the root of f(x) = x^2 - 2 (i.e. finding sqrt(2)), starting from x=1:
x1 = 1 - (1^2 - 2) / (2*1)       = 1 - (-1/2)     = 1.5
x2 = 1.5 - (1.5^2 - 2) / (2*1.5) = 1.5 - (0.25/3) = 1.41667
x3 = ...                                          = 1.41421...  (converging fast on sqrt(2) = 1.41421356...)

Newton-Raphson converges far faster than gradient descent near a solution — often doubling the number of correct digits with every step — but each step is more expensive to compute (it needs second-derivative information) and it can fail badly if the initial guess is far from a solution or the function's curvature is unhelpful there. This is the same speed-versus-robustness trade-off that shows up constantly across optimisation methods: a method that uses more information about the function converges faster when that information is reliable, and fails more surprisingly when it isn't.

"Fails badly" undersells what can actually happen once a function has more than one root. Cayley noticed the problem as early as 1879, trying to extend Newton-Raphson to complex polynomials of degree higher than two¹: which root a given starting point converges to isn't a smooth function of the starting point at all. Colour every starting point in the complex plane by which root it eventually converges to, for a polynomial as simple as f(z) = z³ - 1 (three roots, evenly spaced around a circle), and the three resulting basins of attraction don't meet along clean boundary lines — they interleave in a fractal pattern, so that two starting points a hair's breadth apart can converge to two completely different roots. That's genuine chaos in the technical sense (sensitive dependence on initial conditions), not just "sometimes it fails" — a method that converges in one step from x=1.4 can diverge, or converge to an entirely different answer, from x=1.40001, with no way to predict which without just running it. Interval Arithmetic covers the robust alternative — bisection, which trades Newton-Raphson's speed for a genuine correctness guarantee at every step.

Taylor Expansion

AA Taylor expansion approximates a complicated function near a specific point using a polynomial built from the function's derivatives at that point: f(x) ≈ f(a) + f'(a)(x-a) + f''(a)(x-a)²/2! + .... Truncating after the first-derivative term gives exactly the straight-line (tangent) approximation Newton-Raphson's geometric picture relies on; truncating after the second-derivative term gives a curved (quadratic) approximation, which is the basis for optimisation methods that use Hessian information to model a function's local curvature rather than just its local slope.

Numerical Solutions

NotNot every differential equation or optimisation problem has a clean closed-form solution solvable by algebra alone — most real, messy ones don't. Numerical methods trade an exact algebraic answer for an approximate one computed by repeated, mechanical steps: Newton-Raphson above is one example; numerically integrating a differential equation by taking many small steps and accumulating the change at each one (Euler's method, in its simplest form) is another. The general theme across numerical calculus is the same trade already seen above: exchange a closed-form guarantee for an iterative process that gets arbitrarily close to the true answer, provided it's given enough steps and doesn't run into instability along the way.

References