Linear Algebra for Computing

VectorsVectors and matrices show up constantly in computing — a colour is three numbers, an image is a grid of them, a neural network's weights are a matrix, a 3D rotation is a matrix multiply — because linear algebra is the mathematics of transforming and combining many numbers at once, uniformly. Strang's textbook is the standard, widely-used introduction to the subject and the one this page follows¹.

Matrices and Basic Operations

AA matrix is a rectangular grid of numbers. Addition and scalar multiplication work element-by-element, exactly as expected. Matrix multiplication is the operation that actually does the interesting work, and it isn't element-wise: multiplying an m×n matrix by an n×p matrix produces an m×p matrix, where each output entry is the dot product of a row from the first matrix and a column from the second:

[1 2]   [5 6]   [1*5+2*7  1*6+2*8]   [19 22]
[3 4] x [7 8] = [3*5+4*7  3*6+4*8] = [43 50]

Matrix multiplication is not commutative — A×B generally doesn't equal B×A, and for non-square matrices one order may not even be defined at all. This matters immediately once matrices represent transformations (rotate, then scale is a different result from scale, then rotate) rather than just numbers being combined.

Determinants and Inverses

AA matrix's determinant is a single number summarising something essential about the transformation it represents: how much it scales area (in 2D) or volume (in 3D), and its sign indicates whether it flips orientation. A determinant of zero means the transformation collapses space into a lower dimension — a 2D transformation that squashes the whole plane onto a single line — and exactly when the determinant is zero, the matrix has no inverse: there's no way to undo a collapse that has already thrown information away, since multiple different inputs now map to the same output.

When the inverse does exist, A⁻¹A = I (the identity matrix — the "do nothing" transformation), and it's the tool for solving a system of linear equations written in matrix form: Ax = b becomes x = A⁻¹b. In practice, computing a full matrix inverse is rarely how a system of equations actually gets solved — Gaussian elimination (systematically combining rows to zero out entries below the diagonal, then back-substituting) reaches the same answer more directly and more numerically stably, and is the method underlying most real linear-equation solvers — "more numerically stably" is not a throwaway qualifier here: Trefethen and Bau's standard treatment of linear algebra for computation covers, in careful detail, how floating-point rounding error actually propagates through each of these algorithms, and why the textbook-equivalent method on paper is not always the one a reliable implementation should actually run².

Eigenvalues and Eigenvectors

ForFor most vectors, applying a matrix A changes both their length and their direction. An eigenvector is one of the special exceptions: a vector whose direction A leaves unchanged, only scaling it by some factor — the corresponding eigenvalue, written Av = λv. Finding them means solving det(A - λI) = 0 for λ, then solving for each corresponding v.

What makes eigenvectors worth the effort is that they reveal a matrix's own natural axes — directions along which its behaviour is simple pure scaling rather than the general skew-and-rotate mixture it applies to an arbitrary vector. This is exactly the mechanism Unsupervised Learning uses for PCA: the covariance matrix's eigenvectors are the directions of maximum variance in a dataset, found by exactly this same eigenvalue computation, just applied to a matrix built from data rather than from an arbitrary geometric transform.

Vector Spaces

AA vector space is a set of objects (not necessarily geometric arrows — polynomials and functions both form perfectly good vector spaces) that can be added together and scaled, and still stay inside the same set — the defining requirement is closure under those two operations. A set of vectors is linearly independent if none of them can be written as a combination of the others — each one genuinely adds a new direction the others couldn't reach. The rank of a matrix is the number of linearly independent rows (equivalently, columns) it actually has, and it's the precise, checkable version of "how much real information does this matrix contain" — a matrix that looks like it has many rows but whose rank is much lower has redundant rows that are just combinations of the others, contributing no genuinely new information.

References