Probability and Statistics for Computing

AlmostAlmost every piece of machine-learning and data-science content on this site leans on a small, recurring set of probability and statistics concepts — this page collects them in one place, following DeGroot and Schervish's textbook, a standard and widely-used introduction to the field¹.

Conditional Probability and Bayes' Theorem

P(A|B)P(A|B) — "the probability of A, given that B has happened" — is the probability of A recalculated in a world where B is already known to be true. Bayes' theorem gives a way to flip a conditional probability around, from P(B|A) to P(A|B), which matters enormously in practice because the direction that's easy to reason about (e.g. "if someone has a disease, how likely is a positive test?") is often not the direction actually wanted ("given a positive test, how likely is the disease?"):

P(A|B) = P(B|A) * P(A) / P(B)

Worked example, a classic and deliberately counter-intuitive one: a disease affects 1% of a population (P(disease) = 0.01), and a test is 90% accurate in both directions (P(positive|disease) = 0.9, P(positive|no disease) = 0.1, a false-positive rate). Given a positive result, what's the actual probability of having the disease? P(positive) = P(positive|disease)×P(disease) + P(positive|no disease)×P(no disease) = 0.9×0.01 + 0.1×0.99 = 0.108. So P(disease|positive) = 0.9×0.01 / 0.108 ≈ 0.083 — just over 8%, far lower than the test's "90% accurate" framing makes it sound, because the disease is rare enough that false positives from the large healthy population outnumber true positives from the small infected one. This exact reasoning — a rare event competing against a much larger population of true negatives — is the same one worth applying to any "the system flagged this as suspicious" alert.

Bayesian updating also has a direct, real application in reputation and trust systems. Jøsang and Ismail's Beta reputation system models a relationship's trust score as the parameters of a Beta distribution, updated by literal Bayesian conjugate updating each time a new piece of positive or negative feedback arrives — positive feedback increments one parameter, negative feedback the other, and the distribution's own shape captures both the current trust estimate and how much evidence it's based on². This site's own Distributed Embedding Calculus of Trust takes a related but distinct approach — tracking positive evidence, negative evidence, and uncertainty as a hand-built vector with its own differential-equation update rule, rather than as literal Beta-distribution parameters — but solving substantially the same problem the Beta reputation system solves formally: representing trust as evidence-backed belief that should firm up as evidence accumulates and stay uncertain when it hasn't. Connecting those two treatments is this page's own reading, not a claim either source makes about the other.

Random Variables and Distributions

AA random variable maps outcomes of a random process to numbers, and a distribution describes how probability is spread across its possible values. A few distributions recur constantly: the Bernoulli distribution (a single yes/no trial), the Binomial (the count of successes across n independent Bernoulli trials), and the Normal (Gaussian) distribution — the familiar bell curve, ubiquitous partly because of the Central Limit Theorem, which says the sum (or average) of many independent random variables tends toward a Normal distribution regardless of the shape of the individual variables being summed.

Expectation, E[X], is a distribution's long-run average value — weight every possible outcome by its probability and sum. Covariance measures whether two random variables tend to move together (positive covariance), move oppositely (negative), or show no linear relationship (near zero) — and it's the exact quantity the covariance matrix in PCA is built from, one entry per pair of features.

Hypothesis Testing

HypothesisHypothesis testing asks: is an observed effect likely to be real, or could it plausibly be explained by chance alone? The procedure: state a null hypothesis (the boring, "nothing interesting is happening" default — e.g. "this new website design has no effect on conversion rate"), compute a p-value (the probability of seeing data at least this extreme if the null hypothesis were actually true), and reject the null hypothesis if the p-value falls below a pre-chosen significance threshold (conventionally 0.05).

The p-value is worth being precise about, because it's routinely misread: it is not "the probability the null hypothesis is true," and it is not "the probability the observed effect is real." It's the probability of the observed data (or something more extreme), computed under the assumption that the null hypothesis holds — a small p-value is evidence against the null hypothesis, not a direct measurement of how likely the alternative is. ANOVA (analysis of variance) extends the same basic logic to comparing more than two groups at once, testing whether their means differ by more than chance alone would explain.

Regression and Correlation

CorrelationCorrelation measures the strength and direction of a linear relationship between two variables, scaled to always fall between -1 and 1. Regression goes further, fitting an actual equation that predicts one variable from another (or several others) — the same least-squares fitting covered in depth, with its optimisation machinery, on Supervised Learning. The recurring warning attached to both: correlation, however strong, is not evidence of causation on its own — two variables can move together because one causes the other, because a third variable causes both, or by pure coincidence, and no amount of correlation strength alone distinguishes between those explanations.

PCA, Briefly Revisited

PrincipalPrincipal Component Analysis, covered mechanically on Unsupervised Learning, is worth naming here too because it sits exactly at the intersection of this page and Linear Algebra for Computing: it's built from a dataset's covariance matrix (a statistics concept, covered above) via that matrix's eigenvectors and eigenvalues (a linear-algebra concept) — a clean example of how these two branches of mathematics routinely combine in real data-analysis methods rather than staying in separate boxes.

References