Generative Models: VAEs, GANs, and Diffusion

AA classifier answers "what is this?" A generative model answers a different question entirely: "produce me a new, plausible example of this." Three architectures have dominated that second question over the last decade, each arriving at it from a different mathematical direction — variational autoencoders from probabilistic inference, generative adversarial networks from game theory, and diffusion models from statistical physics.

Variational Autoencoders

AnAn ordinary autoencoder learns two networks: an encoder that compresses an input (an image, say) down to a small vector — the latent space — and a decoder that reconstructs the original input from that vector. Trained to minimise reconstruction error alone, nothing stops the latent space from becoming a scattered, arbitrary lookup table — points that decode to a recognisable output might be surrounded by points that decode to nonsense, which makes the latent space useless for generating anything new, since there's no way to know which points are "valid."

Kingma and Welling's variational autoencoder (VAE) fixes this by changing what the encoder outputs and adding a second training objective¹. Instead of encoding an input to one fixed point, the encoder outputs the parameters (mean and variance) of a probability distribution — the input is encoded to a small cloud of nearby points, not a single one. Training then optimises two things at once: reconstruction quality as before, plus a regularisation term that pulls every one of those clouds toward a simple, shared shape (a standard normal distribution). The second term is what makes the difference: because every part of the latent space is now built from overlapping clouds pulled toward the same simple distribution, sampling a random point from that distribution and decoding it produces a plausible, novel output — the latent space has been made smooth and generative almost as a side effect of how it's regularised, rather than needing to be generative by explicit design.

Generative Adversarial Networks

GoodfellowGoodfellow et al.'s generative adversarial network (GAN) takes a completely different route: two networks trained against each other². A generator takes random noise as input and tries to produce something that looks real. A discriminator is shown a mix of real examples and the generator's fakes, and tries to tell which is which. The two are trained simultaneously, and the loss each one is minimising is built directly from how well the other one is doing:

Discriminator's goal: correctly label real examples as real, fakes as fake.
Generator's goal:     fool the discriminator into labelling its fakes as real.

This is the actual training dynamic, not just "two networks compete": every step, the discriminator gets a little better at spotting the generator's current weaknesses, and the generator gets a gradient signal pointing directly at whatever those weaknesses are, pushing it to fix exactly that flaw. Run for long enough, in the idealised case, the generator's output distribution converges toward the real data's distribution and the discriminator is left guessing at chance — it can no longer do better than 50/50, because there's no longer a systematic difference left to detect. In practice, this adversarial process is notoriously unstable to train (the two networks can fall out of balance in either direction), which is a large part of why diffusion models have taken over many applications GANs used to dominate.

Diffusion Models

DiffusionDiffusion models take a third, almost opposite approach: destroy the data on purpose, then learn to undo the destruction. Training works by taking a real image and adding a small amount of random Gaussian noise to it, repeating that many times until the image is pure noise, then training a single network to predict, at each step, what noise was just added — i.e. to reverse one small step of the corruption³.

Generating a new image, once trained, means running that learned denoising step backward starting from pure random noise: predict the noise, subtract an estimate of it, repeat for however many steps training used, and what emerges is a novel image built up gradually rather than produced in one shot. Each individual denoising step is a comparatively easy, well-defined prediction problem (a network 99% confident about what a NEXT small step of noise removal should look like is much easier to train reliably than a generator that has to produce a whole convincing image at once and fool an adversary) — this is the practical reason diffusion models train more stably than GANs, at the cost of needing many sequential steps to generate a single output rather than one forward pass.

Choosing Between Them

NoneNone of these architectures need the input to already be labelled with a "correct answer" the way supervised learning does — the training signal comes from reconstruction quality, an adversarial game, or a noise-prediction task, not from ground-truth labels, which is why the whole family is usually grouped under unsupervised or self-supervised learning rather than supervised learning, even though the underlying networks doing the work are often the same convolutional and transformer architectures covered on Neural Network Architectures.

References