Neural Network Architectures: From Perceptrons to Transformers

EveryEvery architecture below solves the same underlying problem — learning a function from examples, by adjusting internal numbers until the function's output gets closer to what's wanted — but each one earns its place in the lineage by fixing a specific limitation of what came before it. Understood as a sequence of fixes rather than a list of unrelated inventions, the architecture choices stop looking arbitrary.

The Perceptron and Its Limit

AA single perceptron takes a set of inputs, multiplies each by a learned weight, sums them, adds a bias, and passes the result through a threshold: fire (output 1) if the sum clears the threshold, stay silent (output 0) otherwise. It can learn any function that's linearly separable — one where a single straight line (or, in higher dimensions, a flat plane) can divide the positive examples from the negative ones. It provably cannot learn XOR, the simplest function that isn't linearly separable, because no single straight line separates XOR's true outputs from its false ones on a 2D plot of its inputs — a limitation made famous by Minsky and Papert's 1969 analysis, which is often credited with cooling AI research funding for years afterward.

Multi-Layer Networks and Backpropagation

StackingStacking perceptron-like units into layers — an input layer, one or more hidden layers, an output layer, each layer's outputs feeding the next layer's inputs — fixes the XOR problem: a hidden layer can learn to bend the decision boundary into something curved enough to separate XOR correctly. The question this raises is how to train all those internal weights when only the final output's error is directly observable. Backpropagation answers it: compute how much the final error changes with respect to each weight, working backward from the output layer to the input layer one layer at a time, using the chain rule from calculus to pass that "how much does this weight matter" signal back through every layer it passed through on the way forward. The core idea, stripped of the calculus: each weight gets nudged in whichever direction would have reduced the error just observed, in proportion to how much that specific weight actually contributed to it.

Convolutional Neural Networks

AA fully-connected layer applied directly to an image would need a separate weight for every pixel-to-neuron connection — for a modest 256×256 image, that's over 65,000 weights per neuron in the very first layer, with no way to reuse anything learned about detecting an edge in the top-left corner when that same edge appears in the bottom-right. LeCun et al.'s convolutional neural network (CNN) fixes this with two ideas working together: a small filter (a handful of weights) is convolved across the whole image, so the same small set of weights is reused at every position — the network learns one edge detector, not a thousand independent ones for a thousand positions — and pooling layers periodically shrink the spatial resolution, giving the network some tolerance to a feature shifting by a few pixels¹. Stacked filters build up a hierarchy: early layers learn simple features like edges and colour blobs, middle layers combine those into textures and parts, and later layers combine those into whole objects — a structure that maps naturally onto how the field now approaches feature extraction for object recognition, replacing the hand-designed features that page's earlier methods relied on with features the network learns for itself.

Recurrent Networks and Their Limit

ImagesImages have fixed spatial structure; language, audio, and time-series data have sequential structure of variable length, which a fixed-size feedforward network isn't built for. A recurrent neural network (RNN) processes a sequence one element at a time, feeding its own hidden state from the previous step back in as part of the input to the current step — giving the network a form of memory of what it has already seen. In principle this lets a short-term pattern influence a decision much later in the sequence; in practice, plain RNNs struggle badly with long-range dependencies, because the repeated multiplication involved in backpropagating an error signal back through many time steps tends to shrink it toward zero (the vanishing gradient problem) before it reaches the steps that actually mattered. Hochreiter and Schmidhuber's Long Short-Term Memory (LSTM) architecture fixes this with a dedicated memory cell and learned gates that control what gets written to it, read from it, and forgotten — giving the network an explicit mechanism for preserving a signal across many steps instead of relying on it surviving repeated multiplication by chance².

Transformers and Self-Attention

AnAn RNN's own fix creates a new bottleneck: it processes a sequence strictly one step at a time, which makes it slow to train on the long sequences and huge datasets modern language models use, and it still has to route information about token 1 all the way through the chain to reach token 1000. Vaswani et al.'s transformer architecture removes the recurrence entirely, replacing it with self-attention — every position in the sequence looks directly at every other position in a single step, weighting how much each one should influence it, with no chain of intermediate steps to route through³. This is both more parallelisable (every position's attention can be computed simultaneously rather than one step at a time) and better at genuinely long-range dependencies (token 1 and token 1000 are one attention step apart, not a thousand recurrent steps apart). The mechanics of self-attention itself — queries, keys, and values, and how they combine into an attention weighting — are covered in full on Understanding LLMs, which builds directly on the architecture introduced here.

Training in Practice: Hyperparameters

NoneNone of the architectures above train themselves — getting good results also depends on choices made around the architecture, not just within it. The learning rate controls how large a step each weight update takes (too large and training diverges or oscillates; too small and training crawls, or gets stuck in a poor local solution); batch size controls how many examples are averaged over before each update; and the number of layers and their width control how much capacity the network has to fit the data — more capacity fits the training data more closely, which helps only up to the point where it starts memorising training-specific noise rather than learning the underlying pattern. Systematically searching this space is called hyperparameter tuning, and it is, in practice, as much a part of getting a network to work well as the architecture choice itself.

References