CPU Architecture and Memory Hierarchy

AA processor's job, at the lowest level, never really changes: fetch an instruction, work out what it means, and carry it out. Everything covered on this page is either a refinement of that cycle for speed, or an answer to the fact that memory fast enough to keep up with a modern processor is far too expensive to build in the quantities a program actually needs. Patterson and Hennessy's textbook is the standard reference for both halves of this material¹.

Fetch, Decode, Execute

EveryEvery instruction a processor runs goes through the same three stages: fetch the next instruction from memory (using a program counter that tracks where execution currently is), decode it to work out which operation it specifies and which registers or memory locations it needs, and execute it (perform the arithmetic, load/store the data, or redirect the program counter for a branch). A processor's instruction set architecture (ISA) is the vocabulary of operations it understands — the contract between hardware and the software (or compiler) that targets it. RISC (Reduced Instruction Set Computer) designs keep that vocabulary small and each instruction simple and uniform in length, betting that a compiler can combine simple instructions efficiently; CISC (Complex Instruction Set Computer) designs provide richer, more specialised instructions directly in hardware. Most processors built today, including ARM and RISC-V, follow the RISC philosophy; x86, still dominant in desktop and server processors, is CISC at the instruction-set level even though its internal implementation translates those complex instructions into simpler RISC-like micro-operations before executing them.

Pipelining

RunningRunning fetch, decode, and execute strictly one instruction at a time wastes hardware: while one instruction is being decoded, the fetch circuitry that already finished its job sits idle. Pipelining overlaps these stages across multiple instructions — while instruction 1 executes, instruction 2 decodes, and instruction 3 is fetched, all simultaneously — the same principle as a factory assembly line processing several items in different stages of completion at once. This improves throughput (instructions completed per second) without making any single instruction faster; in fact each instruction still takes the same number of stages, its individual latency is unchanged, but the processor as a whole completes far more work per second because it's never sitting idle between stages.

Pipelining introduces hazards — situations where the overlap breaks the illusion that instructions ran one at a time. A data hazard occurs when an instruction needs a value that an earlier, still-in-flight instruction hasn't finished computing yet — if instruction 2 needs the result instruction 1 is still executing, decoding instruction 2 too early would read a stale value. Real pipelines handle this either by stalling (inserting a bubble, wasting a cycle until the value is ready) or by forwarding the result directly from the execute stage to wherever it's needed, without waiting for it to be written back to a register first.

The Memory Hierarchy

AA processor's registers can be read in roughly one clock cycle; main memory (RAM) takes on the order of a hundred cycles; a disk takes millions. Building all of memory out of register-speed hardware is prohibitively expensive at any useful capacity — so real systems instead build a hierarchy, trading capacity against speed at each level.

What makes this hierarchy work rather than just being a compromise is locality: real programs tend to access the same small set of memory locations repeatedly over a short period (temporal locality — a loop counter, re-read every iteration) and to access locations near ones they've just accessed (spatial locality — the next element of an array, right after the current one). A cache exploits both: when the processor requests a memory location, the cache pulls in that location and its neighbours (spatial locality), and keeps recently-used data around rather than discarding it immediately (temporal locality) — so most requests, in practice, are served from a small, fast cache rather than from slow main memory, even though the cache holds only a tiny fraction of everything in memory.

References