Chapter 15: The Compiler Names Its Own Canon

AA direct continuation of the previous instalment, picking up its own unfinished business rather than opening a new one. Two days carry this chapter: one spent giving isolates a second way to run, the next spent closing a cliffhanger and, almost as a side effect of closing it, settling a question the project had been running on an unstated assumption about for months.

Act LXXVI: Isolates Learn a Second Way to Run a Turn

Isolates,Isolates, built two chapters back, run each turn inside world_run — an in-process swap of every global store for an empty one, the thing that lets a crashed or over-budget turn be discarded wholesale with nothing left behind. That trick only works on a host that can actually do an in-process swap. Native x64's own capability list reports subprocess and fs but never world_swap — correctly, since it compiles to a standalone binary rather than running inside an interpreter with swappable global tables at all — so isolate_start had simply refused outright on that target until this day.

The fix doesn't invent a new primitive; it reuses one the host already has for a different reason. A fresh OS process gives exactly the same "nothing from a previous run is still lying around" guarantee world_run provides in-process, for free, as a side effect of how processes already work — so an isolate step on a host with no world_swap now spawns a genuine, separate pat.exe --ir-run child per turn instead. The isolate's VFS gets snapshotted to a temp file, a small generated runner source restores it, runs the turn's body, and dumps the result back out; the parent reads that result back through the same exec_capture_io machinery the project already had, and reconstructs it into the exact four-tuple shape isolate_step's commit logic already expected — so the commit-or-discard decision above this new backend needed no change at all. An explicit backend override lets a test force the process path even under the interpreter, which has world_swap and would otherwise never exercise the new code — a fast RED/GREEN loop before the slower native confirmation, the same two-speed verification habit this diary keeps finding worth building deliberately rather than relying on luck.

Five smaller items closed out the same day's remaining backlog, one of them worth a sentence on its own: the world backend's own source compiler had never been taught to expand syntax NAME { ... } DSL blocks before tokenizing, so a program using one failed immediately with a generic parse error under world_run while the exact same program succeeded for free under the new process backend — whose spawned real pat.exe gets the native parser's built-in support along with everything else. An asymmetry between two backends that are supposed to behave the same way is itself a kind of bug, even when — as here — the "working" side is the one that happens to be newer.

Act LXXVII: The Race From Chapter 14, Closed — and Already Beside the Point

TheThe previous chapter left a specific, reproduced data race switched off rather than fixed: the native x64 assembler's per-function parallel path, measured at a genuine 10.9x speedup on the raw mechanism, corrupting output unpredictably whenever it actually ran. Closing it started with a wider question than the one bug: a full audit of every place the self-hosted codebase reads get("__vars", key) and writes it back, since a parallel_map/thread_spawn worker does not get its own private copy of that table — it is one shared, process-wide namespace, and a write to it from inside a worker races by construction, no lock available to fix it with, because PatLang has none. The audit catalogued three distinct hazard shapes — shared uniqueness counters, shared resource-handle state, shared accumulators — and, in the same pass, corrected two of its own first-draft claims about which call sites actually reached a live parallel phase, after tracing the real call graph rather than trusting an earlier note about it. Both corrections went into the project's own standing documentation alongside the finding they corrected, not quietly folded away.written right after the bug that proved it was missing

The one hazard that actually mattered for the open race turned out to be exactly as specific as Chapter 14's own symptom suggested: three shared counters inside the x64 codegen and assembler (__x64_str_counter, __x64_cv_counter, xa_id_counter), each incremented from inside x64_assemble_unit_batch's parallel workers, racing to produce the exact "undefined symbol 'S2'"-then-"'S3'"-then-'S0'" nondeterminism already reported. The fix removes the shared state rather than guarding it, twice over: string-literal labels are now derived deterministically from a function's own name plus its instruction index — information every emission pass already has on hand, so nothing needs to be counted globally at all — and the handful of remaining call-with-return/assembler id counters, with no such convenient context available, move to a new host primitive, tls_counter_next, genuinely private per OS thread rather than shared across them. x64_build_use_parallel_units flipped back to true, the blocker it named formally gone. The twist worth stating plainly rather than letting the fix read as a clean, triumphant close: by the time it landed, that parallel path had no live caller left to flip on. The real build driver's own parallel work had already moved, the previous day, to the process-per-turn isolation Act LXXVI built — a different mechanism, achieving the same goal a different way, arrived at before anyone got back around to fixing the first one. Both things are true at once and neither cancels the other: the fix is correct, proven, and no longer load-bearing.

One more bug turned up chasing this one, in the established shape this diary keeps finding: a pre-existing O(n²) cost in the assembler's own symbol-table hashing, invisible until the newly-deterministic labels got longer, traced to a hash function re-checking an entire string's ASCII-ness on every character it read — the project's own documented string-indexing trap, reappearing in a routine nobody had profiled yet. Fixed with the same str_intern/sc_code pattern already used elsewhere, and measured, not assumed: one trivial --x64 compile dropped from roughly three hundred seconds to about one hundred. The full language spec suite — 178 of 178 — passed clean against the rebuilt toolchain once both fixes were in.

Act LXXVIII: The Compiler Names Its Own Canon

TheThe same session that closed Act LXXVII's race surfaced something the project had been running on for a while without ever writing down: when the self-hosted PatLang compiler and its Rust mirror disagree, which one is actually right? The immediate trigger was concrete rather than philosophical — a new host primitive had landed in the wrong chunk on the self-hosted side, and the project's own text-parity check against the Rust mirror caught the mismatch in a single run. That single catch is what the policy, written down the same day, now exists to generalise: self_hosting/lib/*.patlang is declared the canonical implementation of the compiler's logic, full stop, and rust-runtime/src/ir/{hosts,codegen}.rs is kept for exactly two reasons that both stop short of making it primary — bootstrapping the very first self-hosted binary, an unavoidable chicken-and-egg with no PatLang compiler yet to build one with, and standing as an independent oracle the self-hosted emission gets checked against. A mismatch between the two means fixing whichever side is wrong, usually the Rust mirror, unless the self-hosted side itself turns out to be the one with the actual bug. One clarification matters as much as the declaration itself: pat --ir-run, the interpreter, is explicitly not part of what got demoted — only the Rust-source-emission backend, pat --patc, and the habit of treating it as a development target in its own right rather than a bootstrap-and-mirror tool.

The first real work done under the new rule closed a piece Chapter 13 had explicitly left open — in that chapter's own words, "a spawned thread from a block-model closure directly, rather than through the runtime's own closure type," named, scoped, and set aside. thread_spawn, thread_poll, and thread_result, written as ordinary block-model source, now compile and run correctly through the native x64 path. The mechanism is almost understated for what it does: rather than hand-rolling a new runtime-name dispatch trampoline, the lowerer injects one small helper, itself written as ordinary block-model source — bm_closure_entry(name, captured) = apply(name, captured) — reusing apply's already-correct dynamic-call dispatch instead of inventing a second way to do the same lookup. The injection only happens when a program actually calls thread_spawn, specifically so that a build that never touches threading doesn't silently inherit the stricter "every function is a potential call target" rule that dynamic dispatch requires elsewhere.

Two bugs surfaced proving this worked, each invisible until the one before it was fixed. Every block-model function carries an implicit trailing __globals parameter nobody declares by hand — bm_closure_entry included, whose real compiled signature has three parameters where its own source names two. The native thread trampoline pushes exactly as many arguments as the closure's own captured-value count says it needs; two instead of three left the spawned thread reading garbage off its own stack for the parameter it never got, a corruption that only showed up after the main thread had already printed correct output and begun tearing down. The fix supplies a fresh, empty globals list as a third captured value, matching the convention parallel_map's own genuine concurrency had already settled on. The second bug sat one layer further in: thread_result was reading back a two-element [value, globals] pair instead of the bare value it promised, because bm_closure_entry is an ordinary function under the block model's own calling convention, which wraps every return value exactly this way — the native thread trampoline, which knows nothing about that convention, stored the wrapped pair verbatim. Fixed by unwrapping at thread_result's own lowering, the one place in the whole chain that actually knows a bm_closure_entry-shaped callee is on the other end of the thread.

A short, familiar tail closed out the arc: the global variable table made genuinely thread-safe under the block model specifically, a fix to how a native build picks its entry point, and a new standing rule in the project's own conventions about writing a long-running tool's progress to a file as it happens rather than letting it buffer silently until exit — the same lesson this project's Python tooling had already learned, now written down for its PatLang tooling too. Verified: five separate runs of a dedicated thread-spawn check all correct (races are probabilistic, so five clean runs is evidence, not proof, and the write-up says so), the full block-model spec suite at 222 of 222, and the language spec gate still at 178 of 178.

Lessons from this arc, the short version

• An open cliffhanger is worth returning to and closing in public, not quietly dropping once attention moves on. Chapter 14 ended with a named, reproduced bug and an explicit "not yet safe to turn on." This chapter is that exact sentence getting answered, including the part where the answer turned out to be more complicated than a clean fix.
• A fix can be correct, proven, and no longer the thing that matters, at the same time. The parallel-assembly race got genuinely fixed; by the time it did, the project's real parallel workload had already moved to a different mechanism built the day before. Stating both facts plainly is more honest than letting the fix read as the end of the story.
• Write a standing rule down right when a concrete bug just showed what happens without it, not speculatively in advance of ever needing it. The self-hosted-is-canon policy exists because a single parity check had just caught a real mismatch a vaguer convention would have left to chance.
• Reuse a dispatch mechanism that already works instead of hand-rolling a second one that might not. Routing thread_spawn's closures through apply's existing CallDynamic path, rather than a new trampoline, meant one less place for a new kind of bug to hide.
• Each bug in a chain is often invisible only because the one before it hadn't been fixed yet, not because the two are otherwise related. Fixing the missing __globals argument didn't just stop the segfault — it was what made the separate thread_result unwrapping bug visible at all.
• Audit the whole hazard shape, not just the one instance that was reported. Checking every __vars write site, not only the one with a filed GitHub issue, found two of the project's own earlier claims about reachability were themselves wrong — caught by the same pass that was looking for the original bug.

References: What the Literature Already Knew

NoneNone of this arc's findings came from consulting a textbook first; each was found exactly the way the rest of this page describes, by tracing the actual code or reproducing the actual failure. They're listed here afterward because a named technique or formal distinction already had the shape of what turned up in practice.

• A fresh OS process doing an isolate's isolation job in place of an in-process state swap — a process that shares nothing with its siblings can simply be spawned, run its one turn, and be discarded, the same reliability argument Armstrong made for share-nothing processes generally: isolation this complete removes the need to reason about what state is safe to keep¹.
• The exact "undefined symbol" nondeterminism Chapter 14 reported, now traced to a specific unsynchronised shared counter and fixed by removing the shared state — Netzer and Miller's formal distinction between a program that merely behaves nondeterministically and one with a genuine, confirmable data race is precisely the question this arc's tracing answered: not theorised, reproduced, and resolved to one specific unguarded write².
• Declaring one implementation canonical and checking a second, independently-maintained one against it rather than trusting either alone — this is close to a direct instance of differential testing: running the same input through two implementations that should agree and treating any disagreement as a defect in one of them, McKeeman's own name for the general technique a parity check like this one practises³.

See also

TheThe Journey of Building PatLang (Acts I-VI) through the twelfth instalment (Acts LXXI-LXXII) cover everything before the Block Ownership Model's own arc; the thirteenth instalment is where the thread_spawn-from-closures piece this chapter closes was first named and set aside, and the fourteenth is where this chapter's own parallel-assembly race was first found and left deliberately unfixed. The isolates work this chapter extends lives in self_hosting/lib/isolate_process.patlang and isolate_process_runtime.patlang; the canon/mirror policy itself is written out in full in the project's own CLAUDE.md. Closures, threads, fibers, and worlds go fully native within days of where this chapter ends — the next instalment covers that arc directly.