Chapter 16: Closures, Threads, Fibers and Worlds Go Native

AA direct continuation of the previous instalment. Where that chapter closed one long-open piece of the native backend's concurrency story, this one finishes the rest of it in two dense days, then spends a third writing down what it built in a form meant to outlast the two days it took to build it.

Act LXXIX: A Runtime Value That Crosses Threads, Fibers and Worlds

RunningRunning PatLang source inside a native program — world_run, the same primitive two chapters back gave isolates their clean room — had worked natively for a while already; this act is where it stopped being a special case and started behaving like everything else the native backend supports. A native world run now swaps every per-store table on entry and restores it on exit, runs the nested IR on its own fiber, and captures whatever it prints through the same rt_emit_line path ordinary output already goes through. A runtime halt inside that nested run — a missing VFS file, an unsupported host call, an explicit os_exit — yields control back to the world's own parent fiber rather than taking the whole process down with it, releasing the shared variable-table lock first if the halted run was holding it, and reporting failure cleanly rather than crashing. Name-based dispatch — fiber_new and apply looking up a function by a string naming it, rather than by a statically visible call — now makes every callable name in a program a call root, and the apply table itself gained a second entry for every block-model function under its own plain name, so a program can name a function the way it's actually written rather than by its internal compiled label.

Fibers got the same equal treatment the same week: a fiber body now takes exactly (arg, __globals) and returns exactly [value, globals], one consistent convention instead of whatever each call site had been assuming, with the trampoline unwrapping that pair automatically so fiber_resume hands back the plain value either way — mid-generator or finished. world_run itself now refuses outright to start while another thread or fiber is genuinely alive elsewhere in the same process (a world's own nested worlds are the deliberate exception), and a world that halts mid-run marks its fiber dead immediately, so a crash inside one world can never silently block every world that tries to run after it.

One correction landed almost as a footnote to all of this, in the opposite direction from a bug that usually gets reported: built-in forms like emit, print, fact, query, and new were being matched by name before a user's own declared function of the same name, so writing your own emit got silently overridden by the language's. A declared function now wins — as an expression call and as a bare statement — with the compiler printing a warning when it does, so the override is visible rather than a quiet surprise either way. And runtime code, not just PatLang source, can now call a closure value directly through a new bridge function, rt_call_closure1 — available in any compiled program that contains at least one closure literal, and refusing clearly, by name, in one that doesn't.

The last piece closes a contract that had only been half-enforced. thread_spawn passes its closure no arguments at all, by design — a closure with parameters reading unset values was returning silent garbage (zero, in the case that got reported) rather than failing. That contract now gets checked at both of the two places it can actually be violated. A closure literal written right at the call site — thread_spawn(|x| { ... }) — is rejected at compile time, because the lowerer can see its parameter list directly. A closure sitting in a variable instead can't be inspected that way until the moment a thread actually tries to start it, so every closure value now carries its own arity as a fourth field — ["__closure", name, captured, arity] — checked by a new rt_check_spawn_arity right as thread_spawn runs, halting with a message that names the actual problem rather than letting the thread start and quietly misbehave.

Act LXXX: A Guide Whose Every Example Actually Ran, and a Door That Refuses to Open the Wrong Way

EverythingEverything Act LXXIX built got written up the same week in one guide covering closures, threads, fibers, and worlds together, organised around what differs between the interpreter and the native backend and which of it also survives a trip through WebAssembly. Its structuring decision is the one worth naming: every example is a real, standalone file under the project's own test fixtures, and the output printed next to each one is what that file actually produced when compiled and run — not a description of what the feature should do, written from memory of how it's supposed to work. A guide built this way can't drift silently out of sync with the compiler the way ordinary prose documentation can; the moment an example's real output changes, the guide is visibly wrong rather than quietly wrong.each example is a file that was really compiled and really run

The same week closed a gap the guide's own WASM section made obvious once someone had to write it down plainly: nothing had been stopping a program using a native-only construct — fibers, threads, the closure bridge, any of the world helpers — from being pointed at a WebAssembly target at all, where none of those have any implementation. patc1 now refuses that build outright, at compile time, printing the exact list of native-only names the program actually used and exiting non-zero, rather than producing a binary that either fails confusingly inside wasmtime or, worse, fails in a way that looks like it might have worked. The guide gained a verified, runnable example for every pattern it documents and a dedicated section naming precisely what does and doesn't survive the trip to WebAssembly — closures, plain numeric output, and the world backend itself travel cleanly; fibers, threads, the closure bridge, and budgeted(...) blocks do not, and now say so immediately instead of eventually. Two remaining differences are documented as deliberate rather than quietly left unmentioned: a WASM or legacy-Rust-emission build prints one stray trailing line when a script's last statement isn't an explicit print(...), held that way on purpose because an existing parity test already depends on it; and a missing VFS file inside a world reports as a differently-worded error on that path, a wording difference only, both sides agreeing on failure.

A short, busy tail — most of it from the two days immediately before this chapter was written — closed out a cluster of smaller gaps the new machinery had made newly visible. A missing host function inside a world used to end the entire process outright, routing around the same halt-to-parent-fiber mechanism an ordinary os_exit already used correctly; fixed so both paths now yield to the world's parent the same way, verified against two fresh probes — one with no world active, falling through to a real process exit as before, one with a world active, yielding and letting the caller's own "after" line still print. world_run itself turned out to be swapping ten global tables faithfully on entry and exit but missing an eleventh and twelfth — the fiber and thread tables — so a world whose own interpreted source reached a nested world_run was sharing its outer caller's live fiber table by omission rather than by design; fixed, with the fiber-id and thread-id counters deliberately left out of the swap, since one is meant to stay globally monotonic on purpose and the other already lives somewhere the existing swap already covers. A build tool's own comment claimed that omitting the target-triple argument meant "build native" — it actually meant the legacy Rust-emission path, silently missing the native runtime's own shadowing primitives, fixed to default to --x64 as the comment had always claimed it did. Both Rust-side implementations of world_run — the interpreter's and the WASM/legacy-emission path's — turned out never to have applied or returned a world's class list at all, unlike the native path, now fixed and mirrored on both sides per the project's own parity discipline. The object cache's own plain copy-then-write, the root cause of a nondeterministic parity-suite pass count the project had been living with since the original cutover, got replaced with a copy-to-temp-then-atomic-rename, so a concurrent reader now only ever sees nothing or the whole file, never a partial write caught mid-copy. And a parser fix meant to reject "return" immediately followed by a block terminator turned out to match the plain identifiers end, else, and elif too — which a real library file legitimately uses as ordinary variable names — tightened to match only an actual end-of-file or closing brace, the two tokens no identifier can ever be. The arc closes, as this diary's chapters increasingly do, with tooling rather than a feature: a parity-check script now drives the full native/interpreter test suite across every self-test file in the project, its concurrency raised from two workers to six now that the object-cache race above is gone, and a newly-found, deliberately-deferred bug — dynamic division has no case for a boxed float — recorded in the project's own catalogue of known gaps rather than left to be rediscovered by surprise later.

Lessons from this arc, the short version

• A contract enforced at only one of its two violation points is still broken for the case that matters. Rejecting a closure literal with parameters at compile time caught nothing about the same closure sitting in a variable; thread_spawn needed both a static check and a value that carries enough information — arity — to check again at the moment it actually matters.
• A guide built from real compiled-and-run examples can't drift silently the way prose can. Documenting a feature with a file that was actually executed, and the output it actually produced, means the next compiler change that breaks the example breaks it visibly, not quietly.
• Refusing to build at all, with the exact offending names listed, beats a binary that builds and then fails confusingly somewhere nobody's watching. The WASM native-only guard exists specifically so a sandboxed, hard-to-debug failure never has the chance to happen in the first place.
• A root cause can hide for a long time behind a vaguer symptom. "Nondeterministic pass counts" had been true of the parity suite since the original cutover; it took tracing one specific unsynchronised file copy to find the actual mechanism underneath the vague description.
• Swapping nine tables correctly and a tenth not at all is a bug that only the case touching that exact tenth table will ever expose. world_run's missing fiber/thread-table swap sat unnoticed until something specifically nested one world_run inside another's interpreted source.
• Write a newly-found, deliberately-unfixed bug into the project's own record the moment it's found, the same standard this project already applies to deferred mirror-sync work — a known gap that's written down can't be mistaken for a surprise later.

References: What the Literature Already Knew

NoneNone of this arc's design decisions were drawn from a textbook first; each was arrived at the way the rest of this page describes, by building the mechanism and then deciding, concretely, what it should refuse to do. They're listed here afterward because a long-established idea or a named technique already had the shape of what this arc built.

• A closure value carrying its captured environment across a thread, fiber, or world boundary, checked for shape before it's allowed to run somewhere new — the closure itself, as a value pairing captured environment with code, is Landin's own construction, built for his SECD machine and the ancestor of every closure since¹.
• Refusing a WASM build outright, by name, rather than letting it fail confusingly inside a sandbox, and rejecting a malformed thread_spawn call at compile time rather than letting it run and misbehave — both are direct instances of the fail-fast principle: stopping immediately and visibly the moment a problem is detectable is what keeps a defect cheap to find, against the alternative of letting it run on and surface somewhere harder to trace².
• A static check (the literal-closure case) and a dynamic check (the arity field, checked at run time) covering the same contract from two different directions because neither alone can see every violation — the general principle that static typing catches what's visible before a program runs and dynamic checks must cover whatever isn't is textbook type-system design, not specific to this one case³.

See also

TheThe previous instalment finishes the thread_spawn-from-closures piece this chapter's arity work builds directly on, and is where the self-hosted-is-canon policy this arc's own mirror-sync fixes (the classes_in/classes_out gap, the ObjCache race) were made under gets declared. Chapter 12 is where world_run and isolates were first built, in-process. Capabilities & Known Limitations is this site's own status-report page for the project, updated alongside this chapter to reflect everything this arc and the one before it actually shipped. The best-practice guide this chapter describes lives at docs/guides/closures-threads-fibers-worlds.md in the project's own repository, with its verified examples under self_hosting/tools/spec_fixtures/doc_examples/.