The Journey of Building PatLang, Continued Still Further: Taking the Native Backend Seriously

For new readers

This is one instalment in an ongoing, chronological diary of building PatLang, written up in numbered "Acts" as the project actually happened, warts included. You don't need to have read the earlier instalments to follow this one, but it helps to know that PatLang already compiles two ways: interpreted directly, or lowered and handed to rustc to produce a real executable. Neither of those is a genuine PatLang-to-native compiler — rustc is still doing the actual machine-code generation underneath. A separate, mostly-finished experiment does exist that skips rustc entirely and emits real x64 machine code directly — and, tellingly, it's written in PatLang itself, not Rust, matching the project's own stated long-term direction. This instalment is the story of taking that experiment out of storage and actually trying to trust it.

A direct continuation of the previous instalment (Acts XXIX-XXXV: teaching the language's own GOAP planner and induction engine to find working programs by search, three separate vacuous results caught before being written up as wins, and a quieter arc of ordinary site maintenance that turned up two genuine runtime bugs along the way) — split into its own page for the same reason each earlier page split off from the one before it. This arc starts with a scoping question about an open bug report, and ends up somewhere much more concrete: a real, if partial, working native backend, several genuinely dangerous silent-wrong-answer bugs found and fixed in it, and one moment where the project's own stated direction had to correct an answer already given.

Act XXXVI: "we do NOT want Rust backends"

The starting question was an open GitHub issue: PatLang's compiled programs still run through an interpreter loop under the hood, even when handed to rustc — not genuine native codegen. A quick reality check made the scope bigger before it got smaller: "we may not need #1 for speed on a native machine but x64 is not going to be relevant when compiling to wasm, for instance" — correctly pointing out that a native x64 backend and a WASM backend are fundamentally different targets, so finishing one could never resolve the other. Asked how to scope this, the answer that came back added a real architectural requirement rather than just narrowing the ticket: "option 1 but making sure the x64 generator and (potential) wasm (or indeed, other targets) generator(s) are modular steps which act as drop in replacement options for each other in the pipeline."

Investigating what already existed turned up something more interesting than the issue's own text implied. The x64 backend wasn't an abandoned Rust module sitting half-finished next to the interpreter — it was a wholly separate, self-hosted PatLang implementation: its own lexer, parser, lowerer, and NASM-emitting codegen, all written in PatLang, invoked by running a driver script through the ordinary interpreter, never touching the compiler's real Rust Program/Function/Instr types at all. Reporting this back, the natural-sounding next step was floated: port that ~2,900 lines of NASM-emission logic into real Rust against the shared IR, so it could plug into the same dispatch as the interpreter and the rustc-backed path. That recommendation went out in writing, on the public issue tracker, as "the architecturally clean answer."

It was wrong, and the correction arrived directly: "We do NOT want Rust backends though - everything should move towards patlang implementations." The project's own standing philosophy says exactly this already — PatLang is deliberately pushed to self-host, with even the very last dependency on rustc (as a native-codegen backend) named as something meant to go away eventually, not something to lean into further. Porting the x64 backend into Rust wouldn't have been a step toward that; it would have been a step backward, dressed up as tidiness. The recommendation had to be walked back in writing, on the same public thread, rather than quietly dropped: any future backend-selection mechanism should be a thin dispatcher deciding which self-hosted PatLang backend to run through the interpreter — not a place to write or grow backend logic in Rust. Lesson: "architecturally clean" is not a neutral engineering judgement — it can smuggle in a language-choice preference that directly contradicts a project's own stated direction, and the only fix, once posted, is a visible correction, not a quiet edit.

Act XXXVII: a modulo operator, a sign flip, and a bug hiding behind another bug

With the direction corrected, the actual work became: does this half-finished native backend hold up under real use? The starting probe was narrow — "If the grammar supports % for numerics, the code generator should also support it" — and immediately found that float % wasn't implemented at all: it printed a diagnostic, then emitted broken assembly anyway, cascading into a wall of unrelated-looking NASM syntax errors rather than failing cleanly. Fixed with a real floating-point remainder sequence. The follow-up question went straight to the heart of the backend's own honesty: "check the numeric tower handles % correctly too" — and found something worse than a missing feature. A function that happened to also touch BigInt values silently fell back to plain integer arithmetic on a BigInt's raw heap-pointer bits whenever % was used, with no error at all — correct by accident for small numbers, silently wrong for a real BigInt. Since genuine arbitrary-precision modulo didn't exist to fall back to, the fix was a correct fast path for the common case plus a loud, distinctive runtime failure for the case that genuinely couldn't be handled yet — a wrong answer turned into a visible one.

Fixing the crash trail from the float bug surfaced something much more serious sitting right next to it: unary negation of a float — plain -x — was using integer two's-complement negation on the value's raw bit pattern. That is not a sign flip in general; the carry from that operation can propagate straight into a floating-point number's exponent bits. Concretely, -1.5's bit pattern, negated as if it were a plain integer, silently became the bit pattern for -3.0 — twice too large, in the wrong direction of nobody's expectation, with no crash and no diagnostic. It is exactly the kind of bug that survives for a long time precisely because the output still looks like a plausible number. Fixed with a proper sign-bit flip instead of an integer negation.

Verifying the fix meant deliberately mixing floats and closures in one small test program, and that test produced its own, unrelated garbage number — apply_twice(double, 5), which should print 20, printed a sixteen-digit number instead. Investigated on its own terms first: was this a closure bug? A minimal repro with no floats anywhere ran correctly. So what changed when a float was added to the same function? The backend classifies an entire function as either all-float or all-int, never mixed — and the plain integer literal 5, sharing a function with an unrelated float literal, was being encoded as the bit pattern for 5.0. That value then crossed into a genuinely plain-integer function, which multiplied it as an integer — 5.0's bit pattern, doubled twice, wraps around a 64-bit register to land on precisely the garbage number that had been observed. Traced by hand, confirmed to the last digit. A loud compile-time warning went in first as an honest stopgap, then a real, narrower fix: an integer literal passed directly as an argument to a known, statically-resolvable plain-integer function now keeps its integer encoding regardless of what else that particular function happens to contain — filed and closed as its own tracked issue, since the user's own framing of it — "fix the nested-closure bug, with its own issue as that is a nasty" — was exactly right about how easily this specific shape of bug could hide.

A genuinely separate closure bug turned up in the same stretch, this time nothing to do with x64 at all: a closure that captured an outer variable and then shadowed it on its very first use — let n = n + 1, where the right-hand n needs the captured outer value — silently read zero instead, in the plain interpreter, on any backend. The variable-capture analysis treated every name a closure ever declares, anywhere in its body, as fully local, with no notion of "before this point" versus "after" — so the one read that genuinely needed the outer value got excluded along with every other, unrelated redeclaration. Fixed by specifically recognising a name's own self-referential first declaration and forcing capture for exactly that case. Lesson: fixing one bug can make its own log output honest enough to reveal a second, completely unrelated bug sitting right behind it — and "the caller function also happens to contain a float literal" is not a detail to shrug off, if the backend's own typing decisions are made per whole function rather than per value.

Act XXXVIII: the print bug that wasn't a print bug

Confirming the fixes hadn't broken anything else meant re-running the very first small test program from this whole arc — recursive Fibonacci numbers, a string built with plain +, and a small list — and it very nearly passed. The Fibonacci numbers were right. The list was right. The string printed <value> instead of hello world. The natural first guess, given everything else touched so far, was a print-formatting regression somewhere in the same code being edited. It wasn't. The instinct about where the real weight was turned out to be exactly right too: "Should make an issue for the print problem to get it resolved. And I suspect that is the highest value fix to address."

Tracing it properly found something much bigger than a display quirk: ordinary string concatenation via + had never actually been implemented in this backend at all, in either of its two internal code paths. A function that concatenates strings but doesn't otherwise touch the numeric tower was never being classified as needing the tag-aware dispatch strings require, so + fell through to plain integer addition on the two strings' tagged pointers — and two string-tagged addresses added together happen to produce a bit pattern that collides with a completely different value's own tag, an internal coincidence with no relation to the string's actual content. The print routine's own dispatcher, seeing a tag it didn't recognise, correctly fell back to its placeholder — which is what made a fundamental missing feature look, from the outside, like a cosmetic print bug. Even the code path meant to handle exactly this case, once actually reached, turned out to be an unfinished stub that had never done anything but return zero. The real fix touched both: functions containing string literals are now correctly routed to the tag-aware path, and that path now genuinely concatenates two strings, rather than silently discarding one of them.

Rerunning the same original test program afterward matched the ordinary interpreter's output exactly, line for line, for the first time. A broader sweep for the same shape of mistake, prompted only by "test more broadly before calling it done," found the sibling bug immediately: string equality and ordering comparisons were also comparing raw memory addresses rather than actual text — two separately-created strings holding the identical word compared as unequal, and "less than" compared meaningless heap addresses instead of any real alphabetical order. The backend's own existing comment even called this "deliberate," which was simply incorrect against how the real language already behaved elsewhere. Fixed the same way: real content comparison for equality, and a genuine character-by-character ordering comparison for everything else, both verified directly against the plain interpreter's own answers rather than assumed. Lesson: when a "cosmetic" bug is reported alongside strong misgivings that it might actually be the important one, that instinct is worth following all the way to the bottom before settling for the smaller explanation — a print routine correctly failing to recognise garbage data is not the same claim as a print routine being broken.

Act XXXIX: merging a branch nobody had touched in a week, and a benchmark suite that found two more bugs before it even finished being built

With the standalone backend now noticeably more trustworthy, the next question was structural: an entire feature branch containing real, working x64-backend improvements — separate-compile chunk-linking, real file I/O, a register-allocator correctness fix — had been sitting unmerged for over a week. "Also need to look at getting the x64 into merged into main which may require bringing the x64 branch uptodate with main without breaking anything first?" Investigating first, rather than attempting the merge blind, turned up a pleasant surprise: nearly everything on that branch was already shared history with the main line. Exactly one commit stood apart — and it fixed almost the identical bug this arc had just fixed independently, but more thoroughly: it covered a code path this session's own fix had missed entirely, and it closed a genuine crash risk (a bare internal tag value that could be mistaken for a real piece of text under just the right conditions) that this session's own fix hadn't even considered. Rather than picking one version over the other, the better parts of both were combined — the missed code path adopted from the older branch, the crash-safety guard ported across to the code this session had written — verified against everything both branches had ever been tested with before being folded permanently into the main line and the old branch retired.

The final piece was infrastructure rather than a bug fix: building a benchmark suite that runs the same set of representative programs through all three ways PatLang can execute code — directly interpreted, compiled via the existing Rust-backed pipeline, and compiled via this session's own hardened native backend — timing each and checking that they all agree on the actual answer, with the explicit request that the newer, less-proven backend should never be allowed to stop the whole suite: "all three with safe (ie continue on fault) treatment of x64 while logging the fault for fixing." Built and run for the first time, it did exactly what a good test suite is supposed to do to code that has never been examined this closely before: found something wrong immediately. A three-way if/elif/else chain was silently executing its final else branch roughly twice as often as it should — traced to a genuine, previously invisible parser bug: the word "elif" is recognised internally as its own distinct token, but the code responsible for knowing when a branch's body has ended was only ever checking for a different, more generic kind of token, so it quietly skipped straight past the word "elif" instead of stopping there. Every plain if/else in the entire language had been working perfectly the whole time; it took a three-way branch, exercised by a genuinely new kind of test, to expose that the middle case had never actually been handled correctly. Fixed with a small, targeted addition rather than a rewrite, verified against a minimal nine-iteration repro, and confirmed to leave every existing test passing.

A second, quieter false alarm turned up moments later: two supposedly-mismatched benchmark results that looked, printed side by side, completely identical. A byte-level comparison found the actual difference immediately — one single invisible trailing blank line the interpreter's own output happens to carry that a standalone compiled program's output doesn't. The existing text-trimming helper already used throughout the codebase didn't catch it, for a reason worth keeping on record rather than quietly working around forever: it was deliberately written to treat only spaces, tabs, and carriage returns as whitespace, not newlines — whether that was always intentional or simply an oversight from whenever it was first written is now an open, explicitly flagged question, rather than something silently patched over in one more place and left for someone else to wonder about later. Lesson: a brand-new kind of test doesn't just catch brand-new kinds of bugs — a benchmark suite built for entirely different reasons (timing figures, cross-backend confidence) found two genuine, previously-invisible bugs in areas that had been considered settled for a very long time, purely by exercising a combination of language features nothing had happened to combine quite that way before.

Lessons from this arc, the short version

"Architecturally clean" can smuggle in a language-choice preference that contradicts a project's own stated direction. If it does, the fix is a visible correction on the record, not a quiet edit.
A crash that "cascades into a wall of unrelated errors" almost always has one root cause near the very start of the generated output, not many separate ones — check the first reported line, not all of them.
Silent-wrong-answer bugs are the dangerous class, not crashes. A negated float that looks like a plausible, slightly-off number is far easier to miss for a long time than an outright failure.
Fixing one bug can make a log honest enough to expose a second, unrelated one sitting right behind it. Don't stop investigating just because the crash you were chasing stopped happening.
A closure-capture bug and a per-function-typing bug can look identical from a garbage number alone. An isolated repro with one variable removed at a time is what tells them apart.
A misgiving that a "cosmetic" bug might actually be the important one is worth following all the way down. A print routine correctly rejecting garbage data is not the same fault as a print routine being broken — and the strongest fix is often hiding one layer further back than the symptom.
Test more broadly than the one case that just got fixed. The very next thing checked after the string-concatenation fix turned up its direct sibling bug, sitting in the same few lines of code.
Investigate a merge before attempting it. A branch feared to be badly out of date turned out to share nearly all of its history already — and the one real difference contained a safety fix worth adopting rather than overwriting.
A brand-new kind of test can find brand-new kinds of bugs in code considered long settled. A benchmark suite built for timing figures and cross-backend confidence found two genuine, previously-invisible bugs — a three-way branch mishandled since whenever it was first written, and a shared utility's own quiet, undocumented assumption — purely by combining language features nothing had happened to combine quite that way before.