The Journey of Building PatLang, Continued Once More: Teaching It to Find Its Own Answers, Honestly

For new readers

This is one instalment in an ongoing, chronological diary of building PatLang, written up in numbered "Acts" as the project actually happened, warts included. You don't need to have read the earlier instalments to follow this one, but it does assume you already know roughly what PatLang is (a programming language whose own compiler is written in itself) and what "GOAP" and "inductive synthesis" mean in this context; the goal-oriented demo page and the inductive synthesis page both introduce those from scratch if this is your first encounter with them.

In plain terms, this instalment is about an experiment: can the language's own planning and rule-learning machinery find working solutions to small problems by searching for them, rather than a person writing the solution and just calling it "found"? The honest, sometimes uncomfortable part is that three separate times, a result that looked correct on the first test turned out not to be — and each of those near-misses, and how they were caught, is described here rather than left out.

A direct continuation of the previous instalment (Acts XXIV-XXVIII: event handlers becoming real closures, then a genuine class keyword with inheritance and traits, built in five verified slices) — split into its own page for the same reason each earlier page split off from the one before it. This arc starts somewhere unglamorous: a self-healing engine whose PLAN phase had quietly never been implemented at all, just a comment and a hardcoded false. Fixing that properly is what actually motivated everything else here: once PLAN meant something real, the natural next question was whether the same discipline — try known mechanisms before reaching for an LLM — could be pushed all the way down to genuine program synthesis. The rest of this page is that experiment, told with the same rule the whole project has followed from the start: a result only counts once it's been shown to fail the way it should, not just pass the way it's supposed to. Three separate times in this arc, a result that looked green on first run turned out to be vacuous on closer inspection — each one caught before being written up as a win, not after. Same warts-included approach, same commit-history sourcing, same house style as the earlier pages.

Act XXIX: a PLAN phase that had never actually planned anything

The self-healing engine's own RED-GREEN-REFACTOR loop had a PLAN step that read, in its entirety, # TODO: Integrate GOAP planning here followed by let plan_found = false — every missing function fell straight through to an LLM query, every time, with no attempt made first to check whether the answer was already knowable without one. Fixed by trying, in order: a known pure-PatLang implementation built entirely from functions already proven to exist (nothing to verify empirically — the contract is the composition of already-known behaviour), then a known shell-command idiom whose actual input-output contract still gets checked directly against real test data before being trusted. Only when neither establishes the contract does the LLM get consulted at all, and even its suggestion is checked exactly the same way — the LLM proposes; it doesn't get to grade its own answer.

Checking results as genuine values rather than a "looks plausible" heuristic surfaced two bugs that had been sitting there invisibly the whole time. A quote-doubling bug: splicing an already-quoted {FILE} placeholder added a second pair of quotes, so PowerShell saw an empty string literal followed by garbage, Get-Item failed, and the error text silently became 0 once run through an unconditional to_num() — confirmed by a direct repro expecting 70 for a genuinely 70-byte file and getting 0 instead. And a can-never-fail bug: heal_missing_function discarded its own GREEN-phase result and unconditionally returned true afterward, so no wrong answer could ever be reported as a failure — confirmed by a repro with a deliberately wrong expected result that still came back "healed = true." Lesson: a result that only gets checked by whether it "contains roughly the right substring" can hide a completely wrong number for a long time; checking it as the actual value it claims to be is what finally surfaced two real bugs that had nothing to do with each other.

Act XXX: "I'm rather nervous about these hand authored fallbacks"

Turning the same PLAN-before-LLM discipline toward genuine program synthesis started with a bigger question: could PatLang's own logic-induction and GOAP-planning machinery find working programs from BDD-style examples alone, the way the induction engine already did for classification and recursive rules (see Inductive Synthesis)? A first curriculum pass mixed genuine induction wins with hand-authored fallbacks for whatever the induction engine couldn't yet reach — all registered in the same place, distinguished only by a metadata field a reader had to remember to check. The concern arrived directly and was exactly right: "I'm rather nervous about these hand authored fallbacks - that implies you are writing the code rather than the synthesis engine finding the right code to use..." A real category error, not a labelling nuance — an audit of everything registered so far found genuinely automated results sitting in the identical directory as things a person had written and then verified.

The fix was structural, not cosmetic: spec_library/synthesized_registry/ now holds only genuine induction or GOAP wins, spec_library/reference_solutions/ holds anything a person wrote, and curriculum_engine.patlang enforces the split with two separate writer functions rather than one function serving both purposes — a caller has to actively choose the wrong one, rather than one path silently serving both. Six existing entries got moved. Lesson: when someone is nervous that "you" are writing the code rather than the system finding it, that's not a mood to manage — it's usually a real, checkable claim about what's actually in the registry, worth auditing on the spot rather than reassuring past.

Act XXXI: teaching it to count, and to toggle a truth value

With the discipline settled — write search infrastructure, never a target problem's own solution — the question became how far pure induction could actually reach. The existing chain-induction search (synth2_induce_chain) was fixed to a unary classifier shape, genuinely unable to express an accumulator-threading relation like "sum the numbers from 1 to N." synth3_accumulator_search.patlang widens the search space with one more generic metarule shape — target(X,Y) :- zero(X), zero(Y). / target(X,Y) :- chain(X,X2), target(X2,Y2), combine(Y2,X,Y). — and searches over candidate (chain, combine) pairs drawn from already-verified background relations, deliberately including two decoys (a step-by-two chain, a multiplication relation) that were both real and both wrong for this target. Given only three visible examples, the search correctly rejected both decoys and found (pred1, plus) — the exact Peano sum-to-N structure — verified against four blind, held-back queries including two plausible-looking wrong totals, all passed.

The next question came unprompted, from watching that result land: "can it learn to toggle a truth value; if so it can count in twos and should be able to handle the sum of evens...?" is_even had already been induced via exactly this shape — flip a flag each step down a chain, base case at zero — so generalizing it to conditionally accumulate only on the "true" branch reached sum of evens up to N, again found by search rather than specified, again verified against visible and blind cases including a target requiring an extended background-fact range before it passed. Lesson: a search engine's real ceiling is usually the shape of its metarule template, not the difficulty of any one problem — widening the template by one generic shape can open up a whole family of previously-unreachable targets at once.

Act XXXII: GOAP already does more than anyone had asked it to

A parallel question, about the planner rather than the logic engine: "I definitely think the goap way is the approach, but we need to get it to be able to solve across multiple simultaneous input set to output set mappings. Which, again, is something I think we can achieve... maybe goap operating on a list of lists of inputs, maybe?" Checking, rather than building anything new first, found the answer already there: plan() already accepts an arbitrary list of goal facts and searches until all are satisfied simultaneously. Given three independent counters (starting at 3, 5, and 2) and one generic, id-parameterized decrement action, a single plan() call produced exactly ten steps — genuinely interleaved across all three counters, not three sub-plans concatenated — found entirely by the planner's own cost-based search.

The natural next step, "Let's do it!", was packaging a discovered composition into a new, reusable, callable action rather than leaving it as an unrolled N-step plan every time. After independently verifying — across many different N, including one held back as a genuine blind case — that the raw decrement primitive reliably reaches zero in exactly N steps, a new action decrement_to_zero was registered, cheaper than N raw steps. The acid test: a brand-new, never-tested starting value produced a single-step plan using the packaged action, not an unrolled sequence — proof the planner's own search preferred it, not that it had been declared and hoped for. The same mechanism mirrored cleanly onto counting up (increment_to_ten) — which surfaced a genuine architectural boundary along the way: this planner only binds action variables by unifying preconditions against world state, so a variable appearing only in an action's effect can never be resolved from the goal being searched for. A fully general "increment to any target" action was, for that reason, not expressible as first built.

Handed three possible fixes for that limitation, the one that actually mattered was recognising which one fit: a Prolog-style goal-regressive unification scheme (unifying an action's effect against the current goal directly) doesn't match this planner's real architecture — it's a forward, uniform-cost search from the start state, with no single "current goal" node to unify against mid-search. The fix that did fit needed no engine change at all: make the target a genuine precondition by asserting an explicit target_goal(Id, Target) fact in the world state, so it binds through the exact mechanism every other variable already uses. Verified on two brand-new (start, target) pairs with different targets each, both resolved through the single packaged action. Lesson: before proposing an engine change to fix a limitation, check what kind of search the engine actually runs — a fix designed for a different architecture can look plausible and still be solving the wrong problem entirely.

Act XXXIII: the false positive that a linear number line was hiding

Asked to test the same arbitrary-target technique "across a range of synthesis tasks" rather than stopping at one success, a battery of structurally different domains — decrementing to an arbitrary target, three simultaneous counters with different targets each, and a genuinely non-linear "doubling" chain — turned up something the earlier results had never been positioned to catch: the packaged action didn't actually verify reachability at all. Its precondition only required some counter fact and some target-goal fact to exist, then unconditionally jumped to whatever the target said. Concrete proof: with only double(1,2), double(2,4), double(4,8) declared as background — six is not reachable from one by real doubling — plan() still returned a confident one-step "solution" claiming otherwise.

Every earlier acid test had used linear +1/-1 chains, where every intermediate value is reachable by construction — so the missing check had been sitting there the entire time, invisible until a domain existed where success wasn't guaranteed just by the shape of the numbers involved. The fix added a genuine reaches(X, Target) precondition, computed by a real transitive-closure/BFS algorithm over the base chain — generic graph-reachability infrastructure, not a hand-solved answer for any one target. Re-verified: the reachable case still resolved in one step; the genuinely unreachable case now correctly returned no plan at all. The original registry entry was corrected in place with a full account of what had been wrong and why, rather than quietly replaced. Lesson: a packaged shortcut deserves at least one test in a domain where its precondition's "this always matches" shape would be exposed by a real negative case — testing only in domains where success is guaranteed by construction can make a genuinely broken check look, for a while, exactly like a working one.

Act XXXIV: catching a vacuous result before writing it down, and a real Unicode bug found along the way

The same discipline applied one more time, to a genuinely different kind of domain: string operations — concatenation, concatenation with a separator, reversal, and palindrome checking, all requested together. A first attempt built background facts by simulating, in the test harness itself, the exact correct sequence needed for each specific target string — every check passed, including blind cases, and every one of them was hollow: with zero decoys and exactly one legal action at every step, the planner wasn't discovering anything, just confirming a path that had already been fully worked out by hand before being handed over. Caught on review, before being registered as a result, not after: if a background fact set gives the search exactly one legal path at every step, passing tests are not evidence of discovery, no matter how many of them there are.

The honest rebuild generated the full universe of possible next-character facts over a small alphabet, independent of any one target, so every reachable state genuinely had several outgoing actions — most of them real decoys the search had to reject to find the correct string. All five checks (build-from-scratch, concatenation, reversal, palindrome-via-composition, and a genuinely unreachable negative case) passed for real this time. A follow-up concern about the approach's own design was just as sharp as the one that opened Act XXX: "I feel that defining all the alphabet separately is an annoying and slightly dodgy approach, and would be troublesome if we move to unicode." The fix derived the working alphabet directly from whatever strings were actually in play, and replaced the pre-built exponential closure with facts generated only along the real search path — both a cleaner design and, deliberately, a genuine Unicode test.

That test surfaced something unrelated to any of the GOAP work: PatLang's own lexer reads source text one raw byte at a time, never decoding multi-byte UTF-8 sequences at all — a string literal containing a real accented or non-Latin character gets silently corrupted, each byte reinterpreted as its own (wrong) character. Confirmed by direct byte inspection: the source file on disk was correctly UTF-8 encoded throughout; the corruption happened purely inside the lexer. Flagged plainly rather than patched on the spot — a real, separate, pre-existing limitation that this session's own Unicode question happened to be the first thing to actually go looking for it. Lesson: testing a general mechanism honestly sometimes surfaces a completely unrelated bug as a side effect — worth reporting exactly as what it is, not folding it silently into the result it was found while checking.

Act XXXV: writing about the object model surfaced two real bugs that had nothing to do with writing about it

A quieter arc than the GOAP/synthesis run above, but the same discipline applied to ordinary site maintenance turned up two genuine, previously-invisible runtime bugs — neither one went looking for a bug; both were found purely because an example was actually run before being published, rather than trusted from documentation or "it looks like it works on the live site." A backlog review first turned eight real, still-open gaps (native codegen still running an interpreter loop under the hood, the stalled x64 backend, and others) into tracked GitHub issues rather than scattered notes — a small, deliberate shift toward keeping PatLang's own technical debt visible where collaborators would actually look for it, not just in private working notes.

Asked for a new page walking single inheritance, trait composition, plain object composition, and combinations of all three with worked pros and cons, every example was run for real before being written up — and the very first one, transcribed faithfully from the Paradigms Guide's own long-published Account/deposit example, silently failed: depositing 50 into a starting balance of 0 printed 0, not 50. set_var(obj, key, val) — the exact three-argument form shown consistently across half a dozen pages — turned out to only ever be implemented as a two-argument global variable store; called with an object argument, it silently no-op'd rather than erroring, so the gap had sat invisible in the site's own canonical documentation for as long as that example had existed. Given the choice between patching the docs to match the broken behaviour or fixing the runtime to match what had clearly been the intended design all along, the answer was direct: fix the runtime. RED-GREEN tests, both execution paths, the self-hosted mirror re-verified, then an issue filed and closed in the same sitting rather than left to rot.

Building that same page's examples surfaced a second, unrelated finding: the class/trait feature's own site coverage had two real gaps hiding in plain sight — the Standard Library reference never mentioned class_def, the actual host function the class keyword lowers to, and the Capabilities page's own "what actually works" list never mentioned the feature at all, despite it being real, substantial, and already shipped. Both fixed on the spot, the same afternoon a direct question — "Can you check the parslow site to make sure it correctly reflects the class/inheritance/traits work?" — asked for exactly this kind of audit.

A separate report — the IDE and Live Playground pages failing to parse a class/inherits/traits example with unexpected token 'class' — traced to something simpler than a parser bug: the embedded WASM build of the self-hosted compiler powering every page's "run in browser" card simply predated the feature. Rebuilding it (build_portfolio.patlang) and pushing the refreshed binary into 27 pages via the existing resync_wasm.py tool fixed it directly — a genuinely stale build artifact, not a new bug.

The last piece was harder to see coming. Asked whether a write_file-based GUI demo could render live in the general-purpose IDE the way a dedicated demo page already did, the investigation found that the dedicated demo had never actually solved the problem it appeared to: its WASM-specific source used print(page) instead of write_file(...), sidestepping an underlying WASI-shim bug entirely rather than fixing it. Traced directly in Node.js — compiling a real write_file program and running it against the exact deployed browser shim, no browser needed — the shim's path_open/fd_prestat_get stubs always reported zero preopened directories, so Rust's own filesystem layer rejected every real file write before ever attempting to open one; separately, the shim's fd_write ignored which file descriptor it was even writing to, meaning any file content that DID get through would have been silently glued onto whatever else the program had printed, tolerated only because a browser ignores text after </html>. Fixed by simulating one genuine preopened directory, so a real file write now completes properly, with each written file rendered into its own lazily-created iframe — never a placeholder shown in advance, per the direct request that followed: "it would be really nice to be able to redirect any files written to their own individual iframes... without cluttering the place with empty iframes put in place just in case." The two previously hand-duplicated copies of the browser JS harness were folded into one canonical source and rolled out through the same resync_wasm.py tool already used for the WASM binaries themselves — "the shim should be changed once and rolled out via the resync_wasm" — rather than fixed in one page and left to drift in the other thirty. Lesson: a demo "working" on one page proves nothing about a more general mechanism working elsewhere — it can just as easily mean that demo quietly avoided the exact path that was actually broken.

Lessons from this arc, the short version

A "looks roughly right" check can hide a completely wrong value for a long time. Checking a result as the actual value it claims to be, not a plausible-looking substring of it, is what surfaces the bugs a fuzzy check quietly protects.
A hand-authored fallback and a genuine search result are not the same claim, even when both pass the same test. If a person is nervous that "you" wrote the code rather than the system finding it, that's a checkable claim worth auditing immediately, not a mood to manage.
A search engine's real ceiling is usually the shape of its metarule template, not the difficulty of any one problem. Widening the template by one generic shape can open a whole family of previously-unreachable targets at once.
Before proposing a fix for an engine's limitation, check what kind of search the engine actually runs. A fix designed for a different architecture can look plausible and still solve the wrong problem.
A packaged shortcut deserves at least one test in a domain where its precondition's "always matches" shape would be exposed by a real negative case. Testing only where success is guaranteed by construction can make a broken check look like a working one, for a while.
If a background fact set gives a search exactly one legal path at every step, passing tests are not evidence of discovery. No matter how many of them there are.
Testing a general mechanism honestly can surface a completely unrelated bug as a side effect. Worth reporting exactly as what it is, not folding silently into the result it was found while checking.
A demo "working" on one page proves nothing about the same mechanism working elsewhere. It can just as easily mean that demo happened to avoid the exact path that was actually broken.