The Journey of Building PatLang, Continued Once More: Teaching It to Find Its Own Answers, Honestly

A direct continuation of the previous instalment (Acts XXIV-XXVIII: event handlers becoming real closures, then a genuine class keyword with inheritance and traits, built in five verified slices) — split into its own page for the same reason each earlier page split off from the one before it. This arc starts somewhere unglamorous: a self-healing engine whose PLAN phase had quietly never been implemented at all, just a comment and a hardcoded false. Fixing that properly is what actually motivated everything else here: once PLAN meant something real, the natural next question was whether the same discipline — try known mechanisms before reaching for an LLM — could be pushed all the way down to genuine program synthesis. The rest of this page is that experiment, told with the same rule the whole project has followed from the start: a result only counts once it's been shown to fail the way it should, not just pass the way it's supposed to. Three separate times in this arc, a result that looked green on first run turned out to be vacuous on closer inspection — each one caught before being written up as a win, not after. Same warts-included approach, same commit-history sourcing, same house style as the earlier pages.

Act XXIX: a PLAN phase that had never actually planned anything

The self-healing engine's own RED-GREEN-REFACTOR loop had a PLAN step that read, in its entirety, # TODO: Integrate GOAP planning here followed by let plan_found = false — every missing function fell straight through to an LLM query, every time, with no attempt made first to check whether the answer was already knowable without one. Fixed by trying, in order: a known pure-PatLang implementation built entirely from functions already proven to exist (nothing to verify empirically — the contract is the composition of already-known behaviour), then a known shell-command idiom whose actual input-output contract still gets checked directly against real test data before being trusted. Only when neither establishes the contract does the LLM get consulted at all, and even its suggestion is checked exactly the same way — the LLM proposes; it doesn't get to grade its own answer.

Checking results as genuine values rather than a "looks plausible" heuristic surfaced two bugs that had been sitting there invisibly the whole time. A quote-doubling bug: splicing an already-quoted {FILE} placeholder added a second pair of quotes, so PowerShell saw an empty string literal followed by garbage, Get-Item failed, and the error text silently became 0 once run through an unconditional to_num() — confirmed by a direct repro expecting 70 for a genuinely 70-byte file and getting 0 instead. And a can-never-fail bug: heal_missing_function discarded its own GREEN-phase result and unconditionally returned true afterward, so no wrong answer could ever be reported as a failure — confirmed by a repro with a deliberately wrong expected result that still came back "healed = true." Lesson: a result that only gets checked by whether it "contains roughly the right substring" can hide a completely wrong number for a long time; checking it as the actual value it claims to be is what finally surfaced two real bugs that had nothing to do with each other.

Act XXX: "I'm rather nervous about these hand authored fallbacks"

Turning the same PLAN-before-LLM discipline toward genuine program synthesis started with a bigger question: could PatLang's own logic-induction and GOAP-planning machinery find working programs from BDD-style examples alone, the way the induction engine already did for classification and recursive rules (see Inductive Synthesis)? A first curriculum pass mixed genuine induction wins with hand-authored fallbacks for whatever the induction engine couldn't yet reach — all registered in the same place, distinguished only by a metadata field a reader had to remember to check. The concern arrived directly and was exactly right: "I'm rather nervous about these hand authored fallbacks - that implies you are writing the code rather than the synthesis engine finding the right code to use..." A real category error, not a labelling nuance — an audit of everything registered so far found genuinely automated results sitting in the identical directory as things a person had written and then verified.

The fix was structural, not cosmetic: spec_library/synthesized_registry/ now holds only genuine induction or GOAP wins, spec_library/reference_solutions/ holds anything a person wrote, and curriculum_engine.patlang enforces the split with two separate writer functions rather than one function serving both purposes — a caller has to actively choose the wrong one, rather than one path silently serving both. Six existing entries got moved. Lesson: when someone is nervous that "you" are writing the code rather than the system finding it, that's not a mood to manage — it's usually a real, checkable claim about what's actually in the registry, worth auditing on the spot rather than reassuring past.

Act XXXI: teaching it to count, and to toggle a truth value

With the discipline settled — write search infrastructure, never a target problem's own solution — the question became how far pure induction could actually reach. The existing chain-induction search (synth2_induce_chain) was fixed to a unary classifier shape, genuinely unable to express an accumulator-threading relation like "sum the numbers from 1 to N." synth3_accumulator_search.patlang widens the search space with one more generic metarule shape — target(X,Y) :- zero(X), zero(Y). / target(X,Y) :- chain(X,X2), target(X2,Y2), combine(Y2,X,Y). — and searches over candidate (chain, combine) pairs drawn from already-verified background relations, deliberately including two decoys (a step-by-two chain, a multiplication relation) that were both real and both wrong for this target. Given only three visible examples, the search correctly rejected both decoys and found (pred1, plus) — the exact Peano sum-to-N structure — verified against four blind, held-back queries including two plausible-looking wrong totals, all passed.

The next question came unprompted, from watching that result land: "can it learn to toggle a truth value; if so it can count in twos and should be able to handle the sum of evens...?" is_even had already been induced via exactly this shape — flip a flag each step down a chain, base case at zero — so generalizing it to conditionally accumulate only on the "true" branch reached sum of evens up to N, again found by search rather than specified, again verified against visible and blind cases including a target requiring an extended background-fact range before it passed. Lesson: a search engine's real ceiling is usually the shape of its metarule template, not the difficulty of any one problem — widening the template by one generic shape can open up a whole family of previously-unreachable targets at once.

Act XXXII: GOAP already does more than anyone had asked it to

A parallel question, about the planner rather than the logic engine: "I definitely think the goap way is the approach, but we need to get it to be able to solve across multiple simultaneous input set to output set mappings. Which, again, is something I think we can achieve... maybe goap operating on a list of lists of inputs, maybe?" Checking, rather than building anything new first, found the answer already there: plan() already accepts an arbitrary list of goal facts and searches until all are satisfied simultaneously. Given three independent counters (starting at 3, 5, and 2) and one generic, id-parameterized decrement action, a single plan() call produced exactly ten steps — genuinely interleaved across all three counters, not three sub-plans concatenated — found entirely by the planner's own cost-based search.

The natural next step, "Let's do it!", was packaging a discovered composition into a new, reusable, callable action rather than leaving it as an unrolled N-step plan every time. After independently verifying — across many different N, including one held back as a genuine blind case — that the raw decrement primitive reliably reaches zero in exactly N steps, a new action decrement_to_zero was registered, cheaper than N raw steps. The acid test: a brand-new, never-tested starting value produced a single-step plan using the packaged action, not an unrolled sequence — proof the planner's own search preferred it, not that it had been declared and hoped for. The same mechanism mirrored cleanly onto counting up (increment_to_ten) — which surfaced a genuine architectural boundary along the way: this planner only binds action variables by unifying preconditions against world state, so a variable appearing only in an action's effect can never be resolved from the goal being searched for. A fully general "increment to any target" action was, for that reason, not expressible as first built.

Handed three possible fixes for that limitation, the one that actually mattered was recognising which one fit: a Prolog-style goal-regressive unification scheme (unifying an action's effect against the current goal directly) doesn't match this planner's real architecture — it's a forward, uniform-cost search from the start state, with no single "current goal" node to unify against mid-search. The fix that did fit needed no engine change at all: make the target a genuine precondition by asserting an explicit target_goal(Id, Target) fact in the world state, so it binds through the exact mechanism every other variable already uses. Verified on two brand-new (start, target) pairs with different targets each, both resolved through the single packaged action. Lesson: before proposing an engine change to fix a limitation, check what kind of search the engine actually runs — a fix designed for a different architecture can look plausible and still be solving the wrong problem entirely.

Act XXXIII: the false positive that a linear number line was hiding

Asked to test the same arbitrary-target technique "across a range of synthesis tasks" rather than stopping at one success, a battery of structurally different domains — decrementing to an arbitrary target, three simultaneous counters with different targets each, and a genuinely non-linear "doubling" chain — turned up something the earlier results had never been positioned to catch: the packaged action didn't actually verify reachability at all. Its precondition only required some counter fact and some target-goal fact to exist, then unconditionally jumped to whatever the target said. Concrete proof: with only double(1,2), double(2,4), double(4,8) declared as background — six is not reachable from one by real doubling — plan() still returned a confident one-step "solution" claiming otherwise.

Every earlier acid test had used linear +1/-1 chains, where every intermediate value is reachable by construction — so the missing check had been sitting there the entire time, invisible until a domain existed where success wasn't guaranteed just by the shape of the numbers involved. The fix added a genuine reaches(X, Target) precondition, computed by a real transitive-closure/BFS algorithm over the base chain — generic graph-reachability infrastructure, not a hand-solved answer for any one target. Re-verified: the reachable case still resolved in one step; the genuinely unreachable case now correctly returned no plan at all. The original registry entry was corrected in place with a full account of what had been wrong and why, rather than quietly replaced. Lesson: a packaged shortcut deserves at least one test in a domain where its precondition's "this always matches" shape would be exposed by a real negative case — testing only in domains where success is guaranteed by construction can make a genuinely broken check look, for a while, exactly like a working one.

Act XXXIV: catching a vacuous result before writing it down, and a real Unicode bug found along the way

The same discipline applied one more time, to a genuinely different kind of domain: string operations — concatenation, concatenation with a separator, reversal, and palindrome checking, all requested together. A first attempt built background facts by simulating, in the test harness itself, the exact correct sequence needed for each specific target string — every check passed, including blind cases, and every one of them was hollow: with zero decoys and exactly one legal action at every step, the planner wasn't discovering anything, just confirming a path that had already been fully worked out by hand before being handed over. Caught on review, before being registered as a result, not after: if a background fact set gives the search exactly one legal path at every step, passing tests are not evidence of discovery, no matter how many of them there are.

The honest rebuild generated the full universe of possible next-character facts over a small alphabet, independent of any one target, so every reachable state genuinely had several outgoing actions — most of them real decoys the search had to reject to find the correct string. All five checks (build-from-scratch, concatenation, reversal, palindrome-via-composition, and a genuinely unreachable negative case) passed for real this time. A follow-up concern about the approach's own design was just as sharp as the one that opened Act XXX: "I feel that defining all the alphabet separately is an annoying and slightly dodgy approach, and would be troublesome if we move to unicode." The fix derived the working alphabet directly from whatever strings were actually in play, and replaced the pre-built exponential closure with facts generated only along the real search path — both a cleaner design and, deliberately, a genuine Unicode test.

That test surfaced something unrelated to any of the GOAP work: PatLang's own lexer reads source text one raw byte at a time, never decoding multi-byte UTF-8 sequences at all — a string literal containing a real accented or non-Latin character gets silently corrupted, each byte reinterpreted as its own (wrong) character. Confirmed by direct byte inspection: the source file on disk was correctly UTF-8 encoded throughout; the corruption happened purely inside the lexer. Flagged plainly rather than patched on the spot — a real, separate, pre-existing limitation that this session's own Unicode question happened to be the first thing to actually go looking for it. Lesson: testing a general mechanism honestly sometimes surfaces a completely unrelated bug as a side effect — worth reporting exactly as what it is, not folding it silently into the result it was found while checking.

Lessons from this arc, the short version

A "looks roughly right" check can hide a completely wrong value for a long time. Checking a result as the actual value it claims to be, not a plausible-looking substring of it, is what surfaces the bugs a fuzzy check quietly protects.
A hand-authored fallback and a genuine search result are not the same claim, even when both pass the same test. If a person is nervous that "you" wrote the code rather than the system finding it, that's a checkable claim worth auditing immediately, not a mood to manage.
A search engine's real ceiling is usually the shape of its metarule template, not the difficulty of any one problem. Widening the template by one generic shape can open a whole family of previously-unreachable targets at once.
Before proposing a fix for an engine's limitation, check what kind of search the engine actually runs. A fix designed for a different architecture can look plausible and still solve the wrong problem.
A packaged shortcut deserves at least one test in a domain where its precondition's "always matches" shape would be exposed by a real negative case. Testing only where success is guaranteed by construction can make a broken check look like a working one, for a while.
If a background fact set gives a search exactly one legal path at every step, passing tests are not evidence of discovery. No matter how many of them there are.
Testing a general mechanism honestly can surface a completely unrelated bug as a side effect. Worth reporting exactly as what it is, not folding silently into the result it was found while checking.