Last updated: 2026-11-22
Issues Found in the PatLang 2.0 Gates
For new readers
The gates are every check that must pass before a PatLang change is pushed: the compiler's fixpoint, the bootstrap readers, and each suite on each engine. Between 16 and 22 November 2026 they grew from 229 units to 965 and began running in parallel, and much of the work in that period went into making a pass mean what it says. This page lists the issues in the test machinery itself, under four headings: results that passed without testing anything, checks that failed only under load, tests that interfered with each other, and measurements that misled. The v2 toolchain describes how the gates run.
Size of a full run Ephemeral / ToolingKnowledge that evolves in months to a year โ check for updates
| Date | Units | Full run | Change behind it |
|---|---|---|---|
| 15 Nov | 146 + Part 1 | 2 h 36 min (4 h 52 min one after another) | Parallel units within free cores and memory |
| 16 Nov | 229 | โ | Every selftest gated |
| 16 Nov | 335 | 2 h 13 min | Features over ten minutes split into one unit per scenario |
| 16 Nov | 335 | 77 min | Block-model suite retired |
| 19 Nov | 534, then 621 | โ | Package specs gated; v2.0.0 |
| 20 Nov | 859 | โ | Linux units; v2.1.0 |
| 22 Nov | 965 | โ | TLS, sha512 and curve25519 package suites |
Passes that tested nothing FoundationalKnowledge that endures for decades โ core principles
| Issue | Example | Check now in place |
|---|---|---|
| A unit exited 0 while reporting failures | interp_run_selftest passed with two checks still expecting get to give false | A unit fails when its last tests: N passed, M failed line has failures, whatever its exit code |
| A suite listed no tests and passed | language_linux, with its runner's arguments in the wrong place | A runner that lists nothing gives a failing unit, SUITE/listing |
| Selftests that nothing ran | Six of the 29 synthesis selftests were missing from their runner | A repository check: every selftest is gated, or listed as retired with a reason (21 are) |
| Fixtures that nothing ran | rt_aslr and rt_hosts | A repository check fails on a fixture no scenario names |
| Steps that did nothing | Language scenarios whose programs lived in separate fixtures | Each scenario carries its program in a doc string; no-op steps removed (#269) |
| A pass that relied on the old engines | Synthesis selftests that ran pat.exe or patc1.exe | They run on v2; a repository check refuses programs that start a retired tool |
Grepping old logs for FAIL | Logs from earlier runs and hand reruns sat beside the current run's | Each run first moves everything in v2_gates to v2_gates/previous |
| Numbers quoted from a stale build | The first v2 commit quoted block-model and language counts from runs against a stale x64_runtime.obj, corrected in the next commit | Corrected in the next commit's message |
Checks that failed only under load Applied / MethodologicalKnowledge with a 5โ10 year half-life โ stable practice
Running every core at once changed what timing-based checks measured. A check that passed alone and failed in the gates was a fault in the check until shown otherwise.
| Check | Why it failed under load | Change |
|---|---|---|
Scaling checks on clock_ms() | The clock counts time a unit spends waiting for a processor | They count cpu_cycles(), this thread's own cycles |
| Scaling checks on cycles | cpu_cycles still moves by about a third between runs; vi_bytes compared sizes 2.5 times apart against a bound of 3 | Sizes four times apart against a bound of eight: linear work gives four, quadratic sixteen, each a factor of two from the bound |
| Budgeted-loop overhead | One run of a few milliseconds | Best of three runs, at twenty times the work |
| Signals round trip | A peer whose message arrived more than 250ย ms after the accept was dropped | A peer part way through a message is parked and read on at the next poll; a scenario delays a peer 600ย ms |
| Idle-daemon scenario | The test had to find the daemon before its one idle second ran out | The test checks the exit code only the idle path gives, which does not depend on winning the race |
| Build failures that did not recur | Seen only under heavy load, with no output to diagnose them | A failed build prints the compiler's output, its exit status and the phases it finished |
Tests that interfered with each other Applied / MethodologicalKnowledge with a 5โ10 year half-life โ stable practice
- Fixed ports. Signals scenarios claimed a fixed port another program could hold. Fixtures that listen claim port 0. Where a fixed port is part of what is tested, as port 9410 is for the build daemon, the features that use it share a lane and never run together.
- Engines in parallel. The Linux language suite first had a lane of its own, and its build-daemon scenario ran beside the Windows suite's, sharing the port, the task registry and the daemon's history file. Lanes are now per feature file, shared by every language suite.
- Shared steps between features. The build-daemon feature backed up its history file in a step of the signals feature, so it could not run alone; it takes the backup itself. A step registration that
class_inheritance.featureshared with other features was lost when that feature was converted, and was restored. - Scratch paths. Runners wrote to fixed paths in the tree. Each unit now writes only under its own directory, given as
PATLANG_SPEC_OUTand read throughspec_out(). The package specs wrote topackages/*/build/, which only a hand run had created;spec_out()now makes its directory when missing, and a repository check holds package specs to it. - A daemon left running. A lost quit signal left the build daemon holding its port for every later run.
build_daemon_demo --idle-msexits with 3 when nothing signals it.
Measurements that misled Applied / MethodologicalKnowledge with a 5โ10 year half-life โ stable practice
Each speed-up in v2 was measured before and after. The issues below were found while measuring, and each is now a step in the project's change checklist.
| Issue | Rule |
|---|---|
| A trial build meant to change generated code changed nothing, and timed the same as the baseline, which read as "this costs nothing" | Show a trial differs (its --ir output against the baseline's) before timing it |
| Two compilers built in the edited tree built the same runtime, since a compiled program reads the runtime from the working directory | Build a runtime baseline from a checkout of the old sources, and check it lacks the change |
| A trial compiler with a different cache key rebuilds everything on its first run | Say which runs are comparable |
| A cost that grows quadratically can still be a negligible part of a workload's time | Take a suspected cost away in a trial and time that before fixing it |
| Linux peak memory varied by about 15% with address randomisation on | Measure Linux peaks with it off (setarch -R) |
| A compiler built before an edit reused code compiled by the old sources | The compiler's cache key is the one recorded when it was built (The v2 toolchain) |
Gaps in what the suites covered Applied / MethodologicalKnowledge with a 5โ10 year half-life โ stable practice
- Engine agreement. Scenarios compare the engines on programs someone wrote.
lib/engine_fuzz.patlanggenerates programs from seeds, around the calls two review findings were about, and checks that both engines print the same. - Undeclared names. A program calling a function v2 lacks compiles, and stops only when the call is reached. The names suite runs
v2.exe --namesover every demo and gated selftest. - Untested primitives. The code review listed functions no test called (
bool_array_get,sb_len,fact_in,query_inand others) and paths no test took (a write to a peer that has gone, quoting a child's arguments). Each has a test. - Failure paths. The WebAssembly host can fail the Nth call to a named system function, so error handling in file reads is tested.
- Twin implementations. The compiler's value checks have copies in the two Python bootstrap readers. The gates' WebAssembly unit found
ir_wasm.py's copy missing a new value form; the checklist now names the Python twins for every new primitive.
The rule behind the changes FoundationalKnowledge that endures for decades โ core principles
The project's working notes state one rule for all of the above: what goes wrong twice becomes a check, a tool, a rule or a checklist step. Many rows on this page are repository checks or checklist steps that came from a retrospective after the issue was fixed.
See also
The v2 toolchain describes units, keys, lanes and the planner the gates use. Porting to PatLang 2.0: issues by stage covers the move of programs and content onto v2. PatLang 2.0 beyond Windows covers the Linux units. Risk management: lessons from testing real systems draws teaching material from earlier PatLang test failures.
Graduate Attributes
Modelled on this page
- Researcher: Failures in the test machinery are categorised by type and traced to their root causes.
- Reflective: Past errors are converted into permanent checks and checklist steps through retrospective analysis.
- Digitally Fluent: Technical documentation distinguishes between compiler fixpoints, bootstrap readers and engine suites.
University of Reading Graduate Attributes framework.