Performance Debugging: Lessons from a Real Regression and Its Fix

A companion to The Anatomy of a Debugging Strategy and Risk Management: Lessons from Testing Real Systems, grounded in a real performance investigation from Act XIV of the PatLang journey: a re-run benchmark surfaced a genuine ~15% regression, honestly labelled as an unverified suspicion rather than a proven cause, which then turned directly into a real fix delivering a ~30% improvement across every execution path the language has — net faster than before the regression even happened. Correctness debugging and performance debugging share a method, but performance debugging has its own specific failure modes, covered here.

Measure before assuming, even when nothing seems to have changed on purpose

No one had set out to make anything slower. A large batch of unrelated feature work had landed between two points in time, and re-running an existing, already-written benchmark rather than trusting that performance "probably hadn't moved" is what surfaced the regression at all. A benchmark that already exists and takes seconds to re-run is close to free; the cost of skipping it is an unknown, possibly compounding, unmeasured drift that nobody notices until it's much larger and much harder to attribute to any one change.

State correlation and unverified causation as different claims, in those words

The regression's likely cause was identified by simple timing — a specific, unrelated change had landed in the window between the two measurements, and it touched exactly the kind of operation the benchmark exercised heavily. That's a real, useful lead. It is not proof. The write-up said so directly: named as the most plausible contributor, explicitly flagged as correlation rather than a verified cause, with the honest note that no bisection had actually been run to confirm it. Three different claims get routinely collapsed into one sentence in most write-ups — "X definitely caused this," "X might have caused this," and complete silence about cause at all — and they warrant three different levels of confidence from a reader. Say which one you actually have.

An unverified lead is still worth acting on directly

The honestly-hedged suspicion above wasn't a dead end waiting for a bisection that never came. Asked afterward whether an optimisation opportunity existed, going straight to the code path the suspicion had already implicated — the arithmetic dispatch logic the regression's own timing pointed at — found a concrete, fixable inefficiency within the same investigation. The lead didn't need to be proven as the sole cause of the regression to be worth following; it only needed to be a real, specific place to look, which an honestly-stated correlation already is.

Prefer an additive fast-path over a rewrite when the common case is identifiable

The actual inefficiency found was structural but narrow: a general-purpose code path (arithmetic across several numeric representations) was being used unconditionally, even for the single overwhelmingly common case (two plain integers, or two plain floats) that never needed any of that generality. The fix added an early check for exactly that common case, doing the direct, cheap computation and returning immediately — leaving every other, rarer case routed through the exact same general code as before, completely unchanged. This is a lower-risk shape of performance fix than rewriting the general path to be faster for everyone: it touches only the traffic that benefits, and by construction cannot change behaviour for any case it doesn't explicitly intercept.

A performance fix needs the same correctness verification bar as a behaviour change, not a lighter one

"It's just faster, the logic is the same" is exactly the kind of claim that should trigger more scrutiny, not less — a fast-path branch is new code, sitting directly in front of the old code, and a mistake in the new branch's condition can silently produce wrong answers for cases that were supposed to fall through untouched. The fix here was verified with a dedicated edge-case check (values large enough to force overflow into a different internal representation, mixed types that must still be routed through the general path, exact division producing a fraction, comparisons across two different representations of the same value) run through every real way the surrounding system executes code, confirming byte-identical output on all of them. A change made in the name of speed still needs proof of correctness — arguably more proof, since "it's obviously fine, I only added a shortcut" is precisely the kind of reasoning that lets a real bug through unchecked.

Find every independently-maintained copy of the logic you're optimising, not just the one you noticed first

The system in question had more than one implementation of the exact same arithmetic dispatch logic — one for ordinary interpreted execution, and a second, structurally different one embedded inside compiled output, because compiled programs in this system carry their own self-contained runtime rather than linking against a shared library. Fixing only the first copy would have made the interpreted path faster while leaving compiled programs exactly as slow as before, with no obvious signal that anything was still wrong — a silent, partial fix masquerading as a complete one. Before declaring a performance fix done, check explicitly whether the logic being optimised has any sibling copies elsewhere in the system, and fix all of them in the same pass.

Re-benchmark across every real execution path, not just the one that flagged the problem

The original regression was noticed on one execution path. The fix was verified across all four the system actually has (an interpreter, a natively-compiled path, a second, self-hosted-compiled path, and a WebAssembly-compiled path) — and the improvement held, at a similar magnitude, on every single one. That uniformity was itself useful evidence: a fix that helps one path a lot and another path not at all is a sign the "same" logic isn't actually as shared as assumed, exactly the kind of gap the previous section warns about. Measuring every path a fix is supposed to reach, not just the path that happened to surface the original problem, is part of confirming the fix landed where it was meant to.

Distinguish "recovered the loss" from "net improvement," and report which one actually happened

After the fix, every measured path was faster not just than the regressed state, but faster than the original baseline from before the regression ever happened. That's a materially different, better claim than "we're back to where we started," and it's worth stating explicitly rather than leaving a reader to assume the more modest outcome. Equally, had the fix only recovered the loss without exceeding the original baseline, that would have been worth stating plainly too — the specific shape of a performance result (regression, exact recovery, or net improvement) changes what a reader should conclude about the system's overall trajectory, and collapsing all three into "we made it faster" hides real information.

Name the bigger, structural fix explicitly, and decide to defer it on purpose

The fast-path fix addressed one identifiable inefficiency; a second, larger one sat directly behind it (the same data structure's general access pattern, not just the specific operation that was fast-pathed) and would have delivered a further improvement at a meaningfully higher cost and risk, touching more of the system's core machinery. Rather than either scope-creeping the current fix to include it or letting it quietly disappear, it was named specifically and set aside as a deliberate, later decision. A performance investigation doesn't have to fix everything it finds in one pass to be a success — it has to leave an accurate, specific record of what was found, what was fixed, and what was consciously left for later and why.

A short checklist

Have you actually re-run an existing benchmark recently, rather than assumed performance is stable because nothing seemed to change on purpose?
When you name a likely cause for a regression, have you said explicitly whether it's confirmed or just your best current guess?
Does your fix add a narrow, additive fast-path for an identifiable common case, or does it risk changing behaviour for cases it wasn't meant to touch?
Have you verified correctness with the same rigour you'd apply to a behavioural change — including the edge cases the fast-path is specifically supposed to skip?
Does the logic you just optimised have any other independently-maintained copies elsewhere in the system? Did you find and fix all of them?
Have you re-measured every real execution path the fix is supposed to reach, not just the one that originally flagged the problem?
Are you reporting whether the result is a regression, an exact recovery, or a net improvement over the original baseline — the honest, specific one, not just "it's faster now"?
If you found a bigger, riskier optimisation opportunity while fixing a smaller one, have you named it precisely enough that someone could pick it up later without rediscovering it from scratch?