The Criminology of Code, Part 1: Forensics & Investigation

Hand an experienced engineer a broken, undocumented, legacy codebase and watch how they actually work: they don't reach for a textbook algorithm, they reach for a magnifying glass. They secure the scene before touching anything, they ask who was here and when, and they follow the physical traces of what went wrong rather than assuming they can reason their way to the answer from first principles. This is not a loose analogy — the term "software forensics" predates this framing by decades, coined for the specific, disciplined problem of reconstructing what happened in a system from the evidence it left behind [1]. This lecture treats a legacy bug as a crime scene, and borrows investigative discipline rather than mathematical proof as the operating method.

Preserving the Scene

A real crime scene investigator's first rule is: do not disturb the evidence before it is recorded. The software equivalent is to capture a reliable, reproducible failing test case — Kent Beck's "chalk contour" — before touching any code, including code you are confident is unrelated [2]. This matters for a reason that is easy to underweight: refactoring or "cleaning up" before reproduction is secured destroys the ability to tell later whether your change fixed the bug, hid it, or simply moved it somewhere the test suite doesn't look. Beck's test-first discipline generalises directly here — a red test that reliably reproduces the fault is the chalk outline; everything you do afterward is measured against whether that outline eventually goes green and stays green.

Preserving the scene also means resisting the urge to fix the first plausible-looking issue you spot along the way. A codebase under investigation typically has several things wrong with it simultaneously; conflating "a bug" with "the bug that caused this specific outage" is one of the most common ways an investigation goes wrong before it starts.

Establishing Motive and Intent

Legacy code frequently contains decisions that look indefensible in isolation and turn out to have been entirely reasonable given constraints that no longer exist — a database that used to be single-region, an SLA that has since been renegotiated, a library bug that was worked around and never revisited after the library was fixed upstream. Reconstructing that context is detective work in the literal sense: interviewing witnesses (whoever is still around), and reading the physical record.

git blame, commit messages, and code comments are the closest thing software has to a suspect's own statements, made at the time, before they had reason to reconsider. Reading them well is a skill with its own literature: LaToza, Venolia and DeLine's field study of how professional developers actually maintain mental models of unfamiliar code found that reconstructing rationale — the "why", not just the "what changed" — was consistently the hardest and most time-consuming part of a maintenance task, more so than understanding the mechanics of the change itself [3]. Kim, Zimmermann, Pan and Whitehead's work on automatically identifying bug-introducing changes formalised the complementary, more mechanical half of this: given a bug's symptom, algorithmically narrowing down which historical commit is statistically most likely to have introduced it, turning "which commit did this" from an argument into a search problem [4].

graph LR S["Symptom
(the outage)"] --> B["git bisect / blame
(narrow the suspect list)"] B --> C["Commit message +
surrounding context"] C --> M["Reconstructed motive:
what constraint made this
look reasonable then?"] style M fill:#FFC857

Tracing the Modus Operandi

Some bug classes leave physical traces the way a burglar leaves fingerprints: a memory leak leaves a monotonically growing resident-set-size graph; a race condition leaves an intermittent failure rate that correlates with load rather than with any single input; a state-mutation bug leaves a value that is correct at function entry and wrong at function exit, with no line in between that looks obviously culpable. Learning to read these signatures — rather than re-deriving the failure mode from scratch every time — is what separates fast diagnosis from slow.

Zeller's Why Programs Fail systematises exactly this: it treats debugging as the scientific method under time pressure, and builds a taxonomy of failure signatures (memory corruption, race conditions, off-by-one boundary errors) paired with the observation technique that most efficiently distinguishes each from its look-alikes [5]. Perscheid and colleagues' study of how debugging practice has actually evolved among professional developers found that experienced engineers spend a measurably larger fraction of their time on hypothesis formation and evidence-narrowing than novices do, and a correspondingly smaller fraction on blindly stepping through execution — the investigative habit is learned, and it is learnable [6]. Sillito, Murphy and De Volder's catalogue of the actual questions programmers ask while investigating an unfamiliar change is a useful checklist in its own right: "what does this affect?", "is there a way to detect this event before it needs to be caught?", "what happens after this point?" — a working investigator asks these repeatedly, not once [7].

Practical Exercise: Codebase CSI. You are dropped into a broken repository with zero documentation, told only that it experienced a production outage. Using git log, git blame, and whatever reproduction you can construct, write a forensic incident report reconstructing the exact sequence of events: the triggering input or condition, the code path it travelled, the specific commit(s) that made the failure possible, and the earliest point at which the defect could plausibly have been caught. Distinguish clearly between what the evidence proves and what you are inferring from context.

References

  1. Spafford, E. H., & Weeber, S. A. (1993). Software forensics: Can we track code to its authors? Computers & Security, 12(6), 585–595. https://doi.org/10.1016/0167-4048(93)90055-A
  2. Beck, K. (2002). Test-Driven Development: By Example. Addison-Wesley.
  3. LaToza, T. D., Venolia, G., & DeLine, R. (2006). Maintaining mental models: A study of developer work habits. Proceedings of the 28th International Conference on Software Engineering (ICSE '06), 492–501. https://doi.org/10.1145/1134285.1134355
  4. Kim, S., Zimmermann, T., Pan, K., & Whitehead Jr, E. J. (2006). Automatic identification of bug-introducing changes. Proceedings of the 21st IEEE/ACM International Conference on Automated Software Engineering (ASE 2006), 81–90. https://doi.org/10.1109/ASE.2006.23
  5. Zeller, A. (2009). Why Programs Fail: A Guide to Systematic Debugging (2nd ed.). Morgan Kaufmann.
  6. Perscheid, M., Siegmund, B., Taeumel, M., & Hirschfeld, R. (2017). Studying the advancement in debugging practice of professional software developers. Software Quality Journal, 25(1), 83–110. https://doi.org/10.1007/s11219-015-9294-2
  7. Sillito, J., Murphy, G. C., & De Volder, K. (2008). Asking and answering questions during a programming change task. IEEE Transactions on Software Engineering, 34(4), 434–451. https://doi.org/10.1109/TSE.2008.26