Instrumenting Long-Running Programs: Health, Progress, and Resume

A student-facing guide, using PatLang's real signals and queue libraries as the worked example — and a candid look at where the AI assistant building PatLang keeps forgetting to apply its own advice.

The problem this solves

Any program that runs for more than a few seconds has a question hanging over it that a quick script never has to answer: is it still working, or has it died silently? A five-minute build, a long simulation, a batch job processing thousands of records — from the outside, a program that's 90% done and a program that's hung look identical. Nothing is printing. The process is still there in the task list. You genuinely cannot tell.

The fix isn't complicated in principle: build in a way for the running program to report its own state on demand, and a way for it to save enough progress that it doesn't have to start over if it's killed or crashes. What's actually interesting — and useful for your own project — is that doing this properly is a design decision made early, not a feature bolted on later, and it's surprisingly easy to build in a way that looks like it works but doesn't actually deliver timely answers when you need them.

PatLang has real, working library support for exactly this, and its own development history is a genuinely useful case study in what goes wrong when the instrumentation is present but incomplete.

The three-part architecture

1. Signals: "are you alive, and what's happening right now?"

A long-running PatLang program can claim a port and declare, at the top of the file, a small set of when EVENT do ... end handlers — a genuine control surface anyone can read at a glance. Every well-behaved long-running program is expected to answer at least two of these: status (report what you're doing right now, synchronously) and quit (stop cleanly). A third, program-specific action verb (rebuild, refresh, whatever your program's one useful trigger is called) rounds out the usual set.

when status do
  signal_reply("processing record " + to_str(current) + " of " + to_str(total))
end

when quit do
  set_var("should_stop", "1")
end

A second process — a human checking in from another terminal, or another program entirely — can then ask, at any moment: signal_query(PORT, "status", "") and get back a real, current answer, without having to guess from log output or process-list CPU usage whether anything useful is actually happening.

2. The queue: "if this crashes right now, what happens?"

Signals are live and in-memory — the moment either side stops running, that connection is gone. That's fine for "are you alive right now," and completely wrong for anything that must not be silently lost. For that, PatLang's message queue gives you a durable, file-backed publish/consume/acknowledge cycle: a unit of work is written to disk, a worker picks it up, and only an explicit acknowledgement — sent after the work is genuinely finished — removes it. If the worker dies half way through, the work is still sitting there, untouched, the next time anything looks.

This is exactly the pairing your own long-running project components should have: a live signal for "what's happening now," and a durable, disk-backed record for "what's actually been safely completed so far." The second one is what lets you build a genuine save-and-resume capability — kill a long job on purpose, restart it later, and have it pick up from the last acknowledged checkpoint instead of starting from scratch. PatLang's own message queue has been proven against exactly this scenario for real: two worker processes completing their work normally, a third crashing outright on a genuine fatal error mid-task, and a fourth, entirely separate process later finding that abandoned work and finishing it — not a simulation of the idea, four actual separate processes.

3. Budgeted yields: making sure the first two things actually get a turn

Here is the part that's easy to miss, and it's the part worth paying the most attention to. A status handler and a queue checkpoint are useless if the program never actually stops to check for them. If your program's real work is a single, long, uninterrupted loop — crunching through a big computation, walking a large dataset — and that loop never hands control back to anything else, your status handler can be written perfectly and still never fire until the whole loop finishes, because nothing inside the loop ever gives the signal-polling mechanism a chance to run.

PatLang's answer to this is budgeted(ms) { ... }: a cooperative, time-sliced loop that inserts a cheap check on every loop iteration, and voluntarily yields control back once a timeslice is up — giving whatever polling loop is driving your signal handlers, and any queue checkpoint you want to write, a genuine, regular opportunity to run in between chunks of real work. Without this, "instrumented" is a polite fiction: the API exists, but nothing ever calls it in time to matter.

Why this is worth the extra design effort

  • It turns "is it working?" from a guess into a fact. For any project component that might run for more than a few seconds — and especially for anything you'll demo live, or anything a marker might run themselves — the difference between "I think it's still going" and "it just told me it's on step 340 of 500" is the difference between a stressful wait and genuine confidence.
  • It gives you real evidence to put in a report. A component that can report its own progress, on demand, gives you something concrete to screenshot, log, and cite as evidence during your evaluation — exactly the kind of artefact this site's own project-guidance material argues you should be generating continuously rather than reconstructing after the fact.
  • It's a real risk-mitigation decision, not gold-plating. A save-and-resume capability directly reduces the impact of a crash, a power cut, or an accidentally-killed process during a long run — the same "reduce impact" mitigation strategy taught in this site's Risk Management material, applied concretely rather than left abstract.

The part worth being genuinely careful about

PatLang is being built by a human directing an AI coding assistant (Claude) doing the actual implementation work — and this specific piece of architecture is a documented example of where that pairing has repeatedly fallen short, on the record. Two distinct failure patterns show up:

  1. The instrumentation gets left out entirely, unless explicitly asked for. A program gets built, works correctly, and simply never gains a status handler or a durable checkpoint — because nothing about "make this program do X" implies "and also make it report on itself while doing X" unless that's stated as its own explicit requirement. This is not a PatLang-specific problem: it's a general pattern with AI coding assistants across every language — they build what was asked for, precisely, and non-functional requirements like observability tend to be the first thing left off when they weren't named directly.
  2. The instrumentation gets added, but doesn't actually work in time. Even when a status handler exists, if the surrounding loop is a single tight, uninterrupted block of computation with no budgeted() yield anywhere inside it, the handler is real code that's real dead weight — it exists, it's correct, and it still won't answer a query until the whole loop finishes anyway, which defeats the entire point. This is a subtler mistake than leaving the feature out altogether, because it looks finished: the API is there, the handler is there, and it silently doesn't deliver what it appears to promise until someone actually tests it while the real workload is running, not just on a short demo case.

The practical lesson for your own project: if you're using an AI assistant to help build a long-running component, don't assume health/progress monitoring or resumability will be included by default, and don't assume that because a status handler exists, it actually reports promptly. Ask for both explicitly, and then test the claim, not just the code — start a realistic-sized job, and actually query its status while it's still genuinely busy with the real workload, not a toy-sized one. If the response is slow or doesn't come until the job finishes, the instrumentation exists but isn't doing its job, and the fix is almost always a missing yield point inside whatever loop is doing the real work.

A checklist for your own long-running components

  • Does this component have a way to report its current state on demand, without needing to be restarted or have its logs tailed?
  • Does it have a quit-equivalent that stops it cleanly, rather than requiring it to be killed outright?
  • If it does meaningful work in chunks (records, iterations, batches), is progress checkpointed durably enough that a crash loses at most one chunk of work, not everything since the start?
  • If its main work is a single big loop, does that loop actually yield control periodically — and have you proven this by querying its status while a realistically-sized job is still running, not just a short test case?
  • If you asked an AI assistant to build this component, did you ask for health/progress/resume support explicitly, as its own requirement — and did you verify it actually behaves as claimed, rather than trusting that it was included because it sounds like standard practice?