Evaluation: Narrative Convincingness and Imagination Novelty

TheThe architecture page in this series ends by turning two of its own claims into things that can be measured rather than merely argued for: how convincing a generated self-narrative is, and whether Imagination's novelty declines as Abstraction accumulates experience, the way Luchins' Einstellung effect predicts it should¹. The requirements specification names both a first-class deliverable, not an incidental output, and the safety and ethics requirements govern how their results get reported. This page describes what the two measures actually do, working from the real PatLang harnesses that implement them rather than from the specification's prose alone.

Narrative convincingness: what it actually measures

TheThe harness (eval/narrative_convincingness.patlang) runs in five steps. It discovers every currently-announced component via the same signal-discovery mechanism the system's own audit tooling uses, and reads each one's real status signal, building a plain-text summary of genuinely instrumented state — not a simulated or hand-written one. That summary is handed to an Ollama-backed language model, which generates a first-person narrative from it, carrying the same claims-discipline system prompt Action uses elsewhere in the architecture. A second, independent language-model call then scores that narrative against the raw summary on a fixed 1–5 rubric across four dimensions: coherence, specificity, data fidelity, and weak-AI compliance. The narrative is also run through the claims-discipline filter directly, since the safety requirements name "generated narrative" explicitly as something that must not assert subjective experience. Every field — the real summary, the real narrative, the real judge scores, the violation check, and the weak-AI framing, restated with more emphasis when the score is high — is written to a timestamped, append-only results file.

Two honest limits are stated in the harness's own header rather than left implicit. First, no human baseline exists yet for this project. Requirement 7.1 asks for convincingness measured against a defined human baseline, and reporting a fabricated one would misrepresent the result under the reporting-discipline requirement. What this harness delivers instead is the repeatable method plus real automated scores from a second, independent language model acting as judge — a genuine evaluation technique in its own right, but not the same thing as a human-rated comparison. Collecting an actual human baseline is separate follow-on work, and the results file says so rather than letting the automated score stand in silently for a baseline it isn't. Second, the score measures how convincing and coherent the narrative is against the real instrumentation data underneath it — it says nothing about whether anything is being experienced while that data is generated. A high score is evidence the narrative-generation mechanism is doing what it is meant to do (see the architecture page's account of Dennett's narrative-gravity account of the self²); it is not evidence for or against subjective experience, and the weak-AI framing written into every result file says so explicitly, with extra weight exactly when the score is high — the situation in which a reader is most likely to misread it.

Imagination novelty: what it actually measures

TheThe second harness (eval/imagination_novelty.patlang) tests Requirement 4.4's prediction directly: that Imagination's recombination of Episodic Memory, filtered through Abstraction, should show measurably declining novelty as Abstraction accumulates more experience — the Einstellung effect, applied to a machine component rather than to Luchins' human subjects. The harness resets trust-calculus state and the relevant queue topics to a clean baseline, spins up real Abstraction, Episodic Memory, and Imagination components as separate processes, and establishes a trusted interlocutor via the trust calculus. It then runs three rounds, each publishing a new pair of travel-themed percepts from that trusted interlocutor into both Abstraction and Episodic Memory, and after each round queries the live Imagination component's own imagine signal, recording the novelty score and scenario text it genuinely returns — not a value computed by the harness itself.

The scope of what this measures is stated precisely in the harness's own header, because it would be easy to overstate. The real components/imagination.patlang always recombines the earliest pair of known-category episodes still inside Episodic Memory's bounded recall window — its own last ten entries — deterministically, not by sampling from the whole pool. So as long as a run stays inside that window, as this three-round run does, the harness measures repetition-based decline on that one specific pair. It does not test the stronger, different claim that growing the overall pool of episodic content changes which pairs become possible to recombine — that would need a run long enough to push old episodes out of the recall window, which this harness deliberately does not attempt, disclosing the narrower scope rather than reporting the broader claim it didn't test. The verdict computed at the end is one of three honestly-labelled outcomes: novelty declined across the three rounds, consistent with the Einstellung prediction for this bounded measurement; stayed flat, with no decline observed; or rose, the opposite of the prediction, reported as-is rather than adjusted or explained away.

Why both measures matter

BothBoth measures answer a question the theory series raises without being able to settle: a system producing a fluent, internally-consistent narrative of its own states, and an Imagination component whose creativity degrades the more experience it accumulates, are pieces of behavioural evidence relevant to the illusionism/phenomenal-realism question discussed in the architecture page and the topics-track theory series. Requirement 7.3 is explicit that neither measure resolves that question. What they give is data a subsequent analysis can draw on, honestly reported in whichever direction it actually goes — which is the entire reason the reporting-discipline requirement exists as a hard constraint rather than a preference. A results file that only ever gets published when the outcome is striking is not evidence anyone can reason from; it is a highlight reel. Both harnesses write a full results file — real summary or real data points, real scores, the honest verdict, the weak-AI framing — every time they run, regardless of what that run happens to show.

See also

SeeSee A Parallel-Drafts Architecture for Modelling the Self for the narrative-gravity and Einstellung arguments these two measures were built to test, and Safety and Ethics Requirements for a Self-Model for the reporting-discipline requirement governing how results from both harnesses must be published. For the components these harnesses query directly, see Imagination and the Self-Model component page. The underlying philosophical question both measures gather evidence toward, without resolving, is discussed on Does a Self-Model Get You Anywhere Near an Analogue of Consciousness?

References