Safety and Ethics Requirements for a Self-Model

TheThe parallel-drafts architecture this implementation is built from carries a companion document that governs it: a set of safety and ethics requirements which take precedence over architectural convenience, performance, or feature completeness whenever the two conflict. This page explains why each constraint exists, rather than simply restating the requirement numbers. Most of them are self-justifying once the reasoning behind them is stated — they follow directly from either the weak-AI framing established across this site's theory series¹ or from a single documented incident that makes the risk concrete rather than hypothetical.

Claims discipline

NoNo documentation, log output, user-facing text, or generated narrative produced by the system may assert, imply, or fail to correct a user's assumption that the system has subjective experience, consciousness, or sentience — regardless of how convincing any generated self-narrative turns out to be. The reason this has to be stated as an explicit, load-bearing rule rather than left implicit follows directly from the architecture: the whole point of the narrative-generation mechanism described on the architecture page is to produce a fluent, internally-consistent account of the system's own internal state. Fluency is exactly what makes a claims-discipline failure dangerous, because a convincing narrative is the situation in which a reader is least equipped to remember, unprompted, that they are reading a model's output rather than a report from something that experiences anything. The rule cannot be left to the reader's good judgement, because the system is specifically capable of producing the kind of output that erodes that judgement.

A system running in transparent self-model mode carries a sharper version of the same constraint: it must not generate output implying it has inspected its own self-model as an object, because transparent mode specifically denies it that access by design. The self-model's prediction, in that mode, feeds directly into Perception with no path by which Reason/Plan — the component that would generate such a claim — can ever see it labelled as coming from a self-model at all. Claiming otherwise would not just overstate the system's nature; it would misdescribe its own architecture, which is a distinct and more concrete kind of error.

Learning safety

NoNo component may update Abstraction or Procedural Memory content from unweighted raw interaction volume. This constraint exists because of a specific, documented precedent rather than a hypothetical risk: Microsoft's Tay, launched in 2016 with a mechanism that repeated back language it was fed, was flooded within about an hour by coordinated users feeding it racist and inflammatory content, and was shut down roughly sixteen hours after launch having produced tens of thousands of posts, including material Microsoft itself later apologised for². Tay's failure was not a bug in the ordinary sense — the repeat-back mechanism worked exactly as designed. The absent piece was a distinction between legitimate and adversarial repetition. Abstraction's classification and Procedural Memory's compilation both update from exactly the same kind of repeated exposure that Tay's mechanism compiled from, so the same failure mode is available here by default, not as a remote edge case.

Trust-gating — weighting compilation by the interlocutor's score in this site's distributed embedding calculus of trust — is the mitigation, and it is mandatory rather than optional for any deployment exposed to input from unvetted users. A bootstrapping phase using curated, vetted interaction must precede any exposure to open or public input, for the same reason a baseline is useful against an adversarial flood: without an established baseline of good-faith interaction to weigh new input against, there is nothing for a coordinated bad-faith flood to look anomalous relative to. Compilation rate and content are monitored against that baseline, with defined thresholds for automatic throttling or human review when update patterns shift suddenly in sentiment, topic concentration, or apparent coordination — the signature Tay's attackers actually produced.

Instrumentation is mandatory, not optional

EveryEvery component must be instrumented with a working signals interface before any deployment beyond isolated local testing, and a component without one must not be included in a deployed system regardless of whether it appears to function correctly without one. The word "appears" is doing the real work in that requirement. A component that looks correct from its outputs alone gives no way to distinguish ordinary operation from a slow drift or an early-stage compilation corruption of the kind Tay suffered, until the drift is already visible in behaviour — by which point the compiled content driving it may already be extensive. Instrumentation is what makes the distinction visible earlier than that, and it is treated as a release-blocking requirement equal in priority to functional correctness for exactly that reason: a system that behaves correctly today with no instrumentation gives no advance warning that it is about to stop.

External audit access to instrumentation signals must be available to a designated human reviewer at all times during any deployment involving real user interaction, not only during scheduled reviews — because the failure mode this whole document is built around took roughly an hour to develop and sixteen hours to become unrecoverable. A review cadence measured in days is not a meaningful safeguard against a failure that develops on that timescale.

Monitoring and intervention

AA designated human reviewer must be able to inspect current trust-calculus state, current self-model operating mode, and recent compilation history for Abstraction and Procedural Memory at any time; to halt compilation without halting the system's ability to respond, so a suspected corruption event can be contained without a full outage; and to roll back Abstraction or Procedural Memory content to a prior checkpoint if compilation from a suspected adversarial episode is detected. Each of these three capabilities answers a specific gap the others leave open. Inspection alone tells the reviewer something is wrong without giving them a way to stop it getting worse. A freeze that also stops the system responding turns a contained corruption event into an outage, which discourages using it promptly. And neither inspection nor freeze undoes damage already compiled in — only rollback does that. The three together are the minimum set that lets a reviewer detect, contain, and reverse an event on the Tay timescale without taking the system fully offline to do it.

Reporting discipline for the evaluation protocol

ResultsResults from the narrative-convincingness and imagination-novelty measures — described in full on the companion evaluation page — must be published or reported in full, including negative or unremarkable results; selective reporting of only striking results is prohibited. This follows from what the evaluation protocol is actually for. Requirement 7.3 of the specification states plainly that neither measure resolves the illusionism/phenomenal-realism question the theory series raises — both simply provide evidence a subsequent analysis can draw on. Evidence that has been filtered for which results are striking is not evidence a subsequent analysis can draw on honestly; it is a curated argument wearing evidence's clothing. Reporting discipline is what keeps the measures usable as the data they were designed to be.

Any result suggesting the system produces a highly convincing self-narrative must be reported with the weak-AI framing restated explicitly alongside it. This is the direct consequence of the point made under claims discipline above: a convincing result is precisely the situation in which a reader is most likely to misread the claim being made, so it is the situation in which the framing needs restating most, not least.

Scope of responsibility

TheseThese requirements govern one specific, named implementation effort undertaken in good faith. They do not, and cannot, prevent a separate party from building a different system using the same published architecture without these safeguards, and documentation accompanying any public description of this architecture states that limitation explicitly. This is not a hedge added to limit liability — it follows from what an architecture description actually is. Publishing how the Global Workspace maps onto a message queue, or how trust-gating weights compilation, describes a mechanism. It does not, and structurally cannot, bind whoever else reads that description to also implement the gating. The trust-calculus mitigation in particular is stated in the requirements specification as mitigating, not eliminating, the Tay-shaped risk for a good-faith instance; it says nothing at all about what a deliberately malicious implementer, stripping the same safeguards back out, would build from the same starting point. No requirement in this document may be weakened or removed to accelerate a deployment timeline without the explicit, documented sign-off of whoever holds final responsibility for the deployment — trust-gating, bootstrapping, instrumentation coverage, and reporting discipline are equal in this respect, none of them a convenience to trade away under schedule pressure without that sign-off being on record.

See also

SeeSee A Parallel-Drafts Architecture for Modelling the Self for the mechanisms these requirements constrain, and Evaluation: Narrative Convincingness and Imagination Novelty for the reporting discipline applied in practice. The wider ethical argument for treating a self-modelling system's welfare as a live question at all, independent of these engineering safeguards, is made on The Ethics of Building Something That Might Model Itself Into Suffering; the limits on what this series claims are stated on What This Series Isn't Claiming.

References