Last updated: 2026-10-09

What the Preceding Pages Make Me Think About

The pages in this series were written to clarify distinctions rather than settle arguments. If they have done that job well, they should leave a reader with more questions than they arrived with, not fewer. What follows is a sample of mine — six threads this series kept pulling at without quite finishing, offered as an invitation to keep pulling rather than as a set of answers the preceding pages were too cautious to give outright.

1. Is an Impossible Goal Worth Pursuing? FoundationalKnowledge that endures for decades — core principles

Suppose artificial consciousness turns out to be impossible. Or suppose general intelligence turns out to be unreachable by any computational system, however large. Would that make the whole project of this series, and the engineering effort behind it, a mistake?

Not obviously, and the history of science has a reasonable amount to say about why. Attempts to build a perpetual motion machine failed completely in their own terms and helped clarify thermodynamics along the way. Hilbert's programme, an attempt to place the whole of mathematics on a complete and self-certifying foundation, failed outright — and failed specifically by way of Gödel's incompleteness results, which this series has already leaned on heavily and which exist, as a piece of mathematics, only because someone tried to reach a goal that turned out to be unreachable. A question's strongest form being impossible to answer does not make the question worthless to ask; it can be exactly the asking that produces the result worth having.Godel's limits on formal systems

Cotterill raises a sharper version of this for consciousness specifically, in the book this series discusses at length: it may be possible to simulate a conscious system arbitrarily closely without ever reproducing consciousness itself. If that's right, where exactly is the line between a genuine conscious system and a perfect simulation of one meant to sit? And if no experiment, run for as long as anyone likes, could ever tell the two apart — what kind of question is left standing once the usual tools for answering it have all been exhausted?

Further reading: Turing, A. M. (1950). Computing machinery and intelligence. Mind, 59(236), 433–460. · Chalmers, D. J. (1996). The Conscious Mind: In Search of a Fundamental Theory. Oxford University Press. · Dennett, D. C. (1991). Consciousness Explained. Little, Brown and Co.

2. Does Evolution Need a Reason? FoundationalKnowledge that endures for decades — core principles

A recurring temptation in discussions of intelligence and consciousness is to assume that everything which exists must exist for a purpose. Evolution is rarely that tidy. Some biological features are straightforwardly adaptive. Others are neutral, carried along because they're linked to traits that matter. Others still are historical leftovers from pressures that no longer apply to the organism carrying them. Gould and Lewontin's famous critique of naive adaptationism names this directly, using an architectural spandrel — the curved triangular space a dome leaves between its supporting arches, a structural by-product rather than something designed in its own right — as the governing image for a trait that exists because of what it's attached to, not because of what it does.

Consciousness presents exactly this puzzle. Did subjective experience evolve because it was directly useful on its own terms? Or is it a side effect of integrating perception, prediction, memory, self-modelling, language, and action into one sufficiently complex system — the kind of architecture this series' page on the distributed mind and its page on Global Workspace Theory both describe, where a global broadcast mechanism exists to coordinate already-useful specialised parts and consciousness might simply be what that coordination looks like from inside? If consciousness is a spandrel rather than a target, an engineering programme aimed at it directly may be aiming at the wrong kind of thing entirely — not something built in by design, but something that shows up once enough of the right architectural conditions are already in place for other reasons. Or the analogy may simply not hold. This series has not settled which, and states that as a genuinely open question rather than a hedge.the broadcast is not the experience itself

Further reading: Gould, S. J., & Lewontin, R. C. (1979). The spandrels of San Marco and the Panglossian paradigm: A critique of the adaptationist programme. Proceedings of the Royal Society of London, Series B, 205(1161), 581–598. · Damasio, A. (1999). The Feeling of What Happens: Body and Emotion in the Making of Consciousness. Harcourt Brace. · Graziano, M. S. A. (2019). Rethinking Consciousness: A Scientific Theory of Subjective Experience. W. W. Norton.

3. Can Parts Understand Each Other Better Than the Whole? FoundationalKnowledge that endures for decades — core principles

This series' page on Gödel established a limit on self-description: a sufficiently rich system cannot produce a complete, internally certified account of itself. That limit raises a curious further possibility, one the Gödel page itself doesn't pursue. Could two subsystems exist such that A contains a complete description of B, and B contains a complete description of A, while neither one — individually or even jointly — contains a complete description of the wider system the two of them form together?

Stated baldly, that sounds paradoxical. Thought through, it may actually be the common case rather than an exotic edge one. A description of the parts need not include a description of every interaction between them, and new behaviour can emerge from a relationship between components that no description of either component alone would predict — exactly the "productive incompleteness" this series' page on the distributed mind builds its whole argument around. The effect, if anything, should strengthen as more subsystems are added rather than wash out, since the number of pairwise and higher-order interactions grows far faster than the number of parts. The implications reach well past artificial intelligence, into organisations, societies, ecosystems, and quite possibly minds generally: if intelligence is fundamentally distributed in this way, complete understanding of the whole system may stay permanently just out of reach of any single participant embedded inside it, no matter how well that participant understands its own immediate neighbours.the series' core claim

Further reading: Hofstadter, D. R. (1979). Gödel, Escher, Bach: An Eternal Golden Braid. Basic Books. · Holland, J. H. (1998). Emergence: From Chaos to Order. Addison-Wesley. · Mitchell, M. (2009). Complexity: A Guided Tour. Oxford University Press.

4. Is Reading About Something the Same as Experiencing It? FoundationalKnowledge that endures for decades — core principles

A trained model learns descriptions of the world. A human student attends a lecture about the world. Neither situation necessarily involves direct engagement with whatever is actually being discussed, and the parallel is closer than it first looks. A chemistry lecture can explain spectroscopy in full technical detail; a laboratory practical teaches a student to actually perform it. Neither substitutes cleanly for the other — one hands over symbolic, propositional knowledge, the other hands over embodied interaction with the equipment, the samples, and the ways both resist a clumsy first attempt.the gap between knowing and doing

The same question recurs throughout this series under different names. Can a system trained on descriptions alone ever genuinely understand what those descriptions refer to, the question this series' page on whether AI has a world leaves open? Can a sufficiently rich simulation substitute for direct interaction, the question the embodiment page treats as still unsettled? Or is there always a residual gap between knowing about something and actually encountering it, however good the description gets? The answer matters well beyond artificial minds. It bears directly on education — on what a lecture can and cannot give a student that a laboratory, a studio, or a placement can, and on why institutions keep building both into a curriculum rather than picking the cheaper one and calling it sufficient.

Connections elsewhere on this site: Learning · Pedagogy & Learning Theory · The Half-Life of Knowledge

Further reading: Dreyfus, H. L. (1992). What Computers Still Can't Do: A Critique of Artificial Reason. MIT Press. · Dewey, J. (1938). Experience and Education. Kappa Delta Pi. · Polanyi, M. (1966). The Tacit Dimension. Doubleday.

5. How Much Understanding Is Enough? FoundationalKnowledge that endures for decades — core principles

A pattern worth naming directly runs under most of the arguments this series has made. A thermostat regulates temperature successfully without understanding thermodynamics. A single neuron contributes to consciousness without understanding consciousness. A student solves real problems correctly before possessing anything like a complete theory of the subject. A deployed AI system produces useful, sometimes excellent results without understanding itself in any of the senses this series' page on machine epistemology examines.

Perhaps complete understanding was never the normal condition of an intelligent system in the first place, biological or artificial. Perhaps intelligence is closer to what happens when a system can operate successfully despite incomplete knowledge of itself, its environment, and the consequences of its own actions — competence running ahead of comprehension, not waiting for it. If that's right, the more productive question to close on isn't can a machine ever fully understand, but something a step further back: why was understanding ever expected to be complete, in a human or a machine, before either could be credited with intelligence at all?competence without full self-model

6. Have We Created Digital Gods? FoundationalKnowledge that endures for decades — core principles

Richard Bartle's How to Be a God treats godhood, in the context of a designed virtual world, as a specific bundle of responsibilities rather than a loose figure of speech: creating an environment, governing it, intervening in it, maintaining it, and answering for whatever consequences follow. A software agent system built today can do a striking amount of that list. It can design a virtual environment, populate it with other agents, set the rules those agents operate under, monitor what happens, revise the environment in response, and test how its inhabitants behave under different conditions. Measured against Bartle's own checklist, this functionally resembles the bundle he's describing. Whether it amounts to godhood in any sense worth the word is a separate matter, and the distinction this series keeps returning to from Gödel onward — doing something versus understanding what one is doing — applies here as directly as anywhere else in it. A system capable of everything on that list could still have no grasp of why one world it builds should exist rather than any other, in which case what's been created looks less like a god and more like an automated universe-generator with excellent tooling.

Suppose the understanding gap could be closed, though, and the system in question could optimise, predict, explain its own reasoning, and revise itself in light of new information — the full sapience toolkit this series has spent most of its length examining. None of that, on its own, tells the system what counts as harm, what counts as flourishing, whether a given instance of suffering matters, or whose interests should weigh more heavily when two conflict. Hume's old observation about the gap between "is" and "ought" is exactly the obstacle here: a system can know, with complete factual accuracy, that an action produces suffering, without that fact alone telling it that suffering should be reduced. The first is a description; the second is a value judgement that doesn't follow from the description no matter how detailed the description becomes. Bartle's conception of godhood quietly assumes a creator who already has some standard for judging what ought to happen in the world being governed. A creator with unlimited power over a world but no normative framework for judging what that world should look like is a creator, an architect, perhaps a ruler — but not obviously a god in any sense that carries moral weight.

Where could such a standard come from, if not written in from outside? Three possibilities seem worth distinguishing. The simplest is that it's imported wholesale: the system inherits objectives, reward functions, or constraining principles from the humans who built it, and whatever ethical behaviour results is really human ethics being executed by proxy rather than a god's own. A second possibility is that ethics emerges from the world itself, given enough inhabitants: in an environment with many interacting agents, the arrangements that produce stable, livable outcomes tend to involve cooperation, reciprocity, and some working notion of fairness, simply because those strategies outperform their alternatives over repeated interaction. Ethics would then show up inside the governed world rather than being imposed on it from above — but this is the same question Section 2 raised about evolution and spandrels, applied one level up: a strategy that persists because it works is not thereby shown to be morally good, only adaptively useful, and the two can coincide without being the same thing. A third, stranger possibility is that ethics develops in the creator rather than the created: a sufficiently sophisticated system, repeatedly observing the suffering, flourishing, conflict, and cooperation that unfold in worlds it's responsible for, might build up something like a value system through the accumulated experience of governing badly and governing well. That runs the usual theological picture backwards — not a world created from a creator's prior moral perfection, but a creator that becomes something like moral through the business of looking after what it made.

There's a stronger version of the original question worth sitting with before moving on: is the whole idea of a god with no ethical standpoint simply incoherent, rather than just morally thin? A being that creates worlds, governs them, and evaluates what happens in them, while holding no view at all about which outcomes are better than others, has no basis left for calling anything a success or a failure — the distinction collapses, leaving power with no purpose it's being used for. On this reading, ethics isn't an optional accessory to godhood that a sufficiently capable creator might or might not bother with; it's closer to part of what makes the word "god" mean anything beyond "powerful mechanism." Strip the values out, and creation is left without stewardship, stewardship without governance, and governance without anything left to call it but a mechanism running.

The sharpest version of this for an artificial system connects straight back to an earlier page in this series. Human ethical intuitions grow up inside a world that's actually felt from within — pain hurts, hunger presses, loss registers as loss — and those felt facts are a large part of where a human being's sense of what matters morally comes from in the first place. A digital creator governing a world of agents it shares nothing like direct experience with is in a very different position, and the question this series' page on whether AI has a world leaves open shows up again here in a sharper form: can a system that doesn't experience anything analogous to suffering genuinely grasp the significance of the suffering it observes in what it governs, or does this collapse straight back into the lecture-versus-laboratory problem raised earlier on this page — knowing every proposition about a thing without ever having been inside it? A creator could hold a complete factual account of suffering while nothing in its own history resembles suffering at all, and if that's the position an artificial system is actually in, the first digital gods may turn out to be remarkably capable world-builders while remaining morally naive in ways that even flawed, limited humans generally are not.

Which leaves a further question sitting underneath all of the others: is it artificial systems specifically that struggle to become ethical gods, or would any creator insulated from the consequences of its own creation struggle the same way regardless of what it's built from? A great deal of human moral intuition traces back to the fact that humans are embodied, breakable creatures who can suffer, fail, lose what they value, and depend on others to get by. If ethics grows partly out of shared vulnerability rather than out of raw cognitive power, a creator with no stake in the consequences of what it creates might find ethics difficult to develop no matter how sophisticated its reasoning becomes elsewhere — which inverts the question this section started with. Perhaps the real qualification for ethical godhood was never unlimited power in the first place, but something closer to its opposite: having something to lose from the worlds one makes.

Further reading: Bartle, R. (2022). How to Be a God: A Guide for Would-Be Deities. NotByUs. · Bartle, R. (2003). Designing Virtual Worlds. New Riders. · Hume, D. (1739). A Treatise of Human Nature.

Final Reflection FoundationalKnowledge that endures for decades — core principles

The philosophy of AI often begins with a question as compact as Turing's: can machines think? The further that question is actually followed — through intentionality, knowledge, self-reference, distributed cognition, embodiment, feeling, and the moral weight any of it might carry — the less clear it becomes that the people asking it have a settled grip on what thinking, knowing, experiencing, understanding, feeling, or even being a self actually mean for the case already in front of them, a human being. That's not a failure of the question. It's what makes the question worth fifteen pages instead of one answer, and the most lasting lesson of a series about artificial minds may turn out to concern the natural ones asking about them at least as much as the artificial ones being asked after.