Does AI Have a World? Realism, Idealism, and Representation

AA model trained purely on text has only ever seen the word "red" co-occur with other words — "blood," "stop," "anger," "wavelength" — never a single photon. A model trained on paired text and images has seen pixel patterns correlate with the word too. Neither training regime gives the system a world in any sense beyond a very large, very structured set of correlations among its own inputs. Whether that's a real limitation, or simply an accurate description of what any knower — human or artificial — actually has, is this page's question.

The Symbol Grounding Problem FoundationalKnowledge that endures for decades — core principles

Harnad'sHarnad's symbol grounding problem states the worry precisely: a purely symbolic system's symbols get their meaning only from their relations to other symbols, in a closed loop that never touches anything outside the symbol system itself — like trying to learn Chinese from a Chinese-only dictionary, where every definition just points to more Chinese words¹. This series' page on intentionality already develops the closely related original/derived distinction; grounding is the proposed fix on Harnad's own account — some symbols need to connect to the world through something other than more symbols (sensorimotor categorisation, on his proposal) before the whole system's reference can be anything but an ungrounded, self-referential loop.

Does Multimodal Training Actually Ground Anything? FoundationalKnowledge that endures for decades — core principles

PairingPairing text with images looks like exactly the fix Harnad's problem calls for — a non-symbolic sensory channel connected to the symbolic one. Whether it actually is depends on a distinction worth holding onto carefully: correlating two representational streams (pixel statistics reliably co-occurring with word statistics) is not obviously the same achievement as grounding a symbol in the world those streams are both representations of. A camera's pixel array is itself already a representation, mediated by optics, sensor response curves, and compression — one more structured stream of data, not unmediated contact with a photon. Multimodal training may genuinely reduce the problem (two independently-sampled representational streams constraining each other is a real improvement on one closed loop) without fully dissolving it (both streams are still representations, correlated with each other rather than directly anchored in whatever they're representations of).

Realism and Idealism, Properly Stated FoundationalKnowledge that endures for decades — core principles

IdealismIdealism is routinely strawmanned as the claim that nothing exists outside the mind, which almost no serious idealist has defended. The more serious claim, descending from Kant and relevant to this series' page on empiricism and rationalism, is that the world as it can be known is partly constituted by the knower's own representational apparatus — not that reality depends on being perceived, but that the structured, categorised, conceptually organised world any knower has access to is never the same thing as whatever reality is independent of all representation. Realism, by contrast, holds that the objects a representation is about have a determinate nature independent of any particular representation of them, which the representation can get right or wrong. Applied to a trained model: the realist asks whether the model's internal representations track real, mind-independent structure in the world; the idealist-leaning question asks whether "the world" available to the model is, in a precise sense, constituted by its own representational categories — concepts the training process converged on, not concepts simply read off a pre-existing structure waiting to be discovered.

Are Concepts Discovered or Constructed? FoundationalKnowledge that endures for decades — core principles

AA trained model's internal concept boundaries are themselves an empirical question answerable only partially: training on a different corpus, with a different architecture or objective, measurably produces different concept boundaries for the same words — which is exactly what the construction side of this question predicts, and exactly what the previous page's architecture-and-objective dependence would lead you to expect. That a concept's boundary is partly a function of the learner doesn't settle whether it's entirely a function of the learner, with nothing about the world constraining it at all — a realist can grant that training choices shape exactly where a concept's boundary falls while insisting the concept still has to answer to real structure in the world to be useful at all, the way a map's projection is a human choice while the coastline it's projecting still has to be there.

Strongest Objection: This Dispute May Not Be Resolvable From Inside Either Position FoundationalKnowledge that endures for decades — core principles

WhetherWhether a system's representations track mind-independent reality or are constituted by its own representational categories is not obviously the kind of question the system's own outputs could ever settle, for either side — a model producing accurate, useful predictions is equally consistent with "my concepts track the world" and "my concepts are a well-fitted internal structure that happens to predict my own future inputs well," since both would produce the same successful behaviour. This isn't a special defect of trained models; the same underdetermination between realist and idealist readings of success has been argued over human perception and science for centuries. What's specific to this page's case is that a trained model's concept boundaries are at least partly traceable to documented, manipulable causes (architecture, objective, data) in a way human concept formation isn't, which gives this version of the dispute more empirical purchase than the historical one, even if it doesn't resolve it.

Provisional Conclusion FoundationalKnowledge that endures for decades — core principles

AA purely symbolic system's reference is genuinely vulnerable to Harnad's grounding problem, and multimodal training is a real but incomplete answer to it — correlating representational streams, not obviously anchoring any of them outside representation altogether. Whether what results counts as "having a world," in either a realist or an idealist-leaning sense, is a question this page leaves open deliberately: both readings are consistent with everything a system's successful behaviour can show, and this underdetermination is itself the finding, not a gap waiting for a cleverer experiment to close.

Questions for Further Thought

• What kind of evidence, in principle, could distinguish "correlated representational streams" from "a grounded symbol," rather than just asserting one or the other?
• Does adding a robotic sensorimotor channel (rather than another passive data stream) change the grounding question, or just add a third correlated stream?
• If a system's concepts are partly constructed by its training choices, does that make them less real, or only differently caused than a concept arrived at some other way?

Related Topics

• Intentionality: What Makes a State About Something? — the original/derived distinction this page's grounding question directly extends.
• Empiricism, Rationalism, and Machine Learning — the architecture-dependence of concept formation this page's "discovered or constructed" question builds on.
• Meaning, Ontology, and the Limits of Fixed Definitions — this site's fuller treatment of what fixes a concept's boundaries, referenced rather than repeated here.

Further Reading

• Harnad, S. (1990). The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1–3), 335–346.

References