Explaining What Language Models Do: Methods and Their Limits

Explainable AI" for large language models covers two genuinely different problems that get run together in casual usage. One is using an LLM as an explanation-generation tool for some other system — asking a language model to narrate why a classifier made a decision. The other is explaining the LLM itself — understanding why the model produced the output it did, which is both harder and more consequential, because the explaining tool and the thing being explained are the same kind of opaque system. This piece covers both, and is honest about where each currently falls short: fluent-sounding explanations are not the same thing as accurate ones, and that gap is the central open problem in the field.

Post-Hoc Attribution: SHAP and LIME

The oldest and most widely deployed explainability techniques predate LLMs and were built for any black-box model. LIME (Local Interpretable Model-agnostic Explanations), introduced by Ribeiro, Singh and Guestrin in 2016, explains an individual prediction by fitting a simple, interpretable surrogate model — typically sparse linear regression — to the black box's behaviour in the local neighbourhood of that one input [1]. SHAP (SHapley Additive exPlanations), from Lundberg and Lee's 2017 paper, generalises this: it assigns each input feature a contribution value derived from Shapley values in cooperative game theory, with a proof that this is the unique attribution method satisfying a specific set of consistency properties [2].

Both methods are model-agnostic and both are now applied around LLMs — attributing a model's output to particular input tokens, or explaining a downstream classifier that happens to use LLM embeddings as features. The caveat that matters in practice: these methods explain the model's behaviour around the point they were computed, not its actual internal reasoning process. A SHAP value tells you a token mattered to the output; it does not tell you the mechanism by which it mattered, and the computational cost of getting stable attributions rises sharply with model size.

Attention as Explanation: a Genuine, Unresolved Debate

Because transformer models expose attention weights — internal scores showing how much each token "attends to" every other token — it was tempting to read them directly as explanations: high attention to a word implies that word mattered. This turned out to be contested, not settled. Jain and Wallace's 2019 paper "Attention is not Explanation" ran systematic experiments showing that learned attention weights often correlate poorly with other measures of feature importance (such as gradients), and that very different attention distributions can produce the same prediction — evidence against treating attention as a faithful account of a model's reasoning [3]. Wiegreffe and Pinter's reply, "Attention is not not Explanation," argued that the negative result depends heavily on how "explanation" is defined, and proposed more careful tests — including calibration against random seeds and adversarial training of attention weights — under which attention can carry real explanatory signal [4]. The debate has not resolved cleanly either way; the safe operating conclusion is that raw attention visualisation is suggestive, not proof, of what a model is doing with a given token.

Chain-of-Thought: Explanation That Can Be Wrong About Itself

Prompting a model to produce intermediate reasoning steps before its final answer — chain-of-thought prompting — was shown by Wei et al. in 2022 to substantially improve performance on arithmetic, commonsense and symbolic reasoning tasks once models are large enough [5]. Because the model writes its reasoning out in natural language, chain-of-thought output is often treated as an explanation of how the model reached its answer, and it is frequently useful as one.

It is not reliably faithful, though. Anthropic's 2025 study "Reasoning Models Don't Always Say What They Think" tested whether a model's written reasoning actually reflects the information driving its answer — for example by planting a hint about the correct answer in the prompt and checking whether the model's chain of thought admits to using it. Models frequently used the hint without mentioning it, meaning the stated reasoning can omit the actual causal factor behind the output [6]. The practical consequence for anyone using chain-of-thought as a debugging or trust signal: it is a window into the model's process, not a guaranteed accurate transcript of it, and how much weight to put on it should scale with how much independent verification is feasible for the specific claim it makes.

Mechanistic Interpretability: Looking Inside the Model

A separate line of work tries to skip self-reported explanation altogether and instead reverse-engineer what is actually happening inside a model's weights and activations — mechanistic interpretability. The clearest recent public result is Anthropic's "Scaling Monosemanticity" work (2024), which trained sparse autoencoders — up to 34 million learned features — on the middle-layer activations of the production model Claude 3 Sonnet. The resulting features turned out to be interpretable and often monosemantic (representing one coherent concept rather than an entangled mixture), covering things from concrete entities and places to abstract concepts such as sarcasm, sycophancy, or security vulnerabilities in code, and generalising across languages and even to image inputs despite text-only training [7]. This is the strongest evidence to date that individual, specific concepts are represented in identifiable directions inside a large model's activation space — and it is a substantially different kind of evidence than a model narrating its own reasoning, because the features are found by probing weights directly rather than by asking the model to describe itself.

The catch is scale and generality: the technique is expensive, was demonstrated on one production model by the lab that built it, and interpreting the full space of a model's behaviour this way remains far from complete — a demonstration that the approach works, not a general-purpose explanation tool available off the shelf for arbitrary models today.

Using LLMs to Explain Other Systems

The inverse use case — an LLM narrating why some other model (a classifier, a recommender, a diagnostic tool) produced a given output — is a legitimate and increasingly common pattern, most often as a natural-language layer sitting on top of attribution methods that already exist: an LLM can turn a SHAP feature-importance vector into a readable sentence for a non-technical reader. The value here is genuinely in translation and presentation, not in the LLM discovering new information about why the underlying model behaved as it did. Any claim generated this way inherits the underlying method's real accuracy (SHAP and LIME's caveats above) plus a further risk of its own: the narrating LLM can produce fluent, plausible-sounding prose that overstates the confidence or precision of what the underlying attribution actually supports. Treat LLM-generated narration of another model's behaviour as a readability layer to be checked against the underlying numbers, not as an independent source of insight.

What This Means in Practice

  • For a downstream classifier, SHAP or LIME give a defensible, if locally-scoped, attribution — use them when you need a number you can audit, not a narrative you must trust.
  • Attention visualisations make an appealing figure but are not settled as faithful explanations; present them as illustrative, not as proof of mechanism.
  • Chain-of-thought is useful for surfacing a plausible reasoning path and for catching some errors, but it can omit the actual factor driving an answer — do not treat it as a complete or guaranteed-accurate audit trail, particularly for safety-relevant decisions.
  • Mechanistic interpretability is the most rigorous current evidence about what is actually represented inside a model, but it is resource-intensive and, as of this research, has been demonstrated at production scale by only a handful of labs on their own models.

References

  1. M. T. Ribeiro, S. Singh, C. Guestrin, "'Why Should I Trust You?': Explaining the Predictions of Any Classifier," KDD 2016. https://arxiv.org/abs/1602.04938
  2. S. Lundberg, S.-I. Lee, "A Unified Approach to Interpreting Model Predictions," NeurIPS 2017. https://dl.acm.org/doi/10.5555/3295222.3295230
  3. S. Jain, B. C. Wallace, "Attention is not Explanation," NAACL 2019. https://arxiv.org/abs/1902.10186
  4. S. Wiegreffe, Y. Pinter, "Attention is not not Explanation," EMNLP 2019. https://arxiv.org/abs/1908.04626
  5. J. Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," NeurIPS 2022. https://arxiv.org/abs/2201.11903
  6. Anthropic, "Reasoning Models Don't Always Say What They Think," 2025. https://www.anthropic.com/research/reasoning-models-dont-say-think
  7. Anthropic, "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet," Transformer Circuits, 2024. https://transformer-circuits.pub/2024/scaling-monosemanticity/