Avoiding the AI Short-Circuit: Using Generative AI Without Undermining Your Own Learning

Executive Summary

TheThe strongest evidence currently available on generative AI and learning outcomes — a 30-month panel study of 26,811 secondary students — finds that using GenAI to complete homework raises assignment scores by 18% and cuts completion time by 30%, while lowering closed-book exam scores by 20% within six months, with the full penalty on high-stakes entrance exams reaching 18–24% after about two years ^[1]. The mechanism is not mysterious once cognitive-science research on learning is brought in alongside it: coursework score and retained understanding are different things, measured differently, and an intervention that helps one can straightforwardly do nothing for — or actively harm — the other. This page sets out what the evidence actually shows (and how strong each piece of it really is), the cognitive-science mechanism that explains why outsourcing execution to AI produces this specific pattern, four practical strategies for using GenAI as a critique partner rather than an author, and a direct parallel to a theme that runs through the software-engineering and pedagogy material elsewhere on this site: optimising a process is a better long-run investment than optimising one output of it.

The Evidence: What the Research Actually Shows

Strömberg,Strömberg, Lei, and Wu's study is the strongest single piece of evidence here, and it is worth being precise about why: it is a genuine panel — the same 26,811 students tracked for 30 months across monthly closed-book exams, homework scores and completion times in nine subjects, and eventual high-school and college entrance exams — not a one-off survey or a lab experiment with a few dozen participants ^[1]. That design is what lets the authors separate two groups whose submitted homework looks identical: students using AI as a tutor or critic, whose exam scores were largely unaffected, and the roughly 80% of AI users whose usage pattern — exceptionally short completion time paired with high homework scores — is consistent with outsourcing the work outright, among whom the exam penalty concentrates ^[1]. The penalty is also not evenly spread: it is largest in social-science subjects, followed by STEM and languages, and hits younger students, high-achieving students, and boys hardest ^[1]. One caveat worth stating plainly: this is 2026 working-paper evidence (CEPR Discussion Paper No. 21577, also circulated via SSRN) from Chinese secondary education specifically — not yet a peer-reviewed publication, and not directly evidence about UK higher education, where assessment structure, AI-detection norms, and student age all differ. The size and design of the underlying panel make it credible evidence, not proof that the exact same magnitudes transfer to a different education system.

A second, independent line of evidence points the same direction with a different method and a real limitation of its own. Gerlich's 2025 survey of 666 participants found a statistically significant negative correlation between frequency of AI-tool use and critical-thinking performance, and that the relationship was mediated by self-reported cognitive offloading — delegating a mental task to a tool rather than doing it yourself ^[2]. This should be weighted differently from the Strömberg panel: it is cross-sectional survey data, which cannot establish that AI use causes lower critical-thinking scores rather than, say, students already inclined to offload cognitive effort simply using AI tools more — the paper itself has an issued correction, a reminder to check a source's current status rather than its first-found version ^[2]. Taken together, though, a causally-strong panel study and a correlational survey using an entirely different method are converging on the same pattern, which is a more solid evidential position than either study would provide alone.

Why This Happens: The Cognitive Science Behind the Penalty

NoneNone of this requires treating AI as uniquely dangerous — it is a specific case of a much older, well-established finding about how memory works. Craik and Lockhart's levels-of-processing framework established that how deeply material is processed at encoding — shallow, surface-feature processing versus effortful, meaning-based, elaborative processing — predicts how well it is later recalled, independent of how well a task built from that material was performed at the time ^[3]. Letting AI generate the solution and reading it over is shallow processing by this definition, however correct the output is; attempting the problem yourself first is the effortful, elaborative kind. Bjork's concept of desirable difficulties names the practical consequence directly: certain conditions that make learning feel harder and slower in the moment — retrieval practice, spacing, generating your own attempt before seeing a worked solution — reliably produce stronger, more durable memory than conditions that feel easier, because the difficulty is what triggers the encoding and retrieval processes that actually build a lasting representation ^[4]. Removing that difficulty by letting AI do the work removes exactly the mechanism that was going to build the memory.

Roediger and Karpicke's testing-effect experiments make the timing of the trap especially clear, and it maps almost exactly onto the AI short-circuit pattern. Students who studied material repeatedly, versus students who were tested on it repeatedly with no feedback, were compared on a final test given either 5 minutes or up to a week later. At 5 minutes, repeated studying looked better. On every delayed test, repeated testing won by a wide margin ^[5]. Immediate performance and durable learning were driven in opposite directions by the same manipulation — which is precisely the shape of the AI short-circuit: better immediate output (the homework score), worse delayed output (the exam, weeks or months later), from the same underlying choice to skip the effortful step.

Four Strategies for Using GenAI as a Co-Pilot, Not an Outsourcer

TheThe practical response these findings point to is not "avoid AI" — the Strömberg panel's own tutor/critic users show minimal learning loss ^[1] — it is to structure AI use so the desirable difficulty stays in place.

1. Struggle first, prompt second. Spend real time attempting the problem — a draft outline, pseudocode, a sketch of an argument — before opening an AI tool. This is the direct, practical version of the levels-of-processing and desirable-difficulties research above: the attempt itself is the effortful encoding step, and no amount of reading a subsequently-generated correct answer substitutes for having done it.
2. Use the Socratic-tutor pattern, not the direct-answer pattern. "Write an analysis of X" hands the elaborative work to the model. "Here is my thesis and two arguments — quiz me on three potential flaws" keeps the retrieval and evaluation work with you, and gets you something closer to Roediger and Karpicke's tested condition than their studied one.
3. Treat AI as a critique partner, not an author. Ask it to find the error in your own worked solution rather than to produce one from scratch. This keeps AI in the role the tutor/critic group in the Strömberg data occupied — the group with minimal measured learning loss — rather than the outsourcing role the other 80% fell into.
4. Retest yourself, unassisted, after a delay. Close the tool, and 24 hours later attempt a similar problem on paper with no AI help. This is a direct, personal replication of Roediger and Karpicke's own delayed-test condition: if you can't do it unassisted after the delay, the immediate sense of having understood the AI's explanation was the 5-minute result, not the one-week one, and it was misleading you about what you'd actually learned.

A Parallel Elsewhere on This Site: Process Over Product

TheThe underlying trade-off here — a metric that looks good right now (a mark, a passing test, a shipped feature) versus the durable thing that metric was supposed to be a proxy for (retained understanding, correct software, a graduate who can work without a supplied scaffold) — is not specific to GenAI, and this site has independently converged on versions of the same point from software engineering and from other pedagogy pages. Risk Management: Lessons from Testing Real Systems states the software-engineering version directly: "A passing test suite tells you the system behaves correctly in every state your tests actually put it into — not in every state it can actually be in." A test suite that only exercises the easy paths is the coding equivalent of a homework score inflated by outsourced execution — both look like success right up until something (a delayed exam, a real user, an unusual input) finally exercises the state that was never actually built or checked. The Journey of Building PatLang series, an extensive real-world case study in AI-assisted development written directly from that project's own commit history, repeatedly finds the same shape of failure: work that looked complete because a proxy metric (a test pass-rate, a confident-sounding commit message) said so, corrected only once someone actually exercised the real thing rather than trusting the proxy. The specific discipline that project settled on — verify empirically before believing a claim, re-run the exact failing case rather than trusting that a fix looks right, treat "it compiled" and "it works" as different claims — is the software-development analogue of struggling first and retesting unassisted: in both domains, the fix for a shortcut that inflates a proxy metric is deliberately generating the effortful, checkable evidence the shortcut skipped.

The same pattern shows up again, from a third independent direction, in Evaluating Detailed Rubrics's discussion of scaffolding: Wood, Bruner, and Ross's foundational account is explicit that the entire point of a scaffold is that it gets withdrawn deliberately as competence develops, not left in place permanently ^[6]. A rubric supplied at maximal detail throughout a degree and a GenAI tool used as an outsourcer rather than a critic are structurally the same risk wearing different clothes: both are genuinely useful scaffolds that, left in place without ever fading, prevent a student from ever practising the judgement the scaffold was supposed to be temporary support for. The fix in both cases is the same shape, too — not removing the scaffold, but using it deliberately in a way that shrinks over time rather than substitutes indefinitely.

What This Actually Means in Practice

• The single biggest predictor of exam performance is time spent directly wrestling with problem complexity — not time spent, full stop, and not coursework score. Shortcut tools raise the latter while doing nothing for, or harming, the former.
• Use AI to challenge your work, not to produce it. The tutor/critic pattern shows minimal measured learning loss in the strongest available evidence; the outsourcing pattern is where essentially all of the measured penalty concentrates.
• Treat immediate ease as a warning sign, not reassurance. The testing-effect research shows immediate performance and delayed retention can be driven in opposite directions by the same shortcut — a topic that felt clear right after an AI explanation is exactly the case a delayed, unassisted retest is designed to catch.
• Retest yourself unassisted, on a delay, before trusting that you've learned something. This is cheap, and it is the single most direct way to tell whether an AI-assisted session produced understanding or just a temporarily convincing explanation.
• This is a general pattern, not an AI-specific one. The same "proxy metric looks fine, the thing it was meant to measure quietly didn't happen" trade-off shows up in software testing and in rubric design elsewhere on this site — recognising the pattern once makes it easier to catch in a new setting.

References

1. Strömberg, D., Lei, V., & Wu, Y. (2026). The Generative AI Learning Penalty: Evidence from Chinese Secondary Education (CEPR Discussion Paper No. 21577). Centre for Economic Policy Research. Also circulated as an SSRN working paper.
2. Gerlich, M. (2025). AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking. Societies, 15(1), Article 6. https://doi.org/10.3390/soc15010006 (correction subsequently published for the same article).
3. Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11(6), 671–684. https://doi.org/10.1016/S0022-5371(72)80001-X
4. Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. P. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). MIT Press.
5. Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249–255. https://doi.org/10.1111/j.1467-9280.2006.01693.x
6. Wood, D., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2), 89–100. https://doi.org/10.1111/j.1469-7610.1976.tb00381.x