Vibe Coding: What Actually Predicts Success

VibeVibe coding" — Andrej Karpathy's term for building software by describing what you want in prose and iterating on an LLM's output rather than reading or editing its code¹ — is now a default way many students and professionals first reach for AI-assisted programming. It's tempting to treat it as skill-agnostic: if the model writes the code, what you already know shouldn't matter much. A preregistered, peer-reviewed study accepted to CHI 2026 tested that assumption directly, with actual measured skills against actual task performance rather than self-report, and found it's wrong in an informative way.

The study

Thorgeirsson,Thorgeirsson, Weidmann, and Su recruited 100 tertiary-level students, measured their computer-science achievement, domain-general cognitive ability, and written-communication proficiency using validated instruments, then had them build small GUI-oriented web applications purely by prompting an LLM agent — no viewing or editing the generated source, matching Karpathy's own "pure" definition of the term¹. Tasks were designed by an eight-expert consensus process and scored by a human grader against a documented rubric².

Two findings matter for how you should actually use these tools:

• Both CS achievement and writing skill independently predict vibe-coding performance (Pearson's r = .39 and r = .29 respectively), and CS achievement remains a significant predictor even after controlling for domain-general cognitive ability — so this isn't just "smarter people do better at everything." In a joint model, CS achievement contributed roughly twice the unique variance writing skill did, but both added independent predictive value; knowing one doesn't make the other redundant.
• Writing skill's effect is substantially explained by prompt quality. A mediation analysis found that the direct association between writing skill and vibe-coding performance became non-significant once human-graded prompt quality was added to the model, with the indirect path (writing skill → prompt quality → performance) accounting for roughly half the total association. Students with stronger writing skills wrote prompts that were rated higher quality, and those higher-quality prompts were what actually produced better applications — not writing skill acting on the outcome by some other route.

The authors are explicit that this is correlational, not causal — the study wasn't designed to prove that improving your writing would improve your vibe-coding output, only that the two are associated and that prompt quality is a plausible mechanism connecting them. They also note a specific, useful negative finding: prior LLM usage did not correlate positively with either CS achievement or vibe-coding performance in their exploratory analysis, so more experience typing into a chat box is not, by itself, doing the work that CS knowledge or writing skill does.

Why this isn't really about writing style

It'sIt's worth being precise about what "writing skill" meant in this study, because it's not prose flair. The construct was written-communication proficiency — organisation, structure, reasoning, evidence, and, in the authors' own words, "the ability to write reasoned and logically structured content" that lets a reader (here, an LLM) actually understand what's being asked. That is the same skill this site's own material on scaffolding argues genAI rewards when it's used well: a request specific enough to reveal exactly what you don't yet understand is doing real cognitive work before the AI ever responds, in a way a vague one-liner isn't³. This study gives that argument a number: on a genuinely observable coding task, the quality of a written specification measurably predicted the quality of the output, independent of how experienced the writer was with AI tools specifically.

And the CS-achievement finding is a check against the opposite assumption — that fluent prose alone is enough, and that programming knowledge becomes irrelevant once you're not writing the code yourself. It doesn't. The authors' own interpretation is that CS achievement likely helps because it teaches problem decomposition and algorithmic thinking — breaking a goal into the smaller, well-specified pieces a prompt actually needs to contain — which is a design skill, not a syntax skill, and survives the move away from writing code by hand.

Practical implications for how you use these tools

• Vibe coding does not substitute for learning to decompose a problem. The skill CS achievement appears to be standing in for here — breaking a goal into precise, checkable sub-requirements — is exactly what a good prompt needs, and it's a skill you build by doing CS fundamentals, not by prompting more.
• Treat prompt-writing as a real specification-writing task, not a casual request. The study's own vibe-coding tasks rewarded the same things a written specification for a human developer would: explicit constraints, unambiguous scope, a clear definition of what "done" looks like. A one-line prompt is a bet that the model will guess your unstated requirements correctly; the data here says that bet gets worse as the task gets less standard.
• Don't mistake fluency with a chat interface for either skill. Prior LLM usage alone did not predict better outcomes in this study. Comfort with the tool is not the same as the underlying ability to specify a problem well or reason about its structure.
• The result is a lower bound on CS knowledge's role, not an upper one. This study deliberately used "pure" vibe coding — no visible source, no edits. In more permissive workflows where you can inspect and edit generated code, the authors expect CS knowledge to matter at least as much, through direct edits and targeted fixes as well as through prompt quality.

A caveat worth keeping

ThisThis is one preregistered study, on one task domain (GUI-oriented web applications), with one population (tertiary students at institutions in a single metropolitan area), and the authors say so plainly in their own limitations section. It's evidence, not a settled law — but it's real, controlled, human-graded evidence for a claim that's usually argued from anecdote in either direction ("AI makes programming knowledge obsolete" versus "AI-written code is always inferior"), and it's worth taking more seriously than either slogan precisely because it measured real skills against real, independently-scored output rather than assuming the answer.

See also

Scaffolding,Scaffolding, the Zone of Proximal Development, and Using GenAI Well makes the same case this page's data supports, from educational-psychology theory rather than an empirical study. LLM Orchestration, Context Engineering & Agentic AI covers the mechanics of getting good output from a model once you know what you want to ask for. The Team Project: Designing an AI-Assisted Solution is where these skills get applied to an actual piece of coursework.

References