💻 computer science

From Prompting to Verification: How Experience Shapes Vibe Coding Practices

This study of 162 users across non-coder, novice, and professional groups reveals that while experience levels similarly shape perceptions of AI-generated code quality, they significantly diverge in motivations and verification practices, creating a "perception-action gap" where access to software creation is democratized without a corresponding distribution of the expertise needed to evaluate it.

Original authors: Ahmed Fawzy, Amjed Tahir, Kelly Blincoe

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmed Fawzy, Amjed Tahir, Kelly Blincoe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: "Vibe Coding"

Imagine you want to build a wooden chair.

Traditional Coding: You are a carpenter. You measure the wood, cut the joints, sand the edges, and assemble it yourself. You know exactly how every screw fits.
Vibe Coding: You are a director. You tell a magical robot, "Build me a sturdy chair," and the robot instantly builds one. You don't know how the joints are cut, but you sit on it. If it wobbles, you don't fix the wood; you just tell the robot, "Make it sturdier," and hope it fixes itself.

This paper studies people who use this "Vibe Coding" method (using AI to write code via natural language) and asks: Does it matter if you are a professional carpenter, a beginner, or someone who has never held a hammer?

The Study: Three Groups at the Table

The researchers surveyed 162 people and split them into three groups:

Non-Developers: People with no formal coding training.
Novices: People who have taken a class or built small projects but aren't pros yet.
Professionals: Experienced software engineers who do this for a living.

They asked these groups four main questions:

Why do you do it? (Motivation)
How does it feel? (Experience)
Do you trust the result? (Perceived Quality)
What do you do when it breaks? (Quality Assurance)

The Findings: What's the Same, What's Different?

1. The "Feel" and "Trust" are the Same (The Shared View)

Surprisingly, all three groups felt the same way about the experience and the quality of the AI's work.

The Analogy: Imagine everyone at the table is eating a meal cooked by a famous chef. The non-coders, the students, and the pros all agree: "This food is delicious and fast, but sometimes it has a weird texture, and you wouldn't serve it at a fancy wedding."
The Result: Everyone knows AI code is great for quick prototypes but might be "flaky" or "buggy" for serious work. They all feel the same mix of excitement and caution. Experience does not change how they see the AI.

2. The "Why" and "How" are Different (The Different Approaches)

While they see the AI the same way, they use it very differently.

Non-Developers (The Magic Wand Users): They are mostly motivated by access. They say, "I couldn't build this without the AI." They tend to let the AI do almost everything with very little input. It's like handing a blank canvas to the robot and saying, "Make art."
Novices (The Learners): They are motivated by learning. They use the AI to experiment and see how things work. They are like students watching a master chef cook, trying to understand the recipe.
Professionals (The Conductors): They use the AI for work. They don't just say "make a chair"; they give detailed instructions, context, and constraints. They treat the AI like a junior assistant they need to manage carefully.

3. The Big Gap: When Things Go Wrong (The "Perception-Action Gap")

This is the most important finding. Even though everyone knows the AI makes mistakes, only the experts know how to fix them.

The Analogy: Imagine the robot builds a chair, but one leg is wobbly.
- The Non-Developer: Says, "Oh no, it's wobbly!" and immediately tells the robot, "Make the leg longer!" They keep asking the robot to fix it until it stops wobbling, without ever looking at the wood themselves.
- The Novice: Tries to fix it but gets confused. They might give up or keep asking the robot for help.
- The Professional: Says, "Oh no, it's wobbly!" They pick up a wrench, look at the joint, find the loose screw, and tighten it themselves. They check the work before they sit down.

The Paper's Conclusion: There is a "Perception-Action Gap."

Perception: Everyone knows the AI can be wrong.
Action: Only experienced people have the skills to verify and debug the code when it is wrong.

The "Partial Democracy" Metaphor

The authors call this "Partial Democratization."

Democratized: The AI has lowered the barrier to making software. Anyone can now build a digital "chair."
Not Democratized: The ability to inspect and guarantee the quality of that chair is still locked behind a door that only experienced people have the key to.

We are giving everyone a magic wand to create, but we haven't yet given everyone the training manual to know if the creation is safe to use.

What Do People Want?

When asked what would make them trust the AI more, everyone (from non-coders to pros) wanted the same things:

Reliability: "Just make sure it works."
Transparency: "Tell me why you built it this way."
Safety: "Don't give me code that might break later."

Summary

The paper argues that while AI tools have made it easy for anyone to create software, they haven't made it easy for anyone to verify it.

Beginners rely on the AI to fix its own mistakes (re-prompting).
Experts check the work themselves (debugging).
Everyone knows the AI isn't perfect, but only the experts know how to fix it when it fails.

The risk isn't that people don't know AI makes mistakes; the risk is that many people don't have the skills to fix those mistakes when they happen.

Technical Summary: From Prompting to Verification: How Experience Shapes Vibe Coding Practices

Problem Statement
AI code generation tools, powered by Large Language Models (LLMs), have expanded software creation beyond professional developers to include non-developers and novices. This shift has given rise to "vibe coding," a practice where users generate software via natural-language prompts, evaluate outputs primarily through execution, and iterate via reprompting rather than systematic debugging. While prior research has examined AI-assisted programming within specific groups (typically professionals or students) or studied non-experts in isolation, there is a lack of systematic comparison across experience levels. It remains unclear how motivations, interaction styles, perceived code quality, and quality assurance (QA) practices differ between non-developers, novice developers, and professional developers. Crucially, it is unknown whether the widespread awareness of AI-generated code limitations translates into effective corrective action across all experience levels, or if a gap exists between perception and behavior.

Methodology
The authors conducted a quantitative cross-sectional survey of 162 participants, evenly distributed across three experience groups: non-developers (no formal training but use AI tools), novice developers (limited experience via tutorials or coursework), and professional developers (formal or professional coding experience).

Instrument: The questionnaire was grounded in a prior grey literature review that identified four behavioral dimensions of vibe coding: motivations, experiences, perceived code quality, and QA practices. These dimensions were operationalized as Likert-scale constructs (5-point scale) and supplemented with open-ended questions.
Data Collection: Participants were recruited via Prolific. Data quality was rigorously enforced through completion filtering, time-based cognitive plausibility checks (excluding responses under 4 minutes), and response-pattern diagnostics (detecting straight-lining).
Analysis: The study employed non-parametric statistical methods suitable for ordinal Likert data. Differences between the three independent groups were assessed using the Kruskal–Wallis $H$ test, with Benjamini–Hochberg correction for multiple comparisons and Dunn's post-hoc tests for pairwise comparisons. Effect sizes were calculated using epsilon-squared ( $\varepsilon^2$ ). Additionally, ordinal logistic regression was used to determine if QA differences were driven by experience group membership or underlying usage dimensions (e.g., hours of independent coding, adoption duration).

Key Results
The study reveals that experience selectively shapes vibe coding practices, creating a distinct "perception–action gap."

Shared Perceptions (RQ2 & RQ3): There were no statistically significant differences across experience levels regarding subjective experiences (e.g., flow, frustration, hallucinations) or perceptions of code quality. All groups recognized similar strengths (rapid prototyping, clean structure) and limitations (fragility, hidden bugs, unsuitability for production). This indicates a broadly shared general awareness of the risks and benefits of AI-generated code.
Divergent Motivations and Styles (RQ1 & Interaction Style):
- Motivations: Non-developers were most motivated by accessibility and empowerment (building software they otherwise could not). Novices were driven by learning and experimentation. Professionals were more goal-oriented, often using vibe coding in work-related contexts.
- Interaction Styles: Non-developers exhibited higher reliance on "AI-led generation" (delegating code creation with minimal intervention). Professionals engaged in more "interactive dialogue" (iterative prompting and refining) and provided "richer context."
Divergent QA Practices (RQ4): The most significant divergence occurred in verification behaviors.
- Debugging vs. Reprompting: Non-developers relied significantly more on reprompting the AI to fix errors rather than manual debugging. They also reported higher levels of "QA breakdown" (giving up because they could not understand the code).
- Verification Frequency: Approximately 45% of professionals reported always checking AI-generated code before use, whereas non-developers were the only group to report "never" checking code.
- Regression Findings: The regression analysis indicated that these QA differences were not merely a function of group category but were driven by accumulated coding practice and tool exposure (specifically, hours of independent non-vibe coding).

Key Contributions

Empirical Characterization of Vibe Coding: The study provides the first systematic quantitative comparison of vibe coding practices across non-developers, novices, and professionals, moving beyond anecdotal evidence or single-group studies.
Identification of the Perception–Action Gap: The paper identifies a critical decoupling where general awareness of AI code risks is distributed across all experience levels, but the capacity to evaluate, debug, and verify those risks remains experience-dependent.
Refinement of "Partial Democratisation": The authors argue that vibe coding partially democratizes software creation by lowering barriers to production without equally distributing the expertise required for evaluation. The barriers have shifted from writing code to decomposing problems, specifying intent, and validating outputs.
Usage-Driven Behavior: The study demonstrates that QA behaviors are better predicted by specific dimensions of experience (coding practice and tool exposure) than by broad categorical labels alone.

Significance and Claims
The paper claims that the risks of vibe coding depend not only on the capabilities of AI tools but significantly on the user's ability to act on their awareness of those tools' limitations. The authors interpret their findings as evidence of a perception–action gap: while users across the spectrum recognize that AI-generated code can be flawed, less experienced users lack the specific skills to translate that recognition into systematic verification.

Consequently, the authors suggest that:

Tool Design: AI code generation tools should provide stronger scaffolding for less experienced users (e.g., proactive error surfacing, explanatory support, uncertainty signalling) while remaining unobtrusive for experts.
Education: The evaluation of AI-generated code must be treated as a distinct skill separate from prompting, requiring instruction in testing, edge-case identification, and code reasoning.
Future Research: The "perception–action gap" is presented as a hypothesis for future behavioral studies to test, as current findings rely on self-reported data rather than observed behavior.

The paper concludes that as vibe coding broadens participation in software creation, ensuring that the ability to evaluate produced code keeps pace with the ability to produce it is central to safe and effective adoption.