Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Development

This paper introduces Vibe Code Bench, a novel benchmark featuring 100 web application specifications evaluated by autonomous browser agents, which reveals that even the best frontier models achieve only 58.0% accuracy on end-to-end development tasks and highlights self-testing and evaluator alignment as critical factors for success.

The Gist

Imagine you've hired a brilliant, super-fast apprentice to build you a house. You don't give them blueprints; you just say, "I want a cozy cottage with a big kitchen, a garden, and a front door that opens automatically."

In the past, we tested these AI apprentices by asking them to do tiny, isolated tasks: "Can you hammer this nail?" or "Can you mix this specific type of concrete?" They got really good at those small jobs.

But the real question is: Can they actually build the whole house, from the foundation to the roof, and make sure the lights turn on when you flip the switch?

This paper, "Vibe Code Bench," introduces a new way to test AI models on exactly that. Instead of asking them to fix a single broken window, the researchers gave 16 of the world's smartest AI models a challenge: Build a complete, working website from a simple text description.

Here is the breakdown of how they did it and what they found, using some everyday analogies.

1. The Test: "The Vibe Code Bench"

Think of this as a driving test for AI, but instead of driving a car, they are driving a construction crew.

• The Challenge: The AI was given 100 different "job descriptions" (like "build an app to track your habits" or "create a tool for a small business to manage payments").
• The Environment: The AI was dropped into a digital sandbox with a computer terminal, a web browser, and access to tools like databases and payment processors (like Stripe).
• The Goal: The AI had to write the code, set up the servers, and launch the website all by itself.
• The Grader: Once the AI finished, a robotic "inspector" (an autonomous browser agent) actually visited the website. It clicked buttons, tried to log in, made test purchases, and checked if the app actually worked. It didn't just read the code; it used the app.

2. The Results: "The Apprentice is Getting Better, But Still Needs a Supervisor"

The researchers tested 16 top-tier AI models. Here is the verdict:

• The Best Performer: The top model (GPT-5.3-Codex) succeeded in about 62% of the tasks.
• The Reality Check: This means that for nearly 4 out of 10 jobs, the AI either couldn't build the house, or the house collapsed the moment someone tried to walk through the front door.
• The Gap: While AI is amazing at writing a single paragraph of code, building a whole system that connects a database, a login screen, and a payment processor is still a massive, unsolved puzzle.

3. The Secret Sauce: "Self-Testing"

The paper discovered a fascinating habit that separates the good builders from the great ones.

• The "Edit-Only" Builders: Some models just wrote code, hit "submit," and hoped for the best. These models failed often.
• The "Self-Testing" Builders: The top-performing models acted like careful craftsmen. They would write a bit of code, open the browser, click around, check if it worked, find a bug, fix it, and repeat.
• The Analogy: It's the difference between a chef who just throws ingredients in a pot and serves it, versus a chef who tastes the soup, adds salt, tastes it again, and adjusts the heat. The paper found that the more an AI "tasted its own soup" (tested its own code), the better the final dish was.

4. The Human Factor: "Who is the Judge?"

One of the most interesting parts of the study was about how the apps were graded.

• The Problem: If you ask one AI to grade another AI's work, they might disagree. One might say, "The login button is blue, so it's perfect!" while another says, "The button is too small, it's a fail!"
• The Finding: The researchers found that who you choose to be the judge matters a lot. Some AI judges agreed with human experts 93% of the time, while others only agreed 32% of the time.
• The Lesson: You can't just use any AI to grade the work. You have to pick the right "inspector" to get a fair score.

5. The Big Picture: "From 'Can It Write?' to 'Can It Build?'"

For a long time, the question was, "Can AI write code?" The answer is a loud YES.

Now, the question has changed to: "Can AI build software?"

This paper shows that we are getting close, but we aren't there yet. The AI is like a very talented junior developer who needs a senior engineer to look over their shoulder, catch the big mistakes, and make sure the final product actually works for real people.

In short: AI is no longer just a spell-checker for code; it's becoming a builder. But until it can reliably build a whole house without the roof falling off, we still need human architects to keep the blueprints safe.

Technical Summary

Here is a detailed technical summary of the paper "Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Development."

1. Problem Statement

While Large Language Models (LLMs) have shown increasing capability in isolated code generation tasks (e.g., function synthesis, bug fixing), there is a significant gap in evaluating their ability to perform "zero-to-one" application development. Existing benchmarks like HumanEval or SWE-Bench focus on constrained tasks within existing codebases or single functions. They fail to measure the end-to-end process of building a complete, deployable web application from a natural language specification, which involves multi-file code generation, configuration, database integration, deployment, and user-facing workflow execution.

The authors argue that the true promise of "vibe coding" (AI-assisted coding) is enabling non-technical users to build full applications from scratch. Current benchmarks do not capture the complexity, failure modes, or reliability required for this specific use case.

2. Methodology: Vibe Code Bench (VCB)

The authors introduce Vibe Code Bench, a novel benchmark designed to evaluate the full lifecycle of web application generation.

A. Dataset Construction

• Scale: 100 realistic application specifications split into a public validation set (50 tasks) and a held-out test set (50 tasks).
• Domains: Tasks are categorized into three difficulty levels based on user intent:
  • Individual (24 apps): Personal tools (e.g., habit trackers), often without authentication.
  • Solo Founder (45 apps): Business MVPs, often requiring simple email auth.
  • Enterprise Tool (31 apps): Internal business programs, often requiring role hierarchies.
• Complexity: 28% of tasks require integration with external services (Stripe for payments, MailHog for email), introducing challenges like async webhooks and state consistency.
• Evaluation Workload: Each task is paired with 6–23 automated browser workflows, totaling 964 workflows and 10,131 substeps.

B. Generation Harness

• Environment: Models operate in a sandboxed, containerized environment based on a modified OpenHands framework.
• Capabilities: Agents have access to a terminal, a web browser, and specific service integrations (Supabase for backend/auth, Stripe, MailHog).
• Constraints:
  • Time Budget: 5 hours per application.
  • Tech Stack: Mandated to use React + Vite (frontend), Tailwind CSS, and Supabase (backend) to ensure consistent evaluation.
  • Deployment: Agents must produce a docker-compose.yml file and ensure the app runs via Docker.
• System Prompt: Includes strict instructions on testing, deployment, and avoiding mocks (e.g., real Stripe test-mode calls are required).

C. Automated Evaluation Pipeline

• Agent-Based Testing: Instead of unit tests or DOM assertions (which are brittle and implementation-specific), VCB uses an autonomous browser agent (Browser Use) to interact with the deployed application exactly as a human user would.
• Scoring Logic:
  • The agent executes predefined workflows (e.g., "Sign up," "Create post," "Pay subscription").
  • Each workflow consists of substeps.
  • A workflow passes if ≥90% of its substeps succeed.
  • An application's final score is the percentage of workflows that pass.
• Evaluator: The canonical evaluator is Claude Sonnet 4.5, though cross-evaluator studies were conducted.

3. Key Contributions

1. Novel Benchmark Dataset: The first benchmark to test LLMs on generating complete, deployable web apps from text specs with browser-based evaluation.
2. Realistic Generation Harness: A production-like environment supporting multi-hour sessions, Docker deployment, and third-party service integration.
3. Automated Evaluation Pipeline: A robust, implementation-agnostic evaluation method using browser agents to score success based on user-visible behavior.
4. Comprehensive Model Evaluation: Evaluation of 16 frontier models (OpenAI, Anthropic, Google, etc.) with detailed analysis of cost, latency, and error modes.
5. Evaluator Alignment Protocol: A study demonstrating that the choice of evaluator model significantly impacts results, establishing a protocol for human-model alignment.

4. Experimental Results

The study evaluated 16 models across the test split.

A. Performance Overview

• Top Performer: GPT-5.3-Codex achieved the highest accuracy at 61.8% (±4.71% SE).
• Runner-up: Claude Opus 4.6 followed closely at 57.6%.
• General Trend: Performance drops significantly for open-weight models and smaller models. For instance, Grok 4.1 and DeepSeek V3.2 scored below 6%.
• Discriminative Power: VCB is far more discriminative than SWE-Bench. The gap between top and mid-tier models on SWE-Bench is ~2.8%, whereas on VCB it is 42.7%.

B. Cost and Latency

• There is a positive correlation between cost/latency and accuracy, but with diminishing returns.
• Claude Opus 4.6 offers a strong efficiency trade-off, achieving near-top accuracy with significantly lower cost and latency than GPT-5.3-Codex.
• GLM-5 was an outlier with extremely high latency and cost but mediocre performance.

C. Difficulty and Integration

• Difficulty: Performance drops sharply from Easy (44% avg) to Hard (6% avg) tasks.
• Integrations: Apps requiring both Email and Stripe integrations saw a massive drop in performance (e.g., GPT-5.3-Codex dropped from 71% on no-integration apps to 29.6% on dual-integration apps).

D. Behavioral Analysis

• Self-Testing Correlation: A strong positive correlation (Pearson r = 0.72) was found between a model's use of the browser tool (self-testing) during generation and its final success rate. Models that iteratively tested and repaired their apps performed significantly better.
• Edit Volume: The number of file edits had almost no correlation with success (r = 0.09), suggesting how edits are used matters more than quantity.
• Error Modes: The most common failure was Missing Features (46.7%), followed by Authorization Issues (20.4%). Top models rarely failed to start (0% startup failure), whereas bottom models had a 14% startup failure rate.

E. Evaluator Alignment

• Human Alignment: Human reviewers agreed with each other 88.6–93.6% of the time.
• Model Alignment: Evaluator choice matters. Claude Sonnet 4.5 and 4.6 aligned well with humans (86%), while GPT-5.2 showed poor alignment (36%). This indicates that automated evaluation results are highly sensitive to the choice of the "judge" model.

5. Significance and Conclusion

• State of the Art: The results indicate that while AI can write code, reliable end-to-end application development remains a frontier challenge. Even the best model only passes ~62% of complex workflows.
• Shift in Focus: The primary driver of improvement is not incremental feature addition but reducing total failure rates (i.e., ensuring the app actually starts and runs).
• Future Directions: The paper highlights that "vibe coding" requires models to adopt self-testing behaviors and handle external service integrations robustly.
• Limitations: The benchmark currently focuses on a single tech stack (React/Supabase) and does not evaluate code quality, security, or visual design aesthetics, only functional correctness.

Conclusion: Vibe Code Bench provides a rigorous, standardized framework for measuring the "zero-to-one" capability of AI agents. It reveals that while progress is being made, building fully functional, deployable software from natural language without human intervention is not yet a solved problem. The study emphasizes that self-testing and evaluator selection are critical factors in both model performance and evaluation reliability.