🤖 AI

ComBench: A Repo-level Real-world Benchmark for Compilation Error Repair

This paper introduces ComBench, the first repository-level real-world benchmark for C/C++ compilation error repair, which leverages a novel automated framework to mine and verify GitHub CI failures, revealing significant gaps between the syntactic and semantic correctness of current LLM-based repair techniques.

Original authors: Jia Li, Zeyang Zhuang, Zhuangbin Chen, Yuxin Su, Wei Meng, Michael R. Lyu

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jia Li, Zeyang Zhuang, Zhuangbin Chen, Yuxin Su, Wei Meng, Michael R. Lyu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to cook a massive, complex banquet for thousands of people. You have a recipe (the code), but every time you try to start the stove (compile the code), the kitchen explodes with smoke and fire alarms (compilation errors).

In the world of software, these "kitchen explosions" are Compilation Errors. They stop the entire process, forcing developers to stop cooking and spend hours figuring out why the stove won't light.

For a long time, researchers have been trying to build AI Chefs (Large Language Models) that can automatically fix these errors. But there was a big problem: the tests they used to train and check these AI chefs were like practice drills in a tiny, empty kitchen.

The Problem: The "Toy Kitchen" Benchmark

Imagine you are training a chef to fix a broken oven in a 50-story skyscraper. But instead of sending them to the skyscraper, you give them a broken toaster in a toy kitchen.

The Old Way: Researchers gave AI models tiny, isolated snippets of code (like a single line of a recipe) and asked them to fix it.
The Reality: Real software is a giant skyscraper. An error in the basement (one file) might be caused by a broken pipe in the penthouse (another file). The old tests ignored the whole building, so the AI chefs learned to fix toasters but failed miserably when faced with skyscrapers.

The Solution: ComBench (The "Real Skyscraper" Simulator)

The authors of this paper built ComBench. Think of this as a massive, realistic simulation of a real-world construction site where the AI chefs can practice fixing actual, messy, real-world errors.

Here is how they built it, using simple analogies:

1. Mining the "Black Box" (Data Collection)

Instead of making up fake errors, the team went to GitHub (a giant library of software projects) and looked at the security camera footage of the construction sites (CI/CD logs).

They found thousands of times when real developers tried to build software and it failed.
They didn't just grab the error message; they grabbed the entire context: the blueprints, the tools used, and the specific weather conditions (the server environment) at that exact moment.

2. Finding the "Magic Fix" (Ground Truth)

When a building collapses, you need to know exactly how the engineers fixed it.

The team used a sophisticated "time-travel" algorithm to find the exact moment a developer fixed a broken build.
They didn't just take the whole commit (which might include adding a new window or painting a wall); they used a "surgical scalpel" to extract only the tiny patch that stopped the fire. This ensures the AI learns the exact fix, not just a random change.

3. Recreating the "Ghost Kitchen" (Reproducibility)

This is the hardest part. Sometimes, a kitchen only explodes because of a specific brand of gas or a weird setting on the stove.

The team built a time-machine Docker container. They recreated the exact virtual environment where the error happened 6 months ago.
They verified that:
1. The error happens again (Fail).
2. The AI applies the fix.
3. The building stands tall and works perfectly (Pass).

What They Discovered (The Taste Test)

Once they had this realistic simulator, they tested 12 different AI chefs (like GPT-5, Claude, and others) to see how good they were at fixing real-world errors.

Here are the surprising results, translated into everyday terms:

The "Looks Good, Tastes Bad" Gap:
The AI chefs were great at making the smoke stop. 73% of the time, they could make the code compile (the stove didn't explode).
- But, only 41% of the time did the food actually taste right (Semantic Correctness).
- Analogy: The AI might fix the broken stove by unplugging the refrigerator. The kitchen is quiet now (no error), but you have no food to cook. The code works, but the program is broken.
Size Matters, But Not Forever:
Bigger AI models generally did better, but after a certain point, making them bigger didn't help much. They hit a "ceiling" because the problem wasn't just about knowing more facts; it was about understanding how the whole building connects.
The "Agent" vs. The "Solo Chef":
- Solo Chef (Direct Repair): You give the AI the error and the file. It guesses the fix.
- Agent Chef (Iterative): You give the AI a toolbox. It can walk around the building, check other rooms, ask questions, and try different tools until it finds the fix.
- Result: The "Agent" approach was much better for the smartest models. It's like giving a detective a magnifying glass and a map, rather than just asking them to guess the culprit.
The "Hard Cases":
Even the smartest AI (GPT-5) couldn't fix about 30% of the errors. These were usually "cross-file" mysteries where the solution required understanding deep, hidden rules of the specific project that the AI didn't know.

Why This Matters

Before ComBench, we were testing AI on toy problems and thinking they were ready for the real world. This paper says, "Stop! Let's test them in the real skyscraper."

It shows us that while AI is getting very good at fixing syntax (the grammar of code), it still struggles with semantics (the meaning and logic of the code). It's like an AI that can fix a typo in a novel but doesn't understand the plot.

In short: ComBench is the new, realistic driving test for AI code-fixers. It proves that while our AI drivers are getting better at steering, they still need to learn how to navigate the complex traffic of real-world software projects.

1. Problem Statement

Automated Compilation Error Repair (ACER) aims to fix code that fails to compile, a frequent bottleneck in software development. While Large Language Models (LLMs) have improved ACER capabilities, their real-world performance remains poorly evaluated due to significant limitations in existing benchmarks:

Decontextualized Data: Existing benchmarks (e.g., DeepFix, TRACER) use single-file snippets (often <20 lines) from student assignments, ignoring cross-file dependencies and repository-level context.
Lack of Authenticity: Data sources are narrow and synthetic, failing to represent the diverse, complex errors found in large-scale industrial projects.
Biased Task Modeling: Current benchmarks frame repair as a local sequence-to-sequence problem, overlooking the need for global fault localization and context collection in real repositories.
Reproducibility Issues: Existing datasets often lack the ability to reproduce the exact build environments (CI/CD configurations) where errors occurred.

2. Methodology: ComBench Construction

The authors propose ComBench, the first repository-level, reproducible real-world benchmark for C/C++ compilation error repair. It is constructed via a novel, automated framework involving three main stages:

A. Data Collection & Error Extraction

Source: Mining GitHub Actions CI logs from four large-scale, actively maintained C/C++ open-source projects: OpenSSL, Bitcoin, RocksDB, and LLVM.
Dual-Filtering Algorithm: To handle noisy CI logs, the framework uses a dual-filtering approach to identify compilation-related failures. It excludes irrelevant entries (e.g., test results, warnings) and validates lines against specific error patterns (e.g., : error:, undefined reference).
Context Reconstruction: A bidirectional context collector extracts multi-line diagnostic information, including backward include traces and forward diagnostic notes, to form structured error records.

B. Ground-Truth Patch Identification

Multi-Source Search: The system searches for the causal repair commit ( $c_r$ ) across the same branch, dependent branches, and the main branch using ancestry-path analysis.
Dependency-Based Minimization: To isolate the minimal repair patch ( $\Delta_r$ ), the framework analyzes compilation dependencies. It filters commits to ensure the modified files intersect with the dependency closure of the error-producing file, removing unrelated features or refactoring.
Ranking: Candidates are ranked by source type, temporal proximity, and change size.

C. Automated Reproduction Environment

Environment Emulation: To address the "opacity" of CI runners, the framework maps proprietary runner labels to generic Docker images (e.g., catthehacker/ubuntu:act-latest).
Job Configuration Inference: An intelligent algorithm infers specific job parameters (e.g., compiler versions, matrix variables) from human-readable job names in logs to reconstruct the exact build environment.
Fail-Fix-Pass Verification: Every instance undergoes end-to-end validation: the original failure is reproduced, the ground-truth patch is applied, and the build must succeed. Only instances passing this check are retained.

3. Key Contributions

ComBench Benchmark: A repository-level dataset containing 200 execution-verified C/C++ compilation errors from four major projects. It covers seven error categories (Declaration, Type, Member, Syntax, Function, Template, Semantic) and includes full project context, not just snippets.
Robust Construction Framework: A novel pipeline that solves three critical challenges: extracting structured errors from noisy CI logs, inferring causal ground-truth patches from complex Git histories, and reproducing ephemeral CI environments using Docker and act.
Comprehensive Evaluation: A rigorous evaluation of 12 modern LLMs (including GPT-5, Claude 3.7/4, Gemini 2.5 Pro, and various Qwen/DeepSeek models) under two settings:
- Perfect Fault Localization (PFL): Testing pure patch generation with full context.
- Agent-Based Repair: Testing autonomous reasoning, localization, and iterative refinement using the OpenHands framework.

4. Key Results & Findings

The evaluation reveals a significant gap between syntactic correctness and semantic correctness:

The "Compile-but-Incorrect" Gap: Even top-tier models struggle with semantic alignment. GPT-5 achieved a 73% Compile Success (CS) rate but only 41% Semantic Correctness (SC). This indicates that models often fix the syntax while breaking the program's intended logic or repository conventions.
Model Scaling vs. Reasoning: Performance generally scales with model size (e.g., Qwen3-235B outperforms Qwen3-4B), but mid-sized models plateau. Scaling alone does not close the CS-SC gap without better repository-level reasoning.
Agent vs. Direct Repair: Agent-based approaches (using OpenHands) generally outperform direct repair for strong models (e.g., Claude-Sonnet-4) by leveraging iterative context gathering. However, for smaller models, the agent paradigm can degrade performance due to noise and tool-use costs.
Error-Type Specialization: Different models show distinct strengths. For instance, Gemini-2.5-Pro excels at Type Errors, while Claude-Sonnet-4 performs better on Member Errors.
Failure Modes:
- Repository Convention Violations: Models often use standard library headers (e.g., <span>) instead of project-specific ones (e.g., <span.h>), violating architectural constraints.
- Shallow Substitution: Models replace undefined identifiers with arbitrary, nearby defined ones to satisfy the compiler, ignoring domain-specific logic.

5. Significance

Bridging the Gap: ComBench provides the first realistic platform to evaluate ACER tools against the complexities of modern software development, moving beyond toy examples to industrial-scale, cross-file scenarios.
New Evaluation Metrics: By introducing Semantic Correctness and Exact Match alongside Compile Success, the paper highlights that "compiling" is not synonymous with "correct," urging the community to focus on semantic preservation.
Future Direction: The results suggest that future ACER research must prioritize global fault localization, repository-level context understanding, and agent-based iterative reasoning rather than just local code generation. The benchmark is designed to be continuously evolving, ensuring it remains relevant as development practices change.