💻 computer science

Project Prometheus: Bridging the Intent Gap in Agentic Program Repair via Reverse-Engineered Executable Specifications

The paper introduces \textsc{Prometheus}, a novel agentic framework that bridges the intent gap in automated program repair by reverse-engineering executable Gherkin specifications from failure reports and validating them through a Requirement Quality Assurance loop, achieving a 93.97% correct patch rate on the Defects4J benchmark.

Original authors: Yongchao Wang, Zhiqiu Huang

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yongchao Wang, Zhiqiu Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master architect who has hired a brilliant, super-fast construction crew (the AI agents) to fix a leak in your house.

The Problem: The "Berserker" Builder
In the past, you would tell the crew, "There's a leak in the kitchen, please fix it."
The crew, being incredibly smart but lacking common sense, would panic. They might tear down the entire kitchen wall, replace the plumbing with a fire hose, and install a swimming pool in the living room just to be sure the leak is gone. They fixed the symptom (the water) but destroyed the house (the code). This is what the paper calls the "Intent Gap." The builders know how to build, but they don't truly understand what you actually wanted.

The Solution: Prometheus
The authors of this paper, Yongchao Wang and Zhiqiu Huang, built a new system called Prometheus. Instead of letting the builders guess, Prometheus acts as a strict, detail-oriented project manager who speaks a very specific language.

Here is how Prometheus works, broken down into three simple steps:

1. The Detective (The Architect)

Before the builders touch a single brick, Prometheus sends in a Detective.

The Old Way: The detective would just say, "The wall is wet." (Vague!)
The Prometheus Way: The detective analyzes the leak and writes a strict rulebook (called a Gherkin Specification). It says: "If water hits the pipe, it must stop. If the pipe is dry, the wall must stay dry. Do not touch the living room."
The Analogy: This is like translating a messy complaint into a precise legal contract. The builders can no longer guess; they must follow the contract exactly.

2. The Quality Inspector (The Engineer)

This is the paper's secret sauce. Before the builders start working, the Inspector checks the rulebook.

The Inspector takes the rulebook and tests it against the broken house (to make sure the rule actually catches the leak).
Then, the Inspector tests it against the perfect house (the one the original owner fixed) to make sure the rulebook describes the right solution.
The Analogy: Imagine a chef tasting a recipe before cooking. If the recipe says "add salt" but the dish needs sugar, the Inspector catches it before the food is ruined. This prevents the builders from following a wrong instruction.

3. The Surgeon (The Fixer)

Now, the builders get to work. But they aren't the chaotic "Berserkers" anymore. Because they have a strict rulebook and a verified plan, they act like Surgeons.

They make tiny, precise cuts. They fix only the leak. They don't touch the living room.
The Result: The paper found that this method fixed 94% of the bugs, whereas the old "guessing" method only fixed about 76%. More importantly, for the really hard, complex bugs, Prometheus saved 74% of the cases that the old method gave up on.

Why This Matters (The "Aha!" Moment)

The paper discovered something surprising: The problem wasn't that the builders weren't smart enough; it was that the instructions were too vague.

Old Approach: "Fix the bug!" (Result: The builder hallucinates and breaks things).
New Approach: "Here is a step-by-step contract of exactly what the bug is and how it should behave. Now, fix it." (Result: The builder becomes a genius).

The "Digital Archaeology" Challenge

The paper admits one downside: The "Inspector" takes a lot of time and computing power. It's like hiring an archaeologist to dig through layers of old dirt just to make sure the blueprint is right before you build. But the authors argue this is worth it because it stops the builders from wasting time and money on massive, wrong fixes.

The Future

The authors believe the future of AI coding isn't about making the AI "smarter" or "bigger." It's about teaching the AI to ask better questions.

Instead of asking, "Can you fix this?"
We should ask, "Can you write the perfect rulebook for this fix, and then follow it?"

In a nutshell: Prometheus teaches AI to stop guessing and start following a verified, step-by-step contract. It turns a chaotic demolition crew into a precise surgical team, saving software from being "over-engineered" and broken.

1. Problem Statement: The "Intent Gap"

Despite the shift from neural machine translation to Agentic Automated Program Repair (APR) workflows, current systems suffer from a fundamental limitation known as the "Intent Gap."

The Issue: While Large Language Models (LLMs) are proficient at generating code, they struggle to understand the precise intent behind a bug fix.
Current Flaws: Existing approaches rely on vague natural language summaries or adversarial test cases. These are descriptive rather than prescriptive, leading to ambiguity.
Consequence: This ambiguity causes agents to exhibit "Berserker-style" behavior—making structurally invasive, over-engineered, or hallucinated modifications that fix the immediate symptom but violate system invariants or introduce regressions.
Core Hypothesis: The lack of Executable Intent (a specification that is both human-readable and machine-verifiable) is the primary bottleneck, not the reasoning capability of the models themselves.

2. Methodology: The Prometheus Framework

Prometheus is a multi-agent framework that fundamentally alters the repair workflow by prioritizing Specification Inference over direct code generation. It employs a Behavior-Driven Development (BDD) approach using Gherkin syntax (Given-When-Then) as the contract between intent and execution.

The framework operates in three distinct phases (Figure 1):

Phase 1: The Architect (Specification Inference)

Role: Reverse-engineers the missing requirement from runtime failure reports (stack traces, assertion errors).
Mechanism: Uses a "Structured Reasoning" strategy (Chain-of-Thought) to perform Root Cause Analysis before synthesizing a formal BDD scenario (.feature file).
Output: A deterministic, executable specification $S$ that defines the expected behavior, replacing vague natural language instructions.

Phase 2: The Engineer (Requirement Quality Assurance - RQA Loop)

Role: Validates the inferred specification $S$ to prevent "Hallucination of Intent."
Mechanism (Sandwich Verification):
1. Negative Verification: Executes $S$ against the buggy code ( $C_{buggy}$ ). It must fail.
2. Positive Verification: Executes $S$ against the ground-truth fixed code ( $C_{fixed}$ ). It must pass.
Significance: This loop acts as a "Proxy Oracle." Only specifications that satisfy both conditions are propagated to the next phase. In production, this role would be filled by human review of concise Gherkin scenarios rather than full code patches.

Phase 3: The Fixer (Enlightened Repair)

Role: Generates the actual code patch $P$ .
Mechanism: Operates in an "Enlightened" mode, conditioned strictly on the validated specification $S$ .
Constraint: In the study, the Fixer was restricted to a single suspicious file to isolate the impact of specification guidance from fault localization.
Outcome: The agent performs "Surgical Repairs"—minimal, precise edits that satisfy the semantic contract without invasive refactoring.

3. Key Contributions

Prometheus Framework: A novel multi-agent architecture that integrates BDD reverse engineering into the APR loop, shifting the paradigm from "fix this bug" to "reverse-engineer the missing requirement."
RQA Loop: A validation mechanism using ground-truth code as an oracle to ensure the quality of inferred specifications, effectively mitigating hallucination risks.
Intent-Behavior Mirroring Effect: A theoretical observation that the structural invasiveness of an agent's code is a direct mirror of the scope of its input requirement. Precise, atomic specifications lead to precise code; broad, vague requirements lead to "Berserker" behavior.
Empirical Evidence: Comprehensive evaluation on 680 defects from Defects4J v3.0.1, demonstrating that intent-driven repair outperforms strong baselines significantly.

4. Experimental Results

The framework was evaluated on 680 real-world Java defects across 16 projects.

Overall Performance:
- Total Correct Patch Rate: 93.97% (639/680 bugs).
- Rescue Rate: 74.4%. The framework successfully repaired 119 complex bugs that a strong "Blind" agent (Qwen-3.0-Coder without specifications) failed to resolve.
Comparison with SOTA:
- On the 160 "hard" bugs where the Blind agent failed, Prometheus rescued 119.
- This represents a 4.4× improvement over other SOTA agentic tools (TSAPR and RepairAgent) on these difficult cases.
Qualitative Insights:
- Curing Hallucination: Agents stopped inventing non-existent variables (e.g., in Chart-13) when guided by explicit constraints.
- Preventing Over-Simplification: Agents avoided lazy shortcuts that passed tests but violated invariants (e.g., Chart-7).
- Surpassing Ground Truth: In some cases (e.g., JxPath-6), the BDD-guided agent produced a more robust fix than the original human developer by covering edge cases (arrays) the human missed.
- Architectural Integrity: Agents respected dependency constraints (e.g., Mockito-5) rather than destructively removing code to pass tests.

5. Significance and Future Implications

Paradigm Shift: The paper argues that the future of APR lies not in training larger models for "blind" repair, but in aligning code with verified, Executable Specifications. The quality of the question (specification) determines the quality of the answer (patch).
Cost-Benefit Analysis: While the "Engineer" phase (RQA Loop) incurs high computational cost (58% of tokens) due to environment setup and execution, the "Architect" (reasoning) is surprisingly cheap (6.4%). The authors argue this is a high-yield investment compared to generating thousands of invalid patches.
Limitations & Future Work:
- Constraint: The current study restricted agents to single-file changes. Future work aims to introduce a "Detective" agent to unbind this constraint for multi-file refactoring.
- Digital Archaeology: Handling legacy build systems and dependencies remains a high-cost challenge.
- Vision: The ultimate goal is a self-evolving "immune system" for software repositories, where every repair synthesizes a persistent "Skill Directory" (BDD specs + validation logic) to prevent future regressions.

Conclusion: Prometheus demonstrates that by bridging the Intent Gap through reverse-engineered, executable specifications, agentic systems can achieve near-perfect repair rates on complex defects, transforming from aggressive, error-prone coders into precise, surgical instruments.