💬 NLP

ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents

This paper introduces Oracle-SWE, a unified method to isolate and quantify the individual impact of various contextual information signals on software engineering agent performance, aiming to guide research prioritization for autonomous coding systems.

Original authors: Kenan Li, Qirui Jin, Liao Zhu, Xiaosong Huang, Yijia Wu, Yikai Zhang, Xin Zhang, Zijian Jin, Yufan Huang, Elsie Nallipogu, Chaoyun Zhang, Yu Kang, Saravan Rajmohan, Qingwei Lin, Wenke Lee, Dongmei Zha

Published 2026-04-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kenan Li, Qirui Jin, Liao Zhu, Xiaosong Huang, Yijia Wu, Yikai Zhang, Xin Zhang, Zijian Jin, Yufan Huang, Elsie Nallipogu, Chaoyun Zhang, Yu Kang, Saravan Rajmohan, Qingwei Lin, Wenke Lee, Dongmei Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master detective (the AI Agent) trying to solve a complex crime in a massive, messy city (the Software Repository). Your job is to find a specific clue, fix a broken mechanism, and prove the city is safe again.

In the past, researchers have built better detectives and given them better tools. But they didn't really know which specific piece of information was the most important for solving the case. Was it a map of the crime scene? A list of witnesses? A photo of the suspect?

This paper, ORACLE-SWE, is like a "Super-Intelligence Test" for these AI detectives. The researchers asked: "If we magically gave the detective the perfect answer for just one specific clue, how much better would they get?"

Here is the breakdown of their experiment using simple analogies:

1. The Five "Magic Clues" (The Signals)

The researchers identified five types of information that help an AI fix code. They created "Oracle" versions of these—meaning perfect, 100% accurate versions that no real human or AI could actually find in the real world.

The Reproduction Test (The "Smoking Gun"):
- Analogy: Instead of just saying "The car won't start," you give the detective a video of the car failing to start, exactly when and how it happens.
- Result: This was the most powerful clue. When the AI saw the exact failure, it solved the problem much faster. It turns out, AI agents often get confused by vague descriptions ("The car is broken"), but they are great at fixing things when they can see the exact error.
The Execution Context (The "Crime Scene Map"):
- Analogy: A detailed map showing exactly which streets the suspect drove through before the crash, including traffic lights and potholes.
- Result: Very helpful, but only if the map is accurate. If the map is just a guess, it doesn't help much.
The Edit Location (The "Target on the Wall"):
- Analogy: A red arrow pointing exactly to the broken engine part.
- Result: Helpful, but not as magical as the "Smoking Gun." If you know where to fix it but not what is wrong, you might still guess the wrong fix.
API Usage (The "Instruction Manual"):
- Analogy: A manual explaining how to use the specific wrench needed for the job.
- Result: Surprisingly, this wasn't as critical. Why? Because modern AI already knows how to use common tools (like print or list in Python). It only helps if the tool is very obscure.
The Regression Test (The "Safety Net"):
- Analogy: A checklist to make sure you didn't break the radio while fixing the engine.
- Result: The least helpful for finding the solution. It's great for checking the work, but it doesn't tell the detective how to solve the mystery in the first place.

2. The Experiment: The "Perfect World" vs. "Real World"

The researchers ran two types of tests:

Test A: The Magic Box (Oracle Ablation)
They gave the AI the perfect clues one by one.

Finding: Giving the AI the Reproduction Test (the exact error video) boosted success rates the most. It was the "Golden Ticket."
The "Upper Bound": When they gave the AI all five perfect clues at once, the AI solved 97-100% of the problems. This tells us: "If we could just give AI perfect information, it would be almost a god at coding."

Test B: The Realistic Simulation (Validation)
In the real world, we can't give AI perfect clues. We have to ask a "Stronger AI" to find the clues for a "Weaker AI."

Finding: Even when the clues weren't perfect, the strategy worked! The "Stronger AI" found the "Reproduction Test" for the "Weaker AI," and the Weaker AI solved the problem much better than it could alone.
The Takeaway: This proves that if we build systems where one AI helps another find the "Smoking Gun," we can solve hard problems without needing a super-computer for every single task.

3. The Big Lesson for the Future

The paper concludes with a clear roadmap for the future of AI coding:

Focus on the "Smoking Gun": The biggest bottleneck for AI isn't knowing how to write code; it's understanding exactly what is broken. If we can build better tools to automatically generate perfect "Reproduction Tests" (tests that show the error clearly), AI agents will become significantly smarter.
Don't obsess over the "Instruction Manual": We don't need to spend as much energy teaching AI every single library function, because they already know the basics.
Teamwork works: A system where a "Senior AI" helps a "Junior AI" find the right clues is a winning strategy.

In summary:
Imagine you are trying to fix a leaky faucet.

Old Way: You guess where the leak is and try random wrenches.
New Way (ORACLE-SWE): You realize that if someone just handed you a video of the water spraying (Reproduction Test) and a highlighted map of the pipes (Edit Location), you could fix it instantly.
The Paper's Advice: Stop trying to make the wrench smarter; start making the video of the leak clearer. That is the key to unlocking the next generation of AI software engineers.

1. Problem Statement

Recent advances in Large Language Model (LM) agents have improved automated software engineering (SWE), yet their performance on repository-level tasks (e.g., SWE-bench) remains limited. Prior research has identified five critical contextual information signals that agents implicitly rely on:

Reproduction Test: Tests that fail on the buggy code and pass on the fixed code.
Regression Test: Existing tests that must continue to pass after the fix.
Edit Location: The specific files and line numbers where code modifications are required.
Execution Context: The runtime environment, including error stack traces and surrounding code.
API Usage: Definitions and usage patterns of internal or external functions required for the fix.

The Gap: While these signals are known to be important, their individual contribution to task success is not well quantified. Furthermore, it is unknown what the theoretical upper bound of performance would be if an agent had perfect access to these signals (Oracle Information). Understanding this is crucial for prioritizing future research (e.g., should we focus on better test generation or better code localization?).

2. Methodology

The authors introduce Oracle-SWE, a unified framework to isolate, extract, and quantify the impact of these five signals.

A. Oracle Signal Extraction

The authors constructed "Oracle" versions of the five signals from benchmark datasets (SWE-bench-Verified, Live, and Pro) and their corresponding repositories:

Edit Location: Extracted directly from the ground-truth patch (gold patch), providing exact file paths, function names, and line ranges.
Execution Context: Derived from error stack traces generated by running reproduction tests on the buggy code. For cases where native stack traces are unavailable (e.g., assertion errors in tests or Go build failures), a custom stack-trace collector was inserted at the oracle edit locations to capture the deepest reachable call stack.
API Usage: Identified all function calls modified or added in the gold patch. Definitions were resolved via execution coverage (Python) or static type checking (Go).
Reproduction & Regression Tests: Extracted the exact test commands, test names, and source code for "Fail-to-Pass" and "Pass-to-Pass" test cases from the benchmark specifications.

B. Experimental Setup

Base Agent: The authors used SWE-agent, a minimal agent with only two tools (bash commands and a string-replace editor), to avoid confounding factors from complex architectures.
Ablation Study (Upper Bound Analysis): The team injected ground-truth signals into the agent's initial prompt. They tested:
- Single Factor Injection: Measuring success rates when only one signal is provided.
- Accumulative Injection: Adding signals incrementally to observe interactions and the theoretical ceiling when all five are known.
Validation Experiment (Realistic Simulation): To address the unrealistic nature of "Oracle" inputs, a two-stage pipeline was designed:
- Stage 1 (Strong Model): A stronger LM (e.g., GPT-5 or Claude-4.6) attempts to extract the five signals from the repository.
- Stage 2 (Weak Model): A weaker LM (e.g., GPT-4o) attempts to solve the task using the extracted signals.
- This validates whether the signals are valuable even when they must be inferred rather than provided perfectly.

3. Key Contributions

Formal Definition & Extraction: The paper formally defines the five critical information factors in SWE and proposes a generalized method to extract their ground truths from multiple SWE benchmarks.
Quantification of Upper Bounds: It provides the first empirical estimation of the potential contribution (upper bound) of each signal to overall task success, guiding research priorities.
Agent-Based Validation: It validates these oracle insights through a realistic two-stage extraction-and-resolution pipeline, demonstrating that the findings hold even when signals are not perfectly provided.

4. Key Results

A. Contribution Ranking (Upper Bound)

When provided with perfect oracle information, the contribution of signals to success rate follows a consistent hierarchy across different benchmarks and models:

Reproduction Test: The most critical signal. It provides precise failure conditions, resolving ambiguity in natural language issue descriptions.
Execution Context: Highly effective when native error stack traces are available (approx. 25% of instances). Custom collectors are less effective as they often lack specific failure details.
Edit Location: Significant, but less than reproduction tests.
API Usage: Moderate contribution.
Regression Test: The least impactful for solving the bug (though essential for preventing regressions).

Note: The order shifts slightly for SWE-bench-Verified (Edit Location > API Usage > Context) compared to Live/Pro (Context > API Usage > Edit Location), likely due to dataset complexity and model training data leakage.

B. Interaction and Accumulation

Non-Additivity: The benefits of combining signals are not strictly additive. For example, Edit Location becomes significantly more effective when paired with Reproduction Tests (the test output guides how to edit the identified location).
Theoretical Ceiling: When all five factors are combined, the success rate for SWE-bench tasks reaches 97–100% across various model-benchmark pairs. This suggests the defined factor set is nearly complete for solving these tasks.
Cost vs. Performance: Providing more signals generally reduces the number of exploration steps but increases API costs due to longer context windows (e.g., long test logs).

C. Validation Findings

The two-stage validation (Strong Model extracts $\to$ Weak Model solves) confirmed the ablation study results.
Reproduction Tests extracted by a strong model significantly boosted the weak model's performance, often surpassing the weak model's baseline and approaching the strong model's baseline.
Execution Context showed a performance drop in the validation phase compared to the oracle phase, highlighting the practical difficulty of extracting accurate stack traces in real-world scenarios.

5. Significance and Implications

Research Prioritization: The study strongly suggests that Reproduction Test Generation is the highest-leverage area for improving SWE agents. Agents struggle most with ambiguous issue descriptions; high-quality, executable tests that capture corner cases are the most effective guide.
Contextual Understanding: While code graphs and execution context are valuable, their utility is heavily dependent on the availability of native error stack traces. Research should focus on improving the ability to trigger and capture these traces.
Synergy over Isolation: Improving a single factor in isolation yields diminishing returns. The most effective strategy is joint optimization (e.g., improving localization and test generation simultaneously).
Dataset Analysis: The Oracle-SWE methodology serves as a general tool for analyzing SWE datasets, revealing that current benchmarks vary in difficulty and that "perfect" information can solve almost all evaluated tasks, implying that current agent failures are primarily due to information retrieval and synthesis gaps rather than reasoning limitations.

In conclusion, ORACLE-SWE demonstrates that the primary bottleneck for current SWE agents is not a lack of reasoning capability, but a lack of precise, executable failure signals (Reproduction Tests) and accurate localization. Future systems should prioritize generating high-fidelity reproduction tests and leveraging them to guide code localization and editing.