Technical Summary: From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level

Problem Statement

As Large Language Models (LLMs) evolve into autonomous software agents, the core competency required has shifted from performing localized execution reasoning within isolated code snippets to maintaining logical consistency across extensive, interdependent file systems. The authors define this capability as Repository-Level Reasoning.

Current evaluation methods suffer from a critical gap:

1. Isolated Benchmarks: Datasets like CRUXEval and REval operate in "laboratory settings" using isolated snippets, lacking the cross-file dependencies and complex architectural contexts of real-world software.
2. Black-Box Evaluations: Frameworks like SWE-bench evaluate agents in wild repository environments but function as black boxes, checking only whether a problem is solved without providing insight into why an agent fails.

This lack of granular diagnosis obscures the specific cognitive bottlenecks preventing agents from scaling to real-world software engineering tasks.

Methodology: RepoReason

To address these gaps, the authors introduce RepoReason, a white-box diagnostic benchmark centered on Abductive Assertion Verification. The construction pipeline is designed as a five-phase defense system to ensure tasks are logically authentic, resistant to data contamination, and free from superficiality.

1. Design Philosophy: Assertions as Semantic Anchors

Unlike traditional Input/Output prediction or raw trace querying (which allow for shortcut learning via static analysis), RepoReason uses unit test assertions as semantic anchors.

• Abductive Reasoning: The task is framed as "Given the complex execution history, what is the current system state?" The model must deduce a deterministic value to satisfy a masked assertion (e.g., assert len(cache) == <mask>).
• Decoupling: This approach separates core logical reasoning from syntactic noise (e.g., formatting errors, API hallucinations), ensuring failures reflect genuine reasoning deficits.

2. Execution-Driven Mutation Framework (Anti-Memorization)

To prevent models from relying on pre-training data (rote memorization) or spurious correlations, the authors implement an Execution-Driven Mutation mechanism:

• Semantic Oracle: The code execution environment acts as a Semantic Oracle.
• Mutation: A teacher LLM performs unified visual and logical mutations (renaming variables, changing constants, restructuring code) while strictly preserving the original API call sequence to maintain invariant reasoning complexity.
• Ground Truth Regeneration: The mutated code is executed to capture actual runtime values. These values are used to reconstruct assertions that match the original codebase's style (e.g., preserving is vs. == usage).
• Validation: A strict gate ensures executability, assertion validity, and API preservation.

3. Structural Filtering and Deterministic Value Protocol

• Trace-Based Filtering: To eliminate logical superficiality, the system captures runtime traces using sys.settrace(). A weighted scoring system prioritizes computational intensity (call count), execution depth (stack depth), and cross-module dependencies. Only high-complexity tasks (Score ≥ 0.30) are retained.
• Deterministic Value Protocol: To ensure reproducibility, the protocol filters out non-deterministic values (e.g., timestamps, random memory addresses) and enforces that masked values represent concrete runtime states (literals, global constants, or parameter-resolvable constructors).

4. Cognitive Diagnostic Framework

The benchmark quantifies reasoning complexity using three orthogonal metrics derived from dynamic program slicing:

1. Reading Load (ESV - Effective Sliced Volume): Measures the causally relevant source code volume required. High ESV indicates Context Overload.
2. Simulation Depth (MCL - Mutation Chain Length): Measures the number of dynamic state update steps. High MCL indicates State Tracking Deficits.
3. Integration Width (DFI - Dependency Fan-in): Measures the number of independent upstream logical inputs determining the assertion value. High DFI indicates Aggregation Deficits.

Experimental Results

The authors evaluated frontier models (including Claude-Sonnet-4.5, DeepSeek-v3.1-Terminus, GPT-5.2, Kimi-K2, and Qwen3-Coder) across seven diverse Python repositories (ranging from ~1.2k to 776k Lines of Code).

Key Findings

• Performance Degradation: All models exhibited a clear decline in accuracy as task difficulty increased (Easy → Hard).
• The Aggregation Deficit: The most significant finding is a prevalent Aggregation Deficit. As Integration Width (DFI) increases, accuracy drops sharply. When DFI exceeds 20 independent upstream sources, most models (except Claude) fall below 40% accuracy.
• Cliff Effects:
  • Reading: A sharp decline in performance occurs when the Effective Sliced Volume (ESV) exceeds 600 lines of code.
  • Simulation: Significant consistency loss occurs beyond 100 execution steps (MCL).
• Model Specifics:
  • Claude-Sonnet-4.5 maintained the highest accuracy across all levels but still showed fragility in multi-source integration.
  • DeepSeek-v3.1 demonstrated the most balanced architecture, showing exceptional robustness against high DFI.
  • GPT-5.2 was the most vulnerable across all dimensions.
• Partial Correlation: Even after controlling for reading load (ESV) and simulation depth (MCL), DFI remained the only metric with a statistically significant negative correlation with accuracy, confirming it as the primary cognitive bottleneck.

Contributions

The paper claims four primary contributions:

1. From Prediction to Abductive Verification: Pioneering a repository-level paradigm that shifts from direct outcome prediction to verifying masked assertions, decoupling architectural intelligence from syntactic noise.
2. The Semantic Oracle Framework: Resolving the "Richness-Contamination Dilemma" via Execution-Driven Mutation, ensuring tasks are logically authentic yet unmemorized.
3. White-Box Diagnostic Instrument: Introducing a granular diagnostic system (ESV, MCL, DFI) that maps reasoning failures to specific cognitive bottlenecks (context overload, state tracking, aggregation).
4. Empirical Unveiling of Aggregation Deficits: Providing the first quantitative analysis of agentic reasoning across massive codebases, identifying Integration Width (DFI) as the primary barrier for next-generation agents.

Significance and Limitations

Significance:
The paper argues that the path to optimizing the next generation of agentic software engineering requires moving beyond local reasoning. The findings suggest that future research must prioritize high-breadth information synthesis and long-chain state consistency to overcome the identified aggregation deficits. The white-box diagnostic approach offers a roadmap for pinpointing specific cognitive failures rather than relying on end-to-end pass/fail metrics.

Limitations:
The authors acknowledge several constraints:

• Language Scope: Currently limited to Python, though the architecture is designed to be portable.
• Logical Depth: The reasoning depth is partially bounded by the complexity of the original repository test suites, though the mutation engine expands this diversity.
• Trace Depth: A maximum call depth of 3 is used for metric computation to optimize signal-to-noise ratios, filtering out low-level infrastructure functions.
• LLM Involvement: While LLMs are used in the mutation and variable resolution stages, the ground-truth values are derived strictly from concrete code execution, not model generation.