Technical Summary: TrajSpec – Trajectory-Guided Bug Report Specification Refinement

Problem Statement

Repository-level Automated Program Repair (APR) agents rely on bug reports as their primary task specifications. However, standard bug reports often describe only the observed failure symptoms while omitting critical repair-relevant information, such as the underlying failure mechanism, specific behavioral requirements, and the full implementation scope. Consequently, APR agents may inspect irrelevant code, infer incorrect requirements, or generate patches that address the reported symptom without restoring the intended repository behavior. While prior work has focused on improving agent search strategies, localization, or prompt engineering, these approaches often assume the input report provides a sufficient specification. There is a gap in methods that explicitly refine the report by gathering missing specification details from the repository before the downstream repair process begins.

Methodology: TrajSpec

The authors propose TrajSpec, a trajectory-guided approach for repository-supported specification refinement. The system operates on an original bug report ($b$) and a pre-fix repository snapshot ($R_c$) through the following pipeline:

1. Unverified Trajectory Collection:
   TrajSpec executes a trajectory-collection agent using only $b$ and $R_c$. This agent explores the repository, inspects code, and reasons about the issue. Crucially, this run is unverified: TrajSpec does not validate any candidate patches produced during this phase. Any candidate patch is discarded, and only the execution trajectory ($\tau$)—a sequence of thought-action-observation tuples—is retained. This trajectory serves as a source of repository-supported evidence, even if the agent's final patch was incorrect.

2. Hierarchical Evidence Abstraction:
   Raw trajectories are often noisy and lengthy. TrajSpec extracts candidate findings from $b$ and $\tau$ focusing on three dimensions:

   • Failure Mechanism: The source-code behavior explaining the symptom.
   • Behavioral Requirement: The behavior that should hold.
   • Implementation Scope: The code locations involved.

   These findings are organized into a hierarchical representation ($M$) with three levels of detail for each dimension:

   • High-level: A candidate specification conclusion.
   • Mid-level: Diagnostic reasoning and relationships (e.g., code paths, dependencies).
   • Low-level: Concrete repository observations (e.g., specific files, functions, variables).

3. Draft Generation and Repository-Based Review:
   Using $b$ and $M$, an LLM generates a draft refined report ($\hat{b}$) following a fixed schema (Title, Description, RootCause, StepsToReproduce, ExpectedBehavior, ObservedBehavior).

   A repository-based review step then validates $\hat{b}$ against $R_c$. A reviewer agent assesses whether claims in the draft are supported by the evidence in $M$ and the actual source code. It removes unsupported claims, revises uncertain statements, adds missing repository-supported details, and ensures the implementation scope is appropriately bounded. The output is the final refined report ($\hat{b}_f$), which serves as the task specification for the downstream repair agent.

Key Contributions

• Formulation: The paper formulates bug report enhancement for repository-level APR as "repository-supported specification refinement," aiming to make explicit the failure mechanism, behavioral requirement, and implementation scope.
• TrajSpec Framework: Introduction of a method that extracts and hierarchically organizes specification evidence from an unverified trajectory-collection run, reviews this evidence against the source code, and generates a refined report without assuming the trajectory's candidate patches are correct.
• Comprehensive Evaluation: Evaluation on all 300 SWE-Bench Lite instances using Mini-SWE-Agent V2, demonstrating significant performance improvements.
• Generalization: Demonstration that the benefits of TrajSpec generalize across different downstream repair agents (Agentless and AutoCodeRover).
• Component Analysis: Ablation studies confirming that both the hierarchical evidence representation and the repository-based review are critical to performance gains.

Evaluation Results

The authors evaluated TrajSpec on 300 SWE-Bench Lite instances:

• Primary Repair Agent (Mini-SWE-Agent V2):

  • With GPT-5-mini, Pass@1 improved from 41.00% (original reports) to 59.67%.
  • With MiniMax M2.5, Pass@1 improved from 54.67% to 64.33%.
  • TrajSpec outperformed a baseline "Agentic-Base" (which uses trajectory data but lacks hierarchical abstraction and repository review) in both settings.
  • The improvements were broadly distributed across 12 different repositories, with TrajSpec expanding repair coverage while preserving almost all instances previously repaired by original reports.

• Cross-Agent Generalization (Stratified Sample of 100 instances):

  • Agentless: Pass@1 improved from 41.00% to 71.00%.
  • AutoCodeRover: Pass@1 improved from 47.00% to 72.00%.

• Ablation Studies:

  • Removing repository-based review reduced Pass@1 from 59.67% to 48.00%.
  • Removing the hierarchical evidence representation reduced Pass@1 to 47.67%.
  • This confirms that both structuring evidence and verifying claims against the repository are essential.

• Cost Analysis:

  • While TrajSpec incurs an additional cost for report generation (approx. $0.083 per instance with GPT-5-mini), it reduces downstream repair input token usage by ~24% and lowers the monetary cost of the repair run itself. The total end-to-end cost remains modest relative to the significant gains in repair success.

Significance and Claims

The paper claims that TrajSpec provides a promising direction for improving repository-level repair by addressing the "specification problem" inherent in underspecified bug reports. The authors emphasize that:

1. Trajectories are reusable beyond patch generation: Even unverified trajectories contain valuable evidence about failure mechanisms and code scope that can be abstracted and structured to improve the task specification.
2. Verification is critical: Simply using trajectory data is insufficient; a hierarchical structure and a repository-based review step are necessary to filter noise and ensure claims are grounded in the pre-fix code.
3. Task specification matters: Improving the input specification (the bug report) is as important as improving the repair agent itself. TrajSpec demonstrates that providing agents with actionable, repository-supported context consistently improves repair performance across different models and agent architectures.

The authors maintain a modest stance, noting that their evaluation is limited to Python repositories in SWE-Bench Lite and that the effectiveness for other languages or benchmarks remains future work. They also acknowledge that while the refined reports improve automated repair metrics, the study focuses on utility for APR rather than human-perceived report quality.