Technical Summary: TrajAudit

Problem Statement

Agentic coding systems, powered by Large Language Models (LLMs), are increasingly deployed for repository-level software maintenance tasks such as bug fixing and feature development. While effective in simple, single-file modifications, these agents often fail in complex, multi-file scenarios requiring long-horizon reasoning. Understanding where and why these failures occur is critical for iterative refinement and operational reliability.

Existing automated failure diagnosis methods face two primary challenges when applied to repository-level coding trajectories:

1. Observational Noise: Execution trajectories are laden with irrelevant information, such as redundant program structures and verbose code context (often constituting over 70% of the trajectory). This noise interferes with LLM reasoning capabilities.
2. Excessive Length: Trajectories for complex tasks often span 20 to over 100 steps, with each step containing thousands of characters. State-of-the-art LLMs struggle to maintain reasoning quality when processing such long contexts, leading to a significant degradation in diagnostic performance (dropping below 40% accuracy on long-horizon tasks).

Current approaches typically passively consume entire trajectories or fixed segments, treating all steps as equally relevant, which fails to address the specific nature of agentic failures in noisy, long-horizon environments.

Methodology: TrajAudit

To address these challenges, the authors propose TrajAudit, the first automated failure diagnosis framework specifically designed for repository-level coding trajectories. TrajAudit employs an Investigator Agent supported by two synergistic modules to actively explore and fetch relevant fragments from long, noisy trajectories.

Core Components

1. Prior Failure Reasoning Module:

   • Function: Generates a preliminary diagnosis before the agent processes the full trajectory.
   • Mechanism: It prompts an LLM to analyze the failed test code and its corresponding error description to identify the most suspicious region (e.g., specific phases like "Exploration" or "Code Implementation") responsible for the failure.
   • Goal: This preliminary diagnosis serves as prior knowledge, directing the Investigator Agent's focus toward the most probable failure segments and mitigating long-context degradation.

2. Semantic Saliency Folding Module:

   • Function: Selectively compresses trajectory observations to reduce noise.
   • Mechanism: It employs pattern matching (e.g., detecting code patch headers like --- a and +++ b) and keyword detection (e.g., "exception," "fail," "error") to identify failure-relevant context.
   • Action: Observations containing these signals are preserved. Irrelevant observations (e.g., verbose file creation logs, directory trees) are "folded" into a single token placeholder, effectively removing noise while retaining the ability to retrieve full content on demand.

3. Investigator Agent:

   • Function: Acts as the central hub for diagnosis.
   • Mechanism: It receives the preliminary diagnosis and the folded trajectory. It dynamically assesses whether the current context is sufficient for localization. If not, it iteratively invokes APIs to retrieve the full content of specific folded observations on demand.
   • Strategy: This enables a top-down diagnostic approach, starting with a high-level overview and selectively drilling down into details only when necessary, ensuring critical information is preserved while minimizing noise.

Key Contributions

The paper introduces three main contributions:

1. Framework: TrajAudit, an automated framework that localizes the "earliest decisive error step" (the first step where an incorrect action leads to task failure) and provides diagnosis justifications for agentic coding trajectories.
2. Benchmark (RootSE): The first benchmark dedicated to evaluating failure diagnosis in agentic programming.
   • Scale: Comprises 93 complex failure instances sourced from real-world software maintenance tasks (SWE-bench and SWE-bench Pro).
   • Complexity: Contains over 4,500 execution steps and approximately 27 million characters, representing the most complex trajectory diagnosis benchmark to date.
   • Annotation: Includes ground-truth labels for the earliest decisive error step and textual justifications, curated by experts with substantial inter-rater agreement (Cohen's Kappa = 0.78).
3. Evaluation: A comprehensive evaluation of TrajAudit against existing baselines on RootSE, demonstrating significant improvements in localization accuracy and token efficiency.

Experimental Results

Experiments were conducted on the RootSE benchmark using various backbone LLMs (Claude-Sonnet, GPT-5.2, DeepSeek-R1) and compared against baselines including Random Attribution, All-at-Once, Step-by-Step, and Binary Search.

• Localization Accuracy: TrajAudit outperformed the strongest baseline (All-at-Once) by 24.4 percentage points in Exact Step-Level Accuracy in the without-reference setting (50.9% vs. 26.9%). The performance gap widened significantly as trajectory length increased, with TrajAudit showing a >70% accuracy gap over baselines on trajectories exceeding 80 steps.
• Token Efficiency: TrajAudit demonstrated superior efficiency, reducing token consumption by at least 18% compared to the most efficient baseline (All-at-Once) and by over 1200% compared to the Step-by-Step approach.
• Justification Quality: TrajAudit achieved the highest Justification Accuracy (71.0% total), significantly outperforming baselines, particularly on longer trajectories.
• Ablation Study: Removing the Semantic Saliency Folding module caused the most significant drop in accuracy (11.4%), highlighting its critical role in noise reduction. Removing Prior Failure Reasoning also reduced accuracy (3.5–3.9%), confirming the value of preliminary guidance.

Significance and Claims

The paper claims that TrajAudit provides a foundational resource for the community to address failure management in agentic software engineering. By introducing a framework that actively filters noise and leverages prior reasoning, it overcomes the limitations of passive, long-context processing. The authors position RootSE as a necessary step forward, arguing that existing benchmarks (like Who&When) fail to capture the complexity and length of real-world repository-level tasks, thereby masking the limitations of current diagnosis methods.

The work emphasizes that effective failure diagnosis in agentic systems requires moving beyond simple trajectory analysis to active, noise-aware investigation, a capability TrajAudit demonstrates through its superior accuracy and efficiency on complex, real-world software maintenance failures.