Technical Summary: AGENTLENS: Revealing The Lucky Pass Problem in SWE-Agent Evaluation

1. Problem Statement

Current evaluation of Software Engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the test suite (e.g., SWE-bench). This "outcome-only" view treats a principled, efficient solution and a chaotic, trial-and-error process as equivalent. The authors argue this equivalence is empirically false and insufficient for evaluating agent behavior.

Specifically, the paper identifies the "Lucky Pass" problem: a subset of successful trajectories that reach a correct patch through weak, brittle, or inefficient processes. These include:

• Regression cycles: Repeatedly failing and retrying similar edits.
• Blind retries: Repeating actions without new information.
• Missing verification: Applying fixes without running tests.
• Temporally disordered exploration: Performing verification steps before implementation or exploring irrelevant files after a fix is verified.

The paper posits that treating all passing trajectories as equally valuable supervision data (as done in datasets like SWE-Gym) is a coarse proxy for demonstration quality. Furthermore, as models converge on pass-rate benchmarks, process quality becomes a necessary axis for model comparison and risk assessment, as agents succeeding via trial-and-error may behave unpredictably in complex, real-world deployments.

2. Methodology: The AGENTLENS Framework

The authors introduce AGENTLENS, a framework for process-level assessment of SWE-agent trajectories. It operates in four stages:

A. From Raw Logs to Labeled States

Raw agent logs are parsed into structured state sequences. Each state is assigned an intent-stage label from four cognitive phases:

1. Exploration (E): Reading files, searching, listing directories.
2. Implementation (I): Editing or creating source files.
3. Verification (V): Running tests, checking errors, re-reading edited files.
4. Orchestration (O): Bookkeeping and reasoning steps.

Crucially, AGENTLENS employs a context-sensitive intent-stage labeler. Unlike naive approaches that label actions based solely on tool identity (e.g., assuming grep is always exploration), this labeler uses trajectory history. For instance, grep is labeled as Exploration before a patch is written but as Verification if performed after an implementation step to check a specific change. This resolves ambiguities in terminal commands, validated with a Fleiss' $\kappa = 0.933$ across seven annotators.

B. Building a PTA Reference

Instead of comparing a candidate trajectory to a single reference trace, AGENTLENS constructs a Prefix Tree Acceptor (PTA) from $k \ge 2$ passing trajectories for the same task.

• Merging: Shared prefixes are merged into common nodes.
• Branching: Divergent but successful strategies form branches.
• Result: The PTA is a Directed Acyclic Graph (DAG) encoding a space of "known-good" solution strategies rather than a single exemplar. This allows the system to recognize valid alternative paths while flagging deviations from known successful processes.

C. Scoring Candidate Trajectories

A candidate trajectory is scored against the task-level PTA using four complementary signals, combined into a 0–100 quality score:

1. Structural Alignment ($\Phi_{struct}$): Measures if the candidate visits PTA states in the roughly correct order (ordered recall + unordered precision).
2. Set Coverage ($\Phi_{cov}$): Measures the fraction of PTA states matched by the candidate, regardless of order.
3. Trajectory Coherence ($\Phi_{coh}$): Summarizes the intent-stage sequence, rewarding forward progress (e.g., $E \to I \to V$) and penalizing backtracks and blind retries.
4. Temporal Profile Similarity ($\Phi_{temp}$): Compares the distribution of stages across early, middle, and late trajectory segments against the PTA using Jensen-Shannon divergence.

The final score is a weighted sum:
$$f(\tau_c, G) = 0.20 \cdot \Phi_{struct} + 0.15 \cdot \Phi_{cov} + 0.30 \cdot (100 \cdot \Phi_{coh}) + 0.35 \cdot (100 \cdot \Phi_{temp})$$
Behavioral signals (Coherence and Temporal Profile) carry 65% of the weight, reflecting that structural coverage and process quality often fail on different trajectories.

D. Reports and Tiering

Trajectories are classified into tiers based on their composite score:

• Ideal ($\ge 70$): Principled, low-waste, well-ordered.
• Solid ($47 \le score < 70$): Imperfect but coherent.
• Lucky ($< 47$): Weak processes that happen to pass (e.g., regression cycles, blind retries).
• Partial-fail / Off-track: For failing trajectories.

The system also generates structured inefficiency reports, localizing "waste" (regression loops, blind retries, redundant steps, unnecessary exploration, cyclic patterns) to specific trajectory steps.

3. Key Contributions

1. AGENTLENS-Bench: The first process-annotated SWE-agent trajectory dataset. It contains 1,815 trajectories from 47 SWE-bench Verified tasks, featuring quality scores, waste annotations, divergence metadata, and ground-truth PTAs.
2. The Lucky Pass Finding: The authors demonstrate that 10.7% of passing trajectories are "Lucky Passes." These are decomposed into five recurring mechanisms:
   • Minimal & Unverified (15.6%): Short, no verification.
   • Brute-Force Convergence (34.4%): High waste, low coherence.
   • Incomplete Implementation (33.6%): Partial fixes passing due to test gaps.
   • Excessive Exploration (4.1%): Long, unfocused.
   • Divergent-but-Valid (12.3%): Alternative valid approaches.
3. Context-Sensitive Intent Labeling: A validated labeler that resolves exploration-vs-verification ambiguity in terminal commands, achieving 96.0% raw agreement.
4. PTA-Based Process References: A method for merging multiple passing solutions into a reference space of known-good strategies, enabling the scoring of new trajectories against a diverse set of correct behaviors.
5. Open-Source Tooling: Release of the AGENTLENS SDK and web interface, supporting ATIF and OpenHands traces without requiring LLM calls for scoring.

4. Results

The framework was evaluated on 2,614 trajectories from eight model backends (including GPT-4.1, GPT-4o, Claude Opus, and Gemini 2.5 Pro) across 60 SWE-bench Verified tasks.

• Distribution of Passing Trajectories: Among 1,136 passing trajectories, 20.2% were Ideal, 69.1% were Solid, and 10.7% were Lucky.
• Model Ranking Discrepancy: Process-aware scoring significantly alters model rankings compared to pass-rate alone.
  • Example: GPT-4o ranks 8th by pass rate but 3rd by quality score.
  • Example: Claude Opus 4.6 ranks 3rd by pass rate but 6th by quality score, with a 18.7% Lucky rate (the highest among top-tier models).
  • Lucky rates across models range from 0.5% (Opus 4.5) to 23.2% (GPT-4.1).
• Taxonomy Specificity: There is a significant association between model type and Lucky Pass category ($\chi^2(28) = 102.47, p < 0.0001$). For instance, Opus 4.6 disproportionately produces "Minimal & Unverified" passes, while GPT-4.1 dominates "Brute-Force" and "Excessive Exploration" categories.
• Discriminative Power: The composite score separates passing and failing trajectories with an AUROC of 0.766 ($p=0.0017$). No single signal achieved statistical significance alone, highlighting the necessity of fusing structural and behavioral signals.
• Cost Implications: Lucky Passes incur significantly higher computational costs. "Brute-Force" trajectories cost ~25x more tokens than "Minimal" ones, and "Excessive Exploration" costs ~40x more, despite receiving identical binary pass/fail labels.

5. Significance and Claims

The paper claims that AGENTLENS provides a necessary layer of evaluation beyond binary outcomes. Its significance lies in three areas:

1. Data Curation: It challenges the practice of filtering training data solely by pass rate, showing that "successful" trajectories vary wildly in quality. Using low-quality "Lucky" trajectories for supervision may reinforce brittle behaviors.
2. Model Comparison: It offers a new axis for comparing models. As models converge on high pass rates, the ability to solve tasks via coherent, low-waste processes becomes a differentiator. The paper demonstrates that ranking by quality score can shift model positions by up to five ranks.
3. Deployment Risk: It highlights that agents succeeding via trial-and-error may be unpredictable in environments with expensive tests or irreversible actions. By exposing "Lucky Passes," AGENTLENS helps practitioners identify models that rely on brittle exploration rather than principled reasoning.

The authors position AGENTLENS not as a replacement for functional correctness checks, but as a complementary diagnostic tool to improve the evaluation, analysis, and training of software engineering agents. They release the dataset and tools to make process-aware analysis a standard component of the SWE-agent evaluation pipeline.