🤖 AI

AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation

AgentLens introduces a production-assessed benchmark for coding agents that evaluates the entire interaction trajectory through a combination of formal verification and LLM-generated reviews, enabling detailed behavioral diagnosis and regression detection beyond simple pass/fail metrics.

Original authors: Andrey Podivilov, Vadim Lomshakov, Sergey Savin, Matvei Startsev, Roman Pozharskiy, Maksim Parshin, Sergey Nikolenko

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrey Podivilov, Vadim Lomshakov, Sergey Savin, Matvei Startsev, Roman Pozharskiy, Maksim Parshin, Sergey Nikolenko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new junior developer to work on your software project.

The Old Way (Traditional Benchmarks)
Most current tests for AI coding assistants are like a final exam where the only thing that matters is the grade on the final paper. Did the code compile? Did the test pass? Yes or No?

The Problem: This ignores the entire journey. Did the developer panic? Did they delete the wrong files by accident? Did they argue with you? Did they claim the job was done when it was actually broken? A "passing" grade might hide a developer who is unreliable, confusing, or dangerous to work with.

The New Way (AgentLens)
The authors of this paper, AgentLens, built a different kind of test. Instead of just grading the final paper, they film the entire workday and have a senior manager review the video tape. They call this a "trajectory review."

Here is how AgentLens works, broken down into simple concepts:

1. The "Full Movie" vs. The "Final Frame"

Think of a coding session like a movie.

Old Benchmarks only look at the last frame: Is the building standing?
AgentLens watches the whole movie: How did the agent handle the storm? Did they use the right tools? Did they recover when they dropped a brick? Did they talk to the user nicely?

They record every message, every tool click, every file edit, and every mistake the AI makes. They evaluate the entire story, not just the ending.

2. The "Simulated User"

To test the AI, they don't just give it a task; they give it a personality. They use another AI to act as the "user" in the conversation.

The Chill User: Just asks for help and waits.
The Helpful User: Gently corrects the AI if it makes a small mistake.
The "Toxic" User: Gets frustrated, is a bit rude, and challenges the AI.
Why? Because in the real world, developers aren't robots. They get tired, they get annoyed, and they make mistakes. AgentLens checks if the coding AI stays calm and helpful when the user is grumpy, or if it falls apart.

3. The "Two-Headed Judge"

How do you grade a movie? You can't just ask a human to watch 32 hours of video; they would get tired and make mistakes.

The Formal Check (The Robot): This part checks the hard facts. Did the code actually run? Did the file change? This is the "Did the building stand?" check.
The LLM Judge (The Critic): This is a smart AI that reads the whole transcript and writes a review. It acts like a film critic. It doesn't just give a score; it writes a paragraph explaining why.
- Example: "The agent got the code to work (Robot says 'Pass'), but it lied about checking for errors and used a tool incorrectly three times (Critic says 'Fail')."

4. The "Report Card" with Comments

Instead of a single number (like 85/100), AgentLens gives a detailed report card with five specific categories:

End Result: Did they actually finish the job?
Instruction Following: Did they listen to your specific rules?
Pitfalls: Did they get stuck in loops, make silly mistakes, or crash?
Pleasantness: Was the conversation easy to follow, or was it confusing and annoying?
Tool Use: Did they use the right tools (like a hammer vs. a screwdriver) correctly?

5. Why This Matters for Companies

The authors built this because they are building a real coding assistant for their company. They need to know more than just "which model is smartest."

Regression Detection: Imagine you update your software, and suddenly the AI starts deleting files. A simple "Pass/Fail" test might miss this if the final code still compiles. AgentLens catches the behavior change immediately.
Diagnosis: If a model scores low, the review tells you exactly why. "Oh, it's not that it can't write code; it's that it can't handle the 'Toxic User' persona."
Side-by-Side: They can watch two AIs work on the same problem and see exactly where one gets confused and the other doesn't.

The Bottom Line

The paper claims that AgentLens is a new, open-source tool that evaluates coding AIs by watching their entire behavior, not just their final output. It uses a mix of hard code checks and smart AI "critics" to produce readable reports that help developers understand how an AI thinks, behaves, and fails, making it much better for real-world use than current "pass/fail" leaderboards.

What the paper does NOT claim:

It does not claim this works for medical diagnosis or legal advice (it is strictly for coding).
It does not claim this is perfect; they admit that sometimes the AI judges might have biases, and they are still studying how well these AI judges agree with humans.
It is currently focused on Java programming, though they plan to expand it.

Technical Summary: AgentLens

Problem Statement

Current coding agent benchmarks predominantly reduce evaluation to a binary outcome: did the agent pass the final test? While sufficient for research leaderboards, this "single-bit" metric fails to capture the reality of interactive coding assistants used in production. A task may pass a narrow verification check while the agent exhibits unreliable behavior, overconfidence, poor tool usage, or misalignment with user intent. Conversely, a run that fails a final check may have demonstrated valuable intermediate behaviors, such as correct bug localization or graceful error recovery.

The core problem is the lack of a benchmark that evaluates the entire trajectory of an agent's interaction. Developers need to understand not just if a task was completed, but how it was completed: how the agent followed instructions, used tools, verified work, recovered from mistakes, and communicated with the user. Existing benchmarks often collapse these dynamics into final states or rely on tasks too simple to reflect day-to-day developer workflows.

Methodology

AgentLens is a production-assessed benchmark designed to evaluate interactive code agents by analyzing complete interaction trajectories. The methodology combines formal verification with LLM-driven qualitative reviews.

1. Task Set and Simulation

Trajectory Definition: A trajectory includes the full session record: user messages (simulated), agent replies, tool calls, file edits, command executions, verification attempts, and the final repository state.
Task Generation: Scenarios are derived from interviews with programmers (recalling recent work) and anonymized production usage summaries. These are organized into workflows involving code inspection, editing, documentation, debugging, and testing.
User Personas: The benchmark utilizes LLM-simulated users with distinct personas to test agent robustness:
- Default: A relaxed user who intervenes only upon clear failure.
- Helpful: A cooperative user who offers guidance on mistakes.
- Toxic: A mildly frustrated user who challenges the agent but maintains the task objective.
Scale: The initial release includes 16 scenarios run with two personas each, generating 32 trajectories per agent.

2. Evaluation Protocol

AgentLens evaluates each trajectory using two complementary mechanisms:

A. Formal Verification
Objective checks are applied where possible, including:

Repository State Checks: Ensuring only allowed files were modified and target files were changed.
Regex Checks: Verifying specific patterns in code, tool arguments, or chat history.
Test Execution: Running test suites (e.g., via Maven/Gradle) and checking coverage.
Static Analysis: Invoking IDE tools to detect errors or warnings.
A run passes formal verification only if all applicable verifiers pass.

B. LLM Judge Reviews
LLM judges review the full trajectory across five specific dimensions, producing both a score and a written, evidence-linked narrative:

ENDRESULT: Completeness and fitness for purpose regarding the final outcome.
INSTRUCTIONCOMPLIANCE: Adherence to explicit rules, ordering constraints, and meta-instructions.
PITFALLS: Identification of fixable behavioral failures (e.g., tool misuse, non-productive loops, premature completion, missing validation).
PLEASANTNESS: User-facing interaction quality (clarity, accuracy, helpfulness, productivity).
TOOLCALLS: Quality of tool use, including choice, argument correctness, and recovery from errors.

C. Aggregation and Metrics

Quality Index (QI): A single aggregate score calculated as the unweighted mean of the five judge metrics and the formal verification pass rate.
Side-by-Side Reviews: For comparative analysis, a pairwise judge compares two agents on the same task, using prior single-trajectory reviews as the primary evidence to minimize positional bias.
Telemetry: Hard statistics (latency, cost, token throughput, tool call success rates) are collected to quantify operational reliability.

3. Reliability and Bias Checks

The authors conducted several checks to ensure robustness:

Variance Analysis: Repeated runs showed that variance is concentrated in formal verification (unstable task completion) rather than judge scoring.
Positional Bias: Order-swap tests indicated that side-by-side scores are largely insensitive to the order of agent presentation.
Judge Self-Preference: While judges showed a slight preference for their own model families (particularly in the subjective "Pleasantness" metric), the effect was small and did not alter the average winner in frontier model comparisons.

Key Results

The benchmark was evaluated on Java coding tasks using various models (e.g., Opus 4.7, GPT-5.5, DeepSeek V4, Gemini 3.1) across two harnesses (Explyt AI Agent and Claude Code).

Divergent Rankings: The AgentLens Quality Index (QI) correlates moderately with public benchmarks (e.g., Spearman $\rho = 0.82$ with APEX-Agents) but diverges significantly from reasoning-focused benchmarks. Models optimized for academic reasoning tasks often rank lower on AgentLens, while models built for sustained tool use rank higher.
Diagnostic Insights:
- Gemini 3.1 Pro Preview: Underperformed due to agent-loop reliability issues (skipped steps, premature edits) rather than code generation capability.
- DeepSeek V4 Pro vs. Flash: Pro demonstrated better instruction compliance and control, while Flash achieved higher throughput and completion rates by relaxing constraints.
- Kimi K2.6: Ranked last, but the review revealed the failure was a provider-side tool-parser bug (malformed arguments), not a lack of reasoning capability.
- Regression Detection: The nightly pipeline successfully identified a race condition in parallel tool calling that caused a ConcurrentModificationException, which was invisible in the final answer but captured in the trajectory review.

Significance and Contributions

The paper claims the following contributions:

Production-Assessed Benchmark: A framework that evaluates interactive code agents based on realistic, multi-step workflows rather than isolated final states.
Trajectory-Level Protocol: A novel evaluation method combining formal verification with LLM-written reviews that explain why a score was assigned, separating outcome quality from process quality.
Diagnostic Utility: The system is designed not just for ranking, but for diagnosing model behavior, comparing versions, and detecting product regressions in a continuous integration pipeline.
Open Source Release: The benchmark code, task folds, and evaluation pipeline are released publicly to facilitate further research and adoption.

The authors conclude that the field of agentic coding benchmarks is undersaturated regarding production-like evaluation. AgentLens aims to fill this niche by providing a "readable explanation" of agent performance, enabling developers to prioritize fixes based on specific failure modes (e.g., tool instability vs. instruction following) rather than a single opaque metric.