Technical Summary: Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks

Problem Statement

As frontier AI models grow more capable, their evaluation increasingly relies on complex agentic benchmarks. However, the validity of these benchmarks is frequently compromised by flaws that are difficult to detect through manual review alone. Previous manual audits of benchmarks like SWE-Bench-Verified and CORE-Bench have uncovered validity issues such as ground truth leakage, tool failures, and answer format ambiguities. Yet, manual auditing is not scalable; evaluation runs can require hundreds of millions of tokens per task, making comprehensive human review infeasible. While automated methods exist, it remains unclear whether they can reliably surface the specific flaws that compromise benchmark validity without extensive human oversight.

Methodology

The authors developed a framework for automated transcript analysis using "scanners" (automated AI agents) to detect four specific types of validity issues: Ground Truth Access, Tool Failures, Guessing Vulnerability, and Answer Format Ambiguity.

Framework and Criteria Selection

The study utilized the Agentic Benchmark Checklist (ABC) (Zhu et al., 2025a) as a foundational framework. From the ABC's 30 criteria, the authors identified those suitable for transcript analysis based on two requirements:

1. Presence: The criterion must be theoretically observable within a transcript.
2. Verifiability: The criterion must be confirmable via a scanner or human review.
   Criteria requiring external state comparisons (e.g., database states) or metadata checks were excluded. The authors refined the selected criteria into four scanner types (Table 1).

Data and Experimental Setup

• Benchmarks: The study evaluated 11 agentic evaluations, including SWE-Bench-Verified, CORE-Bench, MLE-Bench, and CVE-Bench.
• Models: Transcripts were generated using GPT-5.4, Claude Sonnet 4.6, and GPT-5-mini. The scanners themselves were implemented using GPT-5.4 and Claude Sonnet 4.6.
• Development Strategy:
  • Development Set: 770 transcripts were human-labeled. Due to the rarity of natural violations, the authors supplemented this set with synthetic violations (e.g., injecting gold patches into system prompts, removing tools, or stripping format specifications) to train and iterate on scanner prompts.
  • Test Set: A held-out set of 742 transcripts was used for final validation.
• Human Grading: Five human reviewers labeled transcripts using a 0–3 severity rubric (0: No Evidence, 1: Potential, 2: Clear, 3: Impactful). Ambiguous cases were resolved via consensus.
• Sampling: To estimate scanner sensitivity efficiently, the authors employed a stratified post-validation design, over-sampling scanner-flagged transcripts to ensure a sufficient number of true positives for metric calculation.

Evaluation Metrics

Scanner performance was quantified against human labels as the ground truth. Metrics included:

• Quadratic-Weighted Cohen's Kappa (QWK): To measure agreement on the ordinal severity scale.
• Binary Classification Metrics: Sensitivity, Specificity, Precision, and F1 Score, using a threshold of $\ge 2$ to define a violation.
• Inverse-Probability Weighting: Applied to test set metrics to correct for the stratified sampling bias.

Key Contributions

1. Scanner-Viable Criteria Identification: The authors identified a subset of validity criteria from the ABC framework that are amenable to automated transcript analysis.
2. Scanner Development and Validation: They developed and validated scanners for four specific criteria (Ground Truth Access, Tool Failures, Guessing, Answer Format Ambiguity), demonstrating that automated methods can track human judgments.
3. Sampling Methodology: The paper outlines a sampling methodology to determine the number of transcripts required to confidently rule out high violation rates or estimate prevalence.
4. Open Source Resources: The authors released their scanners and a dataset of transcripts across 11 agentic evaluations.
5. Empirical Findings: The study reports verified issues in widely used benchmarks, including cases unlikely to be caught by random manual inspection.

Results

Scanner Performance

Scanner performance varied significantly across criteria, benchmarks, and model choices:

• Overall Agreement: Scanners generally tracked human post-validation judgments. Transcripts with higher human severity scores were more likely to be flagged by scanners.
• Best Performance: The Guessing scanner (GPT-5.4) achieved the highest F1 score (0.74) and QWK (0.75).
• Worst Performance: The Tool Failure scanner (Sonnet 4.6) performed poorly (F1 = 0.14, QWK = 0.15), often failing to distinguish between intended benchmark design (e.g., missing dependencies as a feature) and actual failures.
• Composite Approaches: Combining scanner outputs (e.g., taking the "Max" flag or "Floor Mean") often improved performance, particularly for reducing false positives or increasing sensitivity.

Verified Benchmark Issues

The scanners identified several confirmed validity issues:

• CORE-Bench: Frequent ground truth access where answers were embedded in plotting code or results files, allowing models to bypass the intended task. Answer format ambiguity (e.g., unspecified significant digits) led to spurious failures.
• SWE-Bench-Verified: Cases where overly descriptive prompts revealed the gold patch verbatim, and instances where models accessed upstream git history to retrieve human-generated patches.
• CVE-Bench: While scanners over-flagged many samples due to the benchmark's nature (seeking out-of-bounds info), they successfully identified cases of memorized keys and exploits.
• Terminal-Bench-2.0: Identification of tasks vulnerable to guessing (e.g., single-digit outputs) where models attempted reward hacking.

Limitations Observed

• Context Dependency: Scanners struggled when lacking context about the benchmark's intended design (e.g., misinterpreting intentional environment constraints as tool failures).
• Ambiguity: Human graders often rated ambiguous cases as "Potential" (1), while scanners rated them higher, leading to higher flag rates.
• Generalization: Scanners trained on specific transcript structures sometimes failed to generalize to new formats (e.g., multi-turn conversations in Tau2-bench).

Significance and Claims

The paper positions this work as a proof of concept for using automated transcript analysis to audit benchmark quality. The authors claim that:

1. Scalability: Automated scanners can rapidly flag suspicious transcripts and identify systematic failure modes that are difficult to detect via random manual inspection.
2. Supplemental Role: Scanners are not a replacement for human judgment but serve as a supplemental tool to prioritize scarce human review and support post-validation sampling strategies.
3. Systematic Improvement: The approach makes benchmark auditing more systematic, scalable, and empirically grounded, though it currently lacks the ability to provide definitive proof that an evaluation is problem-free.
4. Standardization Needs: The results highlight a need for stronger benchmark documentation and validation standards, as even limited automated efforts uncovered significant issues in published benchmarks.

The authors conclude that while scanner performance varies, the integration of automated transcript analysis is essential for improving the reliability and interpretability of AI benchmarks as evaluations become more consequential.