🤖 machine learning

Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents

This paper introduces the Insights Generator (IG), a multi-agent system that automates the diagnosis of LLM agent failures by systematically analyzing large-scale execution trace corpora to produce evidence-backed natural language insights, which have been shown to significantly improve agent performance and outperform manual diagnostic methods in depth and coverage.

Original authors: Akshay Manglik (Emily), Apaar Shanker (Emily), Kaustubh Deshpande (Emily), Jason Qin (Emily), Yash Maurya (Emily), Veronica Chatrath (Emily), Vijay S. Kalmath (Emily), Levi Lentz (Emily), Yuan (Emily

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Akshay Manglik (Emily), Apaar Shanker (Emily), Kaustubh Deshpande (Emily), Jason Qin (Emily), Yash Maurya (Emily), Veronica Chatrath (Emily), Vijay S. Kalmath (Emily), Levi Lentz (Emily), Yuan (Emily), Xue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a team of very smart, but sometimes confused, robot assistants. These robots are trying to solve complex puzzles, like organizing spreadsheets or answering difficult science questions. Sometimes, they get the answer right; other times, they fail.

The Problem: The "Needle in a Haystack" Debugging
Right now, when a robot fails, human managers try to figure out why by looking at a few specific examples of the robot's work. It's like trying to understand why a car engine is making a weird noise by listening to just one car drive by. You might hear a sputter, but you miss the pattern that all the cars sputter when they turn left on a rainy day.

Because the robots produce massive amounts of data (thousands of pages of "thoughts" and actions for every single task), humans can't read it all. They end up guessing based on a tiny sample. This means they miss the "silent failures"—mistakes where the robot finishes the task without crashing, but does it the wrong way.

The Solution: The "Insights Generator" (IG)
The authors built a new system called the Insights Generator (IG). Think of IG not as a single detective, but as a specialized detective agency with a specific workflow to solve the "needle in a haystack" problem.

Here is how the agency works, using a simple analogy:

The Orchestrator (The Agency Manager): This is the boss. It doesn't do the digging itself. Instead, it looks at the big picture and decides what kind of investigation is needed.
The Scouts (The Explorers): These agents are sent out to look at a small, random sample of the robot's work. Their job is to be broad and curious. They look for weird patterns and come up with theories.
- Example Theory: "Hey, I noticed that 80% of the time the robot fails at math, it writes the formula wrong but doesn't crash. It's a 'silent failure'."
The Investigators (The Forensic Auditors): Once a Scout has a theory, the Investigators take over. They don't just look at a few examples; they go through the entire massive database of robot work (the "corpus"). They use powerful tools to count, compare, and prove whether the theory is true for the whole group, not just the few they saw.
- The Result: Instead of a guess, they produce a report saying: "We checked 1,000 failed attempts. 84% of them had this specific silent math error. Here are the exact pages where it happened."

Why is this different?
Most other systems try to fix one robot at a time or categorize errors into a pre-made list (like "Tool Error" or "Logic Error"). IG is different because it discovers new patterns that no one knew existed. It separates the "guessing" (Scouts) from the "proving" (Investigators) so it doesn't get overwhelmed by the sheer size of the data.

What Happened When They Tested It?
The researchers tested this system in two ways:

The "Judge" Test: They had a super-smart AI judge compare reports made by IG against reports made by other systems. IG's reports were rated higher because they had better evidence and deeper explanations. It was like comparing a report that says "The car is broken" (other systems) vs. a report that says "The car breaks specifically when turning left in rain because of a loose wire, and here is the video proof" (IG).
The "Human Fixer" Test: This was the most important part. They gave real human engineers (experts who build these robots) the reports from IG and from other systems. The engineers then tried to fix the robots.
- The Result: Engineers who used the IG reports fixed the robots 30% better than engineers who used the next-best system. The IG reports gave them the exact "where" and "why" they needed to make the right changes, rather than just a vague hint.

The Bottom Line
The paper claims that by using this "Scout and Investigator" team approach, we can finally see the hidden patterns in how AI agents fail. This allows human experts to fix the underlying issues much more effectively, turning a chaotic pile of error logs into a clear, evidence-backed roadmap for improvement.

What the paper does not claim:

It does not say this system fixes the robots automatically without human help (though they tested an automated "patcher" loop, the main success was with human experts).
It does not claim this works for every type of AI problem in the world, only for analyzing the "traces" (logs) of AI agents.
It does not promise to eliminate all errors, but rather to make the process of finding and fixing them much more efficient and accurate.

Technical Summary: Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents

1. Problem Definition

Diagnosing failures in Large Language Model (LLM) agents currently relies heavily on manual inspection of a small subset of execution traces. Practitioners form ad-hoc hypotheses based on these limited samples, a process that fails to scale to production corpora where individual traces can span tens of thousands of tokens. This manual approach misses systematic patterns that only emerge across populations of traces, such as "silent failures" (where code executes without error but implements the wrong logic) or cohort-specific behavioral differences.

Existing debugging methods generally fall into two categories:

Single-trajectory analysis: Techniques like counterfactual replay or intervention-driven debugging focus on isolating failure causes in individual runs but miss systemic patterns across the corpus.
Fixed taxonomy classification: Methods that classify failures into predefined categories cannot surface novel failure modes or silent inefficiencies that fall outside the taxonomy.

The core challenge is Corpus-Level Trace Diagnostics: producing grounded, natural-language insights that characterize systematic behavioral patterns across groups of traces, linked to supporting evidence, without exceeding the context window limits of any single model.

2. Methodology: The Insights Generator (IG)

The authors present the Insights Generator (IG), a multi-agent system designed to perform iterative, hypothesis-driven aggregation over large trace corpora. IG separates hypothesis generation from validation and routes all trace access through a stateful Python data layer, preventing raw trace content from flooding the LLM context.

System Architecture

IG operates via a three-role agent topology coordinated by an Orchestrator:

Orchestrator: The central coordinator that decomposes diagnostic questions, dispatches sub-agents, evaluates the sufficiency of findings, and synthesizes the final report. It performs minimal direct trace inspection to preserve context capacity.
Scout Agents (Hypothesis Generation): These agents explore representative samples of traces (50–200 traces) using LLM-driven summarization and extraction tools. Their goal is breadth: generating diverse, testable hypotheses based on initial patterns (e.g., "verbose database queries correlate with null results").
Investigator Agents (Hypothesis Validation): These agents receive specific hypotheses and validate them at corpus scale. They use programmatic analysis tools to compute distributional statistics, perform cohort comparisons (e.g., success vs. failure groups), and return findings labeled as confirmed, refuted, or inconclusive, supported by quantitative evidence.

Data Processing and Tooling

A critical component of IG is the stateful Python execution environment.

Structured Data Store: Traces are not passed as raw text. Instead, they are processed into a structured store (Polars DataFrames) with a schema cache containing statistics, null rates, and correlations.
Programmatic Tools: Agents interact with traces via callable Python functions (e.g., load_traces, extract, compare_segments). This allows for arbitrary data science transformations (group-bys, statistical tests) without traversing the model context.
Iterative Loop: The system operates in rounds. The Orchestrator decides whether to dispatch more Scouts (to find new patterns) or Investigators (to validate prioritized hypotheses) until coverage is sufficient, at which point it synthesizes a final evidence-backed report.

3. Key Contributions

The paper formalizes corpus-level trace diagnostics as a distinct problem from single-trajectory debugging and introduces:

Iterative Corpus-Level Analysis: A decompose-hypothesize-validate loop that operates over corpora exceeding single context windows, surfacing patterns (like silent computation failures) that single-pass methods miss.
Evaluation Framework: A four-setting framework for assessing trace-analysis systems, varying the evaluator (calibrated LLM judge vs. human expert) and the metric (report quality vs. downstream scaffold performance).
Measurable Practitioner Impact: Empirical evidence demonstrating that insights generated by IG lead to significantly better scaffold modifications compared to other analysis systems.

4. Evaluation and Results

The authors evaluated IG on two benchmarks: SpreadsheetBench (spreadsheet manipulation tasks) and HLE (Humanity's Last Exam, expert-level QA). They compared IG against four baselines: Single-Agent Coding, CC Subagents (Claude Code with task dispatch), Trace2Skill-style systems, and Recursive Language Models (RLM).

LLM-as-a-Judge Evaluation

Using a calibrated LLM judge (Claude Opus 4.6), reports were scored on coverage, pairwise win rate, and rubric-based quality (mechanism, specificity, actionability).

Performance: IG achieved the highest average pairwise win rate (77.9%) and composite quality score (1.32), outperforming CC Subagents (62.4% win rate) and Trace2Skill (52.6%).
Depth: IG led specifically in mechanism explanation and specificity, indicating its findings were more grounded in evidence and detailed in their causal explanations.
Ablations: Removing the typed Scout/Investigator roles (Generic Subagents) or the parallel dispatch (Orchestrator-only) resulted in significant performance drops, confirming the value of the specific role decomposition and parallel processing.

Human Expert Intervention Study

In a study with professional software engineers, participants used IG reports or CC Subagent reports to modify an agent scaffold to improve performance on held-out tasks.

Result: Practitioners using IG reports improved scaffold performance by 30.4 percentage points (pp) over the unmodified baseline.
Comparison: This was a 14.2 pp improvement over the next-best system (CC Subagents, which improved by 16.2 pp). The difference was statistically significant ( $p \approx 0.016$ ).
Efficiency: IG-condition participants made more targeted edits (modifying more files but with smaller net line changes on average), suggesting higher precision in their modifications.

Iterative Patcher Loop

In an autonomous setting where a coding agent iteratively patches a scaffold based on analysis reports:

Systems using reports (IG, RLM, CC Subagents) converged to high test pass rates (~0.81–0.84).
The Pure-Patcher baseline (no analysis report) initially improved but then regressed sharply (to 0.58), inventing edits that contradicted the evaluation contract. This highlights that regenerated corpus-level analysis is necessary to sustain improvement as the scaffold approaches saturation.

Human Expert-as-a-Judge

When human experts rated the reports directly, IG and CC Subagents received comparable aggregate scores. However, IG scored slightly higher on depth and evidence quality, while CC Subagents scored slightly higher on correctness. The authors suggest the LLM judge may be better at evaluating the scale of evidence across large corpora than humans working under fixed time budgets.

5. Significance and Claims

The paper claims that Insights Generator establishes a new standard for diagnosing LLM agents by shifting from manual, sample-based inspection to systematic, corpus-level pattern discovery.

Practitioner Impact: The primary significance is the demonstrated link between diagnostic quality and practitioner effectiveness. The 30.4 pp gain in scaffold performance proves that deep, evidence-backed insights translate directly to better engineering outcomes.
Systematic Discovery: IG successfully identifies "silent failures" and cohort-specific behaviors that standard evaluation metrics and single-trajectory debuggers miss.
Scalability: By decoupling hypothesis generation from validation and utilizing a stateful data layer, IG addresses the scalability bottleneck of analyzing traces that exceed context windows.

The authors conclude that while automated optimization frameworks exist, they often optimize toward fixed objectives. IG complements these by surfacing latent behavioral insights in trace corpora, which can then be consumed by optimization frameworks in a closed loop. The work is modest about generalizability, noting that the human intervention study was conducted on SpreadsheetBench and requires further validation across diverse domains and larger sample sizes.