TraceSIR: A Multi-Agent Framework for Structured Analysis and Reporting of Agentic Execution Traces
TraceSIR is a multi-agent framework that addresses the challenges of analyzing complex agentic execution traces by employing specialized agents to compress data, perform fine-grained root cause analysis, and generate actionable reports, demonstrating superior performance on the newly introduced TraceBench and ReportEval benchmarks.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
🕵️♀️ The Problem: The "Black Box" of AI Agents
Imagine you hire a super-smart robot assistant (an AI Agent) to solve a complex mystery, like finding a specific TV show based on a vague description.
The robot doesn't just give you the answer instantly. It goes on a long journey:
- It thinks, "Maybe it's this show?"
- It searches the internet.
- It reads a Wikipedia page.
- It gets confused, tries a different search, and reads 50 more pages.
- Finally, it gives you the answer: "Tell Me What You Saw."
The Issue: Sometimes the robot gets it wrong. If you just look at the final answer, you don't know why it failed. Did it search the wrong thing? Did it misread a clue? Did it get distracted?
In the real world, these "journeys" (called execution traces) can be thousands of pages long.
- Human Reviewers can't read 50,000 pages of logs for every mistake. It takes too long.
- Standard AI Tools try to read the whole thing at once, but they get overwhelmed (like trying to drink from a firehose) and start hallucinating or giving nonsense answers.
- Current Methods often just look at the final score (Pass/Fail) without understanding the behavior that led to the failure.
🛠️ The Solution: TraceSIR (The Detective Team)
The authors created TraceSIR, which acts like a specialized detective agency for AI mistakes. Instead of one person trying to do everything, they use a team of three specialized AI agents working together.
Think of it as a three-person investigative unit:
1. The Summarizer (StructureAgent) 📝
The Job: The raw data is a messy, 50,000-page novel. The Summarizer is the editor who condenses it into a structured comic book.
- How it works: It takes the long, boring logs and turns them into a clean, 3-column format: Thought (What the robot was thinking), Action (What it did), and Observation (What happened next).
- The Magic: It cuts out the fluff (like long code snippets or repetitive searches) but keeps the critical clues. It turns a "firehose" of data into a "water bottle" that is easy to drink.
2. The Diagnostician (InsightAgent) 🩺
The Job: This is the doctor who examines the "comic book" created by the Summarizer.
- How it works: It looks at the specific case and asks: "Where did the robot go wrong?"
- Did it make a logical leap?
- Did it misunderstand the tool it was using?
- What is the Root Cause? (e.g., "The robot failed because it searched for the actor's name instead of the show's title.")
- The Output: It doesn't just say "Error." It gives a detailed medical report with a diagnosis and a prescription for how to fix it.
3. The Reporter (ReportAgent) 📊
The Job: Imagine you have 100 robots failing 100 different tasks. The Reporter looks at all of them together to find patterns.
- How it works: It aggregates the diagnoses from the Diagnostician.
- "Hey, 40% of the robots failed because they got confused by dates."
- "Here is a trend: When the task involves coding, the robots tend to give up too early."
- The Output: It writes a comprehensive industry report (in Markdown) that engineers can actually use to improve the system. It includes statistics, common failure patterns, and specific suggestions for improvement.
🏆 The Proof: TraceBench & ReportEval
To prove their system works, the team built a testing ground called TraceBench.
- They took 150 real-world failure cases from three different types of AI tasks: Deep Research (finding info), Function Calling (using tools), and Coding (writing software).
- They created a new way to grade the reports called ReportEval. Instead of just checking if the report exists, they asked: "Is this report useful? Is the root cause accurate? Are the suggestions actionable?"
The Results:
When they compared TraceSIR to the current industry standard (ClaudeCode), TraceSIR won hands down.
- Better Diagnosis: It found the real reasons for failure, not just surface-level errors.
- Better Reports: The reports were clearer, more logical, and actually helped engineers fix the problems.
- Scalability: It could handle huge amounts of data that would crash other systems.
💡 The Big Picture
TraceSIR is like upgrading from a "Black Box" (where things just happen and we guess why) to a "Glass Box" (where we can see the gears turning, spot the broken gear, and fix it).
It solves the problem of too much data by summarizing it, too much complexity by breaking it down, and too much noise by finding the signal. It turns a chaotic mess of AI logs into a clear, actionable roadmap for building better AI systems.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.