💻 computer science

CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems

This paper introduces CORRECT, a lightweight, training-free framework that leverages a cache of distilled error schemata to enable rapid, targeted error localization in multi-agent systems, supported by the new CORRECT-Error dataset and demonstrating significant improvements in accuracy with near-zero overhead.

Original authors: Yifan Yu, Moyan Li, Shaoyuan Xu, Jinmiao Fu, Xinhai Hou, Fan Lai, Bryan Wang

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yifan Yu, Moyan Li, Shaoyuan Xu, Jinmiao Fu, Xinhai Hou, Fan Lai, Bryan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a busy restaurant kitchen with a team of specialized chefs (a Multi-Agent System). One chef chops vegetables, another sautés meat, and a third plates the dish. Sometimes, the final meal comes out wrong.

The big problem isn't just that the meal is bad; it's figuring out exactly which chef made the first mistake and at what second they messed up. Was it the vegetable chef who used the wrong knife? Or did the meat chef burn the steak five minutes later?

In complex kitchens, mistakes often spread like a domino effect. One small error by one chef can ruin the whole dish, creating a long, confusing story of what happened. Trying to find the root cause by reading the entire 30-minute video of the kitchen is exhausting and slow.

The Problem: Why Current Methods Fail

The paper explains that current ways of finding these errors are like trying to solve a mystery with two bad tools:

The "Judge" Method: You ask a smart AI (like a food critic) to watch the whole video and guess who messed up. But the video is so long and full of boring details (like "I am opening the fridge") that the critic gets confused and guesses wrong most of the time.
The "Retraining" Method: You try to teach a new AI by showing it thousands of examples of mistakes. But labeling these mistakes is incredibly expensive and hard. It takes experts hours to figure out just one mistake, and there are too many different types of errors to teach the AI everything.

The Solution: CORRECT (The "Cheat Sheet" System)

The authors created a system called CORRECT. Think of it as a smart, living "Cheat Sheet" for the kitchen manager.

Instead of making the AI relearn everything or watch hours of video, CORRECT does three clever things:

Distilling the "Signature": When a mistake happens, CORRECT doesn't save the whole long video. Instead, it extracts the "fingerprint" of the error.
- Analogy: If a chef always forgets to salt the soup before adding the carrots, the "fingerprint" isn't the whole recipe; it's just the rule: "No salt before carrots."
- CORRECT turns messy logs into these tiny, reusable "Error Schemata" (patterns).
The Smart Cache (The Library): It stores these fingerprints in a small, organized library.
- When a new order comes in and something goes wrong, CORRECT doesn't scan the whole history. It quickly looks up its library to find the fingerprint that matches the current situation.
- Analogy: If the soup is salty, the manager instantly checks the library, sees the "Salt before Carrots" rule, and knows exactly where to look.
No Training Required: The best part? The system doesn't need to go back to school. It learns on the fly. If it sees a new type of mistake, it creates a new fingerprint and adds it to the library immediately.

The "Training Data" Problem Solved

To prove this works, the researchers needed a huge list of mistakes to test on. But real mistakes are hard to find. So, they built a synthetic error factory (called CORRECT-Error).

They took successful kitchen runs and used a smart AI to inject realistic mistakes into them, guided by the patterns they saw in real life.
They tested this on human experts, and the experts couldn't tell the difference between the "fake" injected mistakes and real ones. This proved their system could generate endless, high-quality practice data without needing humans to label every single one.

The Results

When they tested CORRECT on seven different types of "kitchens" (from math problems to web navigation):

Accuracy: It found the exact mistake up to 20% better than the best existing methods.
Speed: It did this in sub-seconds without needing expensive computer power to retrain models.
Efficiency: It works even if the library only has a few hundred fingerprints, because most mistakes share the same "fingerprint."

In a Nutshell

CORRECT is like giving a detective a magnifying glass and a list of known criminal signatures, rather than asking them to read every single page of a 1,000-page book to find one typo. It recognizes that while every kitchen disaster looks different on the surface, they often follow the same few patterns. By remembering those patterns, it can spot errors instantly, saving time and money.

Technical Summary: CORRECT

Problem Statement

Multi-agent systems (MAS) are increasingly deployed for complex real-world tasks, yet their reliance on inter-agent coordination, tool use, and long-horizon reasoning makes decisive error recognition particularly challenging. Unlike single-agent systems where failures often stem from a single faulty output, MAS failures frequently arise from cascading effects where a minor error in one agent propagates through downstream interactions, causing task failure.

Current approaches face three fundamental obstacles:

Generality: MAS applications span diverse domains with varying error patterns. Existing "LLM-as-a-judge" methods achieve low accuracy (often ≤10% step-level accuracy) due to the difficulty of generalizing across these patterns.
Data Efficiency: Obtaining labeled data for error recognition is prohibitively expensive and inherently ambiguous. Annotating MAS failures requires expert hours to disentangle interdependent interactions, leading to high disagreement rates among annotators.
Computation Efficiency: Fine-tuning LLMs for error recognition is impractical for dynamic deployments where new error types arise continuously. Furthermore, naive In-Context Learning (ICL) fails because execution logs can exceed 32,000 tokens and contain low-entropy noise, often underperforming zero-shot baselines.

Methodology: CORRECT

The paper introduces CORRECT (COndensed eRror RECognition via knowledge Transfer), a lightweight, training-free framework that leverages an online cache of distilled error schemata to recognize and transfer knowledge of failure structures across new requests. The system operates in three phases:

1. Offline Schema Extraction

Instead of storing raw failure trajectories, CORRECT distills them into compact, reusable error schemata.

Clustering: Failure trajectories are semantically embedded and clustered to group similar error patterns.
Schema Generation: For each cluster, a representative schema is generated using an LLM. Each schema captures:
- Error Signatures ( $\Sigma$ ): Characteristic patterns (e.g., placeholder tokens in tool calls, vague action narration).
- Error Context Analysis ( $C$ ): Conditions leading to the error (e.g., task requirements, prior context).
- Detection Heuristics ( $H$ ): Actionable rules for identifying similar errors.
Optimization: The number of clusters is determined by a data-driven criterion (silhouette score) to balance compactness and coverage.

2. Online Schema-Guided Error Recognition

When a failure occurs in a new request:

Retrieval: The system retrieves the top- $k$ relevant schemata from the cache based on semantic similarity between the target trajectory and stored schemata.
Diagnosis: The LLM is prompted with the target trajectory and the retrieved schemata. The schemata act as condensed expert knowledge, directing the LLM's attention to salient failure patterns without processing full historical logs.
Output: The LLM identifies the precise agent and step that initiated the failure (decisive error).

3. Dynamic Schema Management

To maintain the cache's relevance and quality, CORRECT employs two mechanisms:

Schema Expansion: If user feedback confirms a successful recognition of a novel error (low similarity to existing schemata), a new schema is generated and cached.
Schema Distillation: Frequently accessed schemata (high "hotness") are refined by generating multiple candidates and selecting the one with the highest accuracy when replayed against prior trajectories.

Key Contributions

CORRECT Framework: The first schema-guided detector that systematically distills recurrent MAS failures into compact error schemata. Unlike fine-tuning or LLM-as-a-judge approaches, it transfers error knowledge at inference time via schema retrieval, achieving up to 20 points improvement in step-level localization accuracy while remaining training-free.
CORRECT-Error Dataset: A large-scale benchmark of over 2,000 annotated multi-agent trajectories. Generated via a novel bootstrap error-injection pipeline, this dataset combines scalability with high fidelity. It uses a small set of human-annotated errors as seeds to synthesize diverse error scenarios that are indistinguishable from real-world failures (validated by human evaluation where synthetic errors were misclassified as genuine 47.1% of the time).

Experimental Results

Experiments were conducted across seven diverse MAS applications (including HotpotQA, MMLU-Pro, GAIA, and the WHO&WHEN dataset) using various models (Qwen, Llama, GPT, Gemini, DeepSeek).

Accuracy Gains: CORRECT improves step-level error localization accuracy significantly. On the WHO&WHEN dataset, it improved Qwen-2.5-7B's exact-step accuracy from 3.5% to 12.1% and GPT-5 from 8.6% to 17.2%. Across the CORRECT-Error benchmark, it achieved an average gain of 28.7% over existing advances.
Generalization: The framework demonstrates strong transferability; schemata distilled from human-crafted trajectories effectively improve performance on algorithm-generated data.
Efficiency: The approach introduces negligible overhead (less than 3.5% latency increase) and requires no retraining.
Robustness: Performance remains stable even with small cache sizes (10% of the schema library) and in the presence of retrieval noise.
Practical Utility: Targeted interventions (e.g., restarting agents) based on CORRECT's localization improved end-to-end success rates from 10% to 20%.

Significance and Claims

The paper claims that CORRECT addresses the critical bottleneck of error recognition in scalable MAS deployments. By shifting from raw log analysis to condensed structural knowledge transfer, the framework enables:

Targeted Debugging: Precise localization of the "decisive error" (the earliest step whose correction flips the outcome to success) allows for targeted interventions (e.g., upgrading specific agents or refining orchestration logic) rather than costly system overhauls.
Scalability: The training-free, cache-based design allows systems to adapt to dynamic error patterns in subseconds without the data and compute costs of fine-tuning.
Foundation for Reliability: The introduction of CORRECT-Error provides a rigorous, extensible benchmark for the community to evaluate and improve MAS reliability, moving beyond the limitations of current small-scale, manually annotated datasets.

The authors emphasize that their work establishes a foundation for building more reliable, interpretable, and scalable multi-agent systems by treating error recognition as a systems problem solvable through knowledge distillation and retrieval.