Technical Summary: EARS – Explanatory Abstention for Reliable Sub-Agent Modeling

Problem Statement

In large-scale enterprise Multi-Agent Systems (MAS), a centralized architecture is commonly employed where a powerful coordinator delegates user requests to lightweight, domain-specialized sub-agents. While this design enhances modularity and cost efficiency, it introduces significant reliability challenges. Specifically, sub-agents built on smaller, fine-tuned models often lack the calibration to recognize their own limitations. Instead of abstaining from ambiguous, under-specified, misrouted, or unsupported requests, these sub-agents frequently "over-answer," producing hallucinated outputs or unfaithful execution results.

Existing research has primarily focused on improving reliability at the coordinator layer (e.g., better routing or planning). However, failures occurring at the sub-agent level—where the agent returns an unfaithful result rather than a clear signal of inability—remain largely unexplored. Current abstention methods are typically designed for single-agent systems, where refusal is a final user-facing action, rather than a communication signal within a multi-agent protocol.

Methodology: EARS Framework

The authors propose EARS (Explanatory Abstention for Reliable Sub-Agent Modeling), a production-oriented framework that reframes sub-agent abstention as an inter-agent communication protocol. Rather than simply refusing a request, a sub-agent exposes an actionable failure state to the coordinator, including a specific failure category and a rationale.

EARS operates through a two-stage, iterative pipeline:

1. Offline Calibration Stage

• Human Annotation & Taxonomy: The authors define a taxonomy of four mutually exclusive failure modes:
  1. Ambiguous Query: User intent is unclear due to underspecified information.
  2. Insufficient Input: Intent is clear, but execution requires missing information.
  3. Missing Capability: The sub-agent lacks the functionality to support the request.
  4. Misrouting: The request belongs to a different sub-agent.
     Human annotators label interaction data using a hierarchical decision procedure, generating 602 abstention instances and 600 solvable queries.
• LLM-as-a-Judge Calibration: To scale data curation, the authors calibrate an ensemble of LLMs (acting as judges) on the human-annotated seed set. They employ an iterative refinement process where a secondary LLM analyzes prediction errors to update instructions, ensuring the judges generalize well without overfitting.

2. Online Data Flywheel Stage

• Consensus Data Curation: The calibrated judges (from OpenAI, Claude, and Gemini families) are applied to large-scale production interaction data. A conservative unanimous agreement strategy is used: a sample is labeled as an abstention case only if all judges agree on the category. This ensures high-precision training data.
• Sub-Agent Fine-Tuning: The curated data is used to Supervised Fine-Tune (SFT) the sub-agents. Training instances include the user query, context, and the expected response: either a task-specific output (for solvable queries) or a structured abstention response with a category and rationale.
• Deployment Loop: The updated sub-agent is redeployed into the MAS to collect new interactions, feeding back into the data curation stage for continuous improvement.

Key Contributions

1. Formulation of Explanatory Abstention: The paper formulates sub-agent abstention not as a user-facing refusal, but as an actionable coordination signal for production MAS, providing specific failure categories and rationales to the coordinator.
2. Data Curation Pipeline: The authors introduce a pipeline that iteratively identifies interaction states requiring abstention, utilizing an ensemble of calibrated LLM-as-a-Judge models with unanimous agreement to ensure high-quality supervision.
3. Large-Scale Evaluation: The framework is evaluated on a real-world, large-scale e-commerce platform supporting enterprise business intelligence workflows, demonstrating practical viability.

Experimental Results

The system was evaluated on an e-commerce assistant handling merchant queries regarding shop operations, specifically focusing on a Business Intelligence (BI) sub-agent (built on Qwen3-32B-Instruct) that generates SQL-like DSL queries.

• Judge Performance: The calibrated ensemble of judges achieved high accuracy (90.4% for the best single model, 93.1% for 4-judge consensus) in identifying abstention categories, with a clear trade-off between cost and precision.
• Sub-Agent Performance:
  • Overall Pass Rate: EARS improved the overall response pass rate from 68.5% (Baseline) to 78.9%, a relative gain of 15.2%.
  • Semantic Correctness: Significant improvements were observed in customer segmentation queries (Semantic pass rate increased from 61.9% to 82.7%), where abstention cases are more frequent (23.8% of queries).
  • Syntax Validity: Syntax validity remained high for both query types, indicating that the abstention training did not degrade the quality of valid DSL generation.
• Human Evaluation: In a shadow deployment, human experts rated the EARS system's session-level success at 67.1%, compared to 2.4% for the incumbent system on the same sessions where abstention signals were triggered. The model also achieved 94.0% abstention precision against expert judgments.

Significance and Claims

The paper claims that EARS provides a practical mechanism for improving reliability in large-scale centralized MAS. By shifting the focus from top-down coordination to sub-agent-side explanatory abstention, the framework addresses a critical gap in system reliability: the communication breakdown when a sub-agent cannot fulfill a request.

The authors emphasize that their approach transforms the sub-agent from a "black box" that may hallucinate into a transparent component that signals specific failure modes. This allows the coordinator to perform downstream recovery actions, such as clarification, rerouting, or fallback, rather than presenting the user with a hallucinated or incorrect answer. The results demonstrate that equipping sub-agents with the ability to calibrate their responses to capability constraints significantly enhances the overall robustness of the multi-agent system.