ReDAct: Uncertainty-Aware Deferral for LLM Agents

1. Problem Statement

Large Language Model (LLM) agents are increasingly used for complex sequential decision-making tasks in embodied environments (e.g., ALFWorld, MiniGrid). However, these agents inherit the tendency of LLMs to hallucinate. In sequential settings, a single incorrect action can irreversibly degrade the agent's trajectory, leading to task failure.

While larger, more capable LLMs hallucinate less, they incur significantly higher inference costs per token. Conversely, smaller, cheaper models are cost-effective but prone to errors. Existing model routing methods typically treat model selection as a per-query problem (choosing one model for an entire request) and do not address the dynamic, step-by-step nature of agentic trajectories where deferral decisions must be made continuously.

The Core Challenge: How to balance the trade-off between inference cost and decision reliability in sequential agent tasks without relying exclusively on expensive large models.

2. Methodology: ReDAct Framework

The authors propose ReDAct (Reason-Defer-Act), a framework where an agent is equipped with two LLMs:

1. Small Model ($M_{small}$): A cheap, fast model used by default.
2. Large Model ($M_{large}$): A reliable, expensive model used only when necessary.

The Workflow:

1. Reasoning: The agent uses the small model to generate a reasoning trace based on the current state and history (following the ReAct paradigm).
2. Action Proposal & Uncertainty Estimation: The small model proposes an action. Simultaneously, an Uncertainty Quantification (UQ) metric is calculated for this specific action proposal.
3. Deferral Decision:
   • If the uncertainty score ($u_t$) is below a calibrated threshold ($\tau$), the small model's action is accepted.
   • If $u_t > \tau$, the decision is deferred to the large model. The large model generates its own reasoning and action, which is then accepted unconditionally.
4. Execution: The accepted action is applied to the environment, transitioning the agent to the next state.

Key Technical Choices:

• Action-Level vs. Reasoning-Level UQ: The authors empirically tested whether to compute uncertainty during the reasoning phase or the action selection phase. They found that action-level UQ provides a significantly stronger signal for deferral (higher PRR and ROC-AUC) because free-form reasoning is harder to quantify reliably.
• Uncertainty Metrics: The study evaluated several information-theoretic measures that utilize token probabilities directly (avoiding the computational cost of multiple generations):
  • Perplexity (PPL)
  • Sequence Probability (SP)
  • Mean Token Entropy (MTE)
  • Result: Perplexity (PPL) emerged as the most effective metric for guiding deferral.
• Threshold Calibration: The threshold $\tau$ is calibrated on a small set of episodes to achieve a target average number of large model calls (e.g., $K=5$ calls per episode). This ensures a controlled trade-off between cost and performance.

3. Key Contributions

1. ReDAct Framework: A novel method for sequential agentic tasks that enables dynamic, step-level deferral from small to large models based on predictive uncertainty.
2. Empirical Validation of Action-Level UQ: The paper demonstrates that uncertainty quantification at the action selection stage is far superior to reasoning-stage uncertainty for detecting errors in sequential environments.
3. Cost-Performance Pareto Optimality: The authors show that deferring only ~15% of decisions to the large model allows the system to match (and sometimes surpass) the performance of using the large model exclusively, while drastically reducing inference costs.
4. Comprehensive Evaluation: Extensive experiments on ALFWorld (text-based household tasks) and MiniGrid (grid-world navigation) using frontier-sized models (70B to 480B parameters).

4. Experimental Results

• Performance: On ALFWorld, using a small model (e.g., Qwen3-80B) with PPL-based deferral to a large model (GPT-5.2 or Qwen3-480B) achieved a success rate of ~0.80, matching the large model's standalone performance (~0.79–0.80).
• Efficiency: The deferral strategy invoked the large model for only ~15% of the total steps.
• Cost Analysis:
  • Using the large model exclusively resulted in high costs (e.g., ~$45–52 for ALFWorld).
  • ReDAct reduced costs significantly (e.g., ~$15–16 for Qwen3-80B + GPT-5.2) while maintaining high success rates.
  • The cost of ReDAct was only marginally higher than using the small model alone, despite the occasional expensive calls.
• Deferral Distribution: Unlike random deferral (uniform distribution), UQ-guided deferral showed a right-skewed distribution in ALFWorld. The small model deferred more frequently in later steps of failed episodes, indicating it correctly identified when it was "digging itself into a hole" and needed help.
• Baseline Comparison: ReDAct consistently outperformed random deferral baselines and single-model baselines.

5. Significance and Conclusion

The paper addresses a critical bottleneck in deploying LLM agents: the high cost of reliability. ReDAct provides a practical, cost-aware routing mechanism that allows developers to deploy small, cheap models in production while retaining the reliability of large models for critical decision points.

• Scalability: By deferring only when necessary, the approach makes high-performance agentic systems economically viable.
• Generalizability: While tested on text-based environments, the methodology (uncertainty-driven deferral) is applicable to other modalities, including visual-language agents.
• Future Direction: The work establishes that adaptive switching within an agent is a promising direction for efficient AI deployment, moving beyond static model selection to dynamic, context-aware decision-making.

In summary, ReDAct proves that uncertainty-aware deferral is a highly effective strategy to mitigate hallucinations in sequential tasks, achieving near-optimal performance with a fraction of the computational cost.