Technical Summary: R2V-Agent – Teaching SLMs When to Ask for Help

Problem Statement

Efficient agentic systems operating in interactive environments (e.g., software engineering, web navigation, system administration) face a fundamental trade-off between reliability and cost. Reliability often depends on expensive frontier Large Language Models (LLMs), particularly when tasks involve long-horizon planning, recovery from failed tool calls, or reasoning under incomplete observations. Conversely, Small Language Models (SLMs) are cost-effective but brittle under out-of-distribution perturbations.

Existing LLM cascades and routers typically perform query-level allocation, deciding which model to use before execution begins or after a full response is generated. This approach is ill-suited for agentic Partially Observable Markov Decision Processes (POMDPs), where task difficulty is trajectory-dependent. A task may start in-distribution but become risky mid-trajectory due to flaky tool calls, truncated observations, or compounding local errors. Routing the entire task to an SLM is brittle, while routing the entire task to an LLM is unnecessarily expensive. The core challenge is to determine when a deployed local SLM should defer to a stronger teacher LLM during the execution of a multi-step agent trajectory.

Methodology

The authors propose R2V-Agent, a risk-calibrated framework for step-level SLM-LLM routing. The system factorizes execution into four components:

1. Distilled SLM Policy ($\pi_\theta$): A local model trained to handle routine steps.
2. Teacher LLM ($\pi_T$): A stronger model invoked only when necessary.
3. Process Verifier ($V_\phi$): A lightweight module that scores candidate actions at each step.
4. Calibrated Router ($r_\psi$): The central contribution, which estimates residual failure risk at each step.

Training Pipeline

The system is trained as an ordered pipeline rather than a joint multi-loss objective to ensure the router learns the failure modes of a fixed policy:

1. Offline Distillation (SLM Training):

   • Behavioral Cloning (BC): The SLM is warm-started using teacher trajectories that have been augmented with perturbation operators (simulating tool failures, missing observations, prompt injection, etc.).
   • Verifier-Guided DPO: The BC policy generates candidate actions, which are scored by the verifier. Preference pairs are constructed between verifier-approved/teacher-recovered actions and lower-quality BC samples. The SLM is refined using Direct Preference Optimization (DPO) with the BC policy frozen as the reference.
   • Consistency Regularization: A Jensen-Shannon Divergence (JSD) term encourages the SLM to produce stable action distributions across different perturbation realizations of the same latent state, improving robustness.

2. Router Training:

   • Once the SLM policy is fixed, the router is trained on the residual failures of this policy.
   • Features: The router ingests a 15-dimensional vector including token-level entropy, verifier score statistics (mean, spread, best/worst), candidate consistency, and trajectory state features.
   • Labels: The routing label $y_t$ is binary, indicating whether the SLM-only rollout containing the current context eventually failed the episode.
   • Objective: The router minimizes a Brier-calibrated loss to estimate the conditional probability of episode failure. Crucially, it employs a Conditional Value-at-Risk (CVaR)-constrained objective. This penalizes routers that are cheap on average but brittle under worst-case perturbation seeds, ensuring robustness against tail risks.

Inference

At test time, the distilled SLM executes by default. The router estimates the residual risk $p_t = r_\psi(f_t)$. If $p_t$ exceeds a calibrated threshold $\tau^*_{route}$ and the LLM budget remains, the system escalates to the teacher LLM; otherwise, it proceeds with the SLM action.

Key Contributions

1. Stateful Step-Level Routing Formulation: The paper formulates SLM-LLM agent routing as a stateful allocation problem in perturbed POMDPs, moving beyond static query-level decisions.
2. Residual-Risk Routing Objective: The authors introduce a routing objective combining Brier-calibrated probability estimation with CVaR-constrained tail-risk control. This ensures the router escalates only when the estimated risk justifies the cost, avoiding brittle policies.
3. Empirical Validation: The framework is evaluated across three benchmarks (HumanEval+, TextWorld, TerminalBench) and four SLM backbones (Gemma-9B, LLaMA-3.1-8B, Qwen2.5-7B/14B), demonstrating that calibrated step routing significantly reduces frontier-model calls while preserving high task success.

Results

R2V-Agent improves the reliability-cost frontier across all tested benchmarks:

• HumanEval+ (Code Generation): In a low-failure regime, R2V achieves 94.3% success with only 0.6% LLM escalation, a significant improvement over the SLM-only baseline (91.9%) with negligible cost.
• TextWorld (Embodied Control): R2V recovers performance from 64.6% (SLM-only) to 98.2% success with 41.7% escalation. This is close to the hindsight oracle rate (35.4%) and significantly more efficient than heuristic routers, which require nearly 100% escalation to achieve similar success.
• TerminalBench (Shell Engineering): R2V improves success from 81.1% to 93.3% using 33.9% LLM calls. This is roughly half the cost of a heuristic router (66.0% escalation) at comparable reliability.

Ablation studies confirm that both the consistency regularization (improving SLM stability) and the full feature set (combining verifier signals and entropy) are critical. The system also demonstrates graceful degradation in closed-source settings where token log-probabilities are unavailable, using "pseudo-entropy" derived from verifier scores.

Significance

The paper argues that efficient agents do not need to choose between cheap but brittle SLM execution and expensive full LLM execution. By treating routing as a stateful, step-level decision grounded in residual risk estimation, R2V-Agent provides a practical middle ground. The work highlights that:

• Task difficulty in agents is dynamic and trajectory-dependent, rendering static query-level routing insufficient.
• Calibrated probabilistic routing, constrained by tail-risk objectives (CVaR), is superior to heuristic or entropy-based thresholds for managing compute allocation in interactive environments.
• Separating the training of a stable local policy from the training of the router allows for precise isolation of compute allocation strategies from policy improvement techniques.

The authors note that the system's performance remains dependent on the quality of the verifier and risk features, and that some failures (particularly in complex shell tasks) remain difficult to detect online, leaving room for future work in verifier improvement and dynamic budget adaptation.