When (and How) to Trust the Expert: Diagnosing Query-Time Expert-Guided Reinforcement Learning

Technical Summary: Diagnosing Query-Time Expert-Guided Reinforcement Learning

Problem Statement
Many continuous-control domains (industrial control, locomotion, trajectory tracking) deploy competent but suboptimal controllers, such as tuned PID loops, hand-designed gaits, or Model Predictive Control (MPC) systems. These controllers often serve as a "floor" that Reinforcement Learning (RL) agents are expected to surpass. However, the literature on query-time expert-guided RL—where an agent can query a pre-existing expert controller during training—lacks a unified evaluation framework. Existing methods (e.g., IBRL, JSRL, Residual SAC) have been proposed in isolation on different benchmarks with varying expert qualities and hyperparameter optimization (HPO) budgets. Consequently, practitioners lack a clear understanding of which integration mechanism extracts the most value from an expert, nor do they understand the specific failure modes that arise when experts are imperfect (e.g., undertuned, biased, or noisy).

Methodology and Experimental Protocol
The authors harmonize the comparison of five expert-guided methods (IBRL, two JSRL variants, Residual SAC, and a new method called EDGE) on a shared Soft Actor-Critic (SAC) backbone. The evaluation protocol is rigorous and standardized:

• Benchmarks: Four continuous-control tasks: FourTank (MIMO setpoint tracking), Plane3DCircle (3D aircraft tracking with catastrophic failure), GlassFurnace (long-horizon thermal control), and CheetahRun (locomotion). Two of these (Plane3DCircle and GlassFurnace) are released as new benchmarks.
• Seeds and Budget: 100 seeds for control tasks and 50 for locomotion, trained for 1 million environment steps.
• Expert Quality: A "degradation sweep" tests robustness against expert undertuning, action bias, and observation noise.
• Metric: Performance is measured using Expert-Normalized Advantage (ENA), which normalizes returns relative to the expert's performance and a theoretical physical ceiling.

Key Contributions

1. A Common-Protocol Benchmark: The paper establishes a head-to-head comparison of expert-guided RL methods under identical HPO and evaluation conditions, revealing that no single method dominates across all task structures.
2. Taxonomy of Failure Modes (F1–F3): The harmonized comparison surfaces three specific failure modes that single-paper evaluations miss:
   • F1 (Critic Blind Spot): In methods using argmax action selection (like IBRL), the critic becomes "anchored" to the expert's action on states where the expert wins. This prevents the critic from receiving fresh Bellman targets for the policy's own actions on those states, causing the method to underperform even a no-expert SAC baseline when the expert is very strong (near the RL ceiling).
   • F2 (Residual Saturation): Additive correction methods (Residual SAC) are bounded by the action space. On tasks with weak experts, the necessary correction to overcome the expert's structural errors exceeds this bound, capping performance well below other methods. Conversely, on near-optimal experts, this bound allows Residual SAC to inherit the expert's performance immediately.
   • F3 (Warm-Start Buffer Poisoning): Methods that use a training-time handoff (running the expert for a fixed warm-start period before switching to the policy) suffer when the expert is imperfect. The replay buffer collected during the warm-start reflects the specific (potentially flawed) expert state, and the policy fails to recover when deployed on a different or perturbed system.
3. Decision Rule: The authors derive a testable decision rule mapping three observables (expert quality, task termination, and expected perturbation type) to a recommended method. For instance, if the expert is near-ceiling and non-terminating, IBRL is predicted to fail (F1); if the expert is weak and the task is terminating, Residual SAC is predicted to fail (F2).
4. EDGE (Expert-Driven Guided Exploration): The paper introduces EDGE, a new method designed to avoid the identified failure modes. EDGE replaces the deterministic argmax gate with a quality-aware pessimism-gated stochastic mixing mechanism. It uses an ensemble of critics to compute a Lower Confidence Bound (LCB) and applies a softmax gate to probabilistically select between the expert and the policy. Crucially, it augments the state with the expert's internal controller state (e.g., PID integrator state) to ensure the critic has full visibility.

Results

• No Dominant Method: Each method excels in a specific regime but fails predictably elsewhere. For example, Residual SAC ties for the top on FourTank (near-ceiling expert) but collapses on Plane3DCircle (weak expert). IBRL is dominated by no-expert SAC on FourTank due to F1.
• Robustness: Under expert perturbations (undertuning, bias, noise), clean rankings reverse. Methods that maintain a fallback to the expert at every step (JSRL-curriculum, EDGE) are more robust to training-time expert degradation, while Residual SAC is more robust to deployment-time action bias due to its bounded correction.
• EDGE Performance: EDGE is competitive across all tasks. On Plane3DCircle, it significantly outperforms literal Thompson sampling and other baselines, demonstrating that the combination of a stochastic gate and LCB scoring is effective.
• The "RL-Near-Ceiling" Wall: On tasks where the expert is already very close to the performance of a no-expert RL agent (e.g., FourTank at 98% of the no-expert RL return), no query-time expert method clears the expert within the 1M-step budget. The authors note this may be a fundamental limitation or a budget effect, but it suggests that for highly competent experts, the overhead of query-time mixing may not yield gains within standard training horizons.

Significance and Claims
The paper's primary contribution is the benchmark, taxonomy, and decision rule, which provide practitioners with a framework to select an expert-guided method based on the specific characteristics of their expert and task. The authors explicitly state that EDGE is offered as a "design probe" to demonstrate that the identified axes (gate form and scoring rule) are individually exploitable, rather than a claim of State-of-the-Art (SOTA) dominance.

The work highlights that the integration mechanism is critical: a method that works well with a perfect expert may fail catastrophically with a slightly imperfect one. By identifying F1, F2, and F3, the paper moves the field from isolated method proposals to a systematic understanding of when and why expert-guided RL succeeds or fails. The authors conclude that for experts that are already near the physical or RL ceiling, the most practical solution within current budgets may be to deploy the expert directly, as the marginal gain from RL integration is currently negligible.