🤖 machine learning

E2HiL: Entropy-Guided Sample Selection for Efficient Real-World Human-in-the-Loop Reinforcement Learning

The paper proposes E2HiL, an entropy-guided sample selection framework for real-world human-in-the-loop reinforcement learning that improves sample efficiency and success rates by actively filtering out noisy and shortcut samples to retain only those with moderate influence on policy entropy.

Original authors: Haoyuan Deng, Yudong Lin, Yuanjiang Xue, Haoyang Du, Qianzhun Wang, Boyang Zhou, Zhenyu Wu, Ziwei Wang

Published 2026-08-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haoyuan Deng, Yudong Lin, Yuanjiang Xue, Haoyang Du, Qianzhun Wang, Boyang Zhou, Zhenyu Wu, Ziwei Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long been masters of the factory floor, performing repetitive tasks with unerring precision. Yet, when asked to navigate the messy, unpredictable chaos of a human home or a dynamic workshop, they often stumble. The core difficulty lies in teaching a machine to adapt. Traditional programming fails because it cannot account for every possible variation in the real world. Instead, researchers turn to a method where the robot learns by doing, trying actions, failing, and slowly figuring out what works. This process, known as reinforcement learning, is powerful but notoriously slow and expensive. To speed it up, scientists have introduced a human teacher into the loop. When the robot makes a mistake, a person can step in, correct the action, and show the machine the right way. This "human-in-the-loop" approach has shown promise, but it comes with a heavy price: it requires a vast amount of human time and effort, often overwhelming the teacher with unnecessary corrections.

A team of researchers has now developed a smarter way to manage this partnership, a system they call E2HiL. Rather than accepting every correction a human offers, this new framework acts as a discerning editor, deciding which lessons are truly valuable and which are distractions. The researchers discovered that the key to efficient learning lies in a concept called "entropy," which in this context measures how much the robot is exploring new possibilities versus sticking to what it already knows. If a robot becomes too confident too quickly, it stops exploring and gets stuck in a local solution, missing better ways to perform a task. Conversely, if it remains too uncertain, it wastes time wandering aimlessly. The team found that existing methods often let the robot's confidence collapse too early, causing it to give up on exploring before it has truly learned.

To solve this, the E2HiL system analyzes every single interaction between the robot and the human to gauge its impact on the robot's learning curve. It looks for a specific pattern in the data: interactions that cause the robot's confidence to drop too sharply or those that have almost no effect at all. The system identifies these as "shortcut" samples, where a human might force a quick fix that prevents the robot from understanding the underlying mechanics, and "noisy" samples, which are corrections so minor they add no value. By filtering out these unhelpful moments, the system ensures that the robot only learns from interventions that offer a steady, manageable reduction in uncertainty. This allows the robot to maintain a healthy balance between trying new things and refining its skills, leading to faster mastery with far less human fatigue.

The researchers tested this approach on ten different real-world tasks using various robotic arms, ranging from simple pick-and-place operations to complex manipulations like folding towels and inserting blocks. They compared their new system against the current best methods for human-guided learning. The results were striking. The E2HiL framework increased the robot's success rate by nearly 25 percent while simultaneously reducing the number of times a human needed to intervene by about 9 percent. In one specific task involving a cube that needed to be touched, the system allowed the robot to learn two distinct skills in sequence, whereas the standard method struggled to move past the first one. The system achieved this not by changing the robot's hardware or its basic learning algorithm, but by simply being more selective about the data it used to update its brain.

What makes this finding particularly significant is that the system works regardless of the specific robot or the type of task it is performing. It functions as a plug-and-play module that can be added to existing learning frameworks without requiring a complete redesign. The researchers verified that their method does not rely on perfect initial guesses or flawless data; it adapts as the robot learns, stabilizing the learning process even when early data is imperfect. By focusing on the quality of the human-robot interaction rather than just the quantity, E2HiL demonstrates that the path to efficient robotic learning is not about more human effort, but about smarter selection. This approach offers a clear path forward for deploying robots in real-world settings where time and human attention are limited resources, proving that sometimes the most effective way to teach a machine is to teach it what to ignore.

Technical Summary: E2HiL – Entropy-Guided Sample Selection for Efficient Real-World Human-in-the-Loop Reinforcement Learning

1. Problem Statement

Real-world robotic manipulation faces significant challenges in sample efficiency and the high cost of human labor. While Human-in-the-Loop Reinforcement Learning (HiL-RL) leverages human corrective takeovers to accelerate policy learning, existing frameworks suffer from low sample efficiency. They typically treat all intervention samples uniformly, failing to distinguish between informative guidance and unhelpful corrections. This leads to:

Inefficient Human Effort: Substantial human interventions are required for convergence, resulting in high labor costs.
Unstable Entropy Dynamics: Uniform sampling often causes premature policy entropy collapse (leading to suboptimal convergence) or ineffective updates due to noisy samples.
Exploration-Exploitation Imbalance: The inability to regulate entropy at the sample level hinders the trade-off between exploring new states and exploiting known skills.

The core question addressed is how to achieve stable, sample-efficient HiL-RL in real-world manipulation by actively selecting informative samples rather than using all available data equally.

2. Methodology: E2HiL Framework

The authors propose E2HiL, a framework that integrates entropy-guided active sample selection into the HiL-RL pipeline. The method operates on the premise that a stable reduction of policy entropy correlates with improved sample efficiency and a better exploration-exploitation trade-off.

A. Sample Entropy Dynamics Estimation

The framework quantifies the influence of each sample $(s_t, a_t)$ on policy entropy using an influence function.

Theoretical Basis: Drawing from recent findings in Large Language Model RL, the authors approximate the change in policy entropy ( $\Delta H$ ) using the covariance between action log-probabilities and soft advantages.
Influence Value ( $c(s_t, a_t)$ ): The influence of a sample is estimated as:
$c(s_t, a_t) \approx -\eta \cdot \widehat{\text{Cov}}(\log \pi_\theta(a_t|s_t), \pi_\theta(a_t|s_t) \cdot A_{\text{soft}}(s_t, a_t))$
where $A_{\text{soft}}$ is the soft advantage and $\eta$ is the learning rate.
Interpretation:
- High positive covariance indicates samples that induce sharp entropy drops (potential "shortcut" samples causing premature collapse).
- Near-zero covariance indicates "noisy" samples with negligible influence on entropy.

B. Entropy-Bounded Sample Selection

To stabilize learning, E2HiL employs a selection mechanism that prunes uninformative samples based on their estimated influence values.

Dynamic Bounds: The framework defines an adaptive interval $[\ell, u]$ for the absolute influence value $|c(s_t, a_t)|$ . These bounds are computed per batch (e.g., using the 5th and 90th percentiles).
Pruning Strategy:
- Shortcut Samples: Samples with extremely high influence (sharp entropy drops) are masked out to prevent premature policy collapse.
- Noisy Samples: Samples with negligible influence (near-zero covariance) are masked out to prevent slowing down learning.
- Retention: Only samples with "moderate" influence values are retained for policy updates.
Objective Function: The actor loss is modified to include an indicator function $I(s_t, a_t)$ , which is 1 if the sample falls within the bounds and 0 otherwise:
$L_{\text{E2HiL}}(\theta) = -\frac{\sum I(s_t, a_t) \cdot l_t}{\sum I(s_t, a_t) + \epsilon}$
This ensures gradients are only computed for entropy-consistent samples.

3. Key Contributions

E2HiL Framework: The proposal of an efficient real-world HiL-RL framework that reduces human intervention costs by actively selecting informative samples based on entropy dynamics.
Sample-Level Entropy Characterization: The derivation of influence functions to characterize how individual samples affect policy entropy via the covariance of action probabilities and soft advantages. This enables the identification and removal of "shortcut" and "noisy" samples.
Empirical Validation: Extensive evaluation across 10 real-world manipulation tasks involving multiple embodiments (Lerobot SO-101 and A1 X arms) and learning frameworks.

4. Experimental Results

The paper validates E2HiL against state-of-the-art baselines (HIL-SERL and ConRFT) in real-world settings:

Success Rate: E2HiL improves the average success rate by 24.9% compared to baselines. Specifically, on the Lerobot platform, it achieved a 45% relative improvement in success rate across four tasks. On the A1 X platform, it improved the average success rate by 17.5% over HIL-SERL and 6.6% over ConRFT.
Human Intervention: The framework reduces the average human intervention rate by 9.3% (10.1% on Lerobot, 9.4% on A1 X), significantly lowering labor costs.
Entropy Stability: E2HiL maintains a smoother entropy curve, avoiding the premature collapse observed in baselines. This allows the agent to continue exploring and acquire multiple skills (e.g., handling objects at different positions) more effectively.
Ablation Studies: Comparisons with random dropping, advantage clipping, and spatial clipping confirm that the performance gains stem specifically from entropy-informed sample selection, not generic regularization or data reduction. The method is robust to variations in hyperparameters (clipping bounds and Monte Carlo sample counts) and does not rely on perfectly accurate Q-value estimates in early training phases.

5. Significance and Claims

The authors position E2HiL as a policy- and embodiment-agnostic plug-and-play module. Its significance lies in:

Generalizability: It functions effectively across different robotic platforms and learning frameworks without requiring architecture-specific modifications.
Efficiency: It addresses the critical bottleneck of human labor costs in real-world RL by filtering out unhelpful interventions, thereby accelerating convergence.
Theoretical Insight: It bridges the gap between LLM-RL findings (regarding entropy dynamics) and robotic manipulation, demonstrating that regulating entropy at the sample level is crucial for stable real-world learning.

The paper concludes that entropy-guided sample selection is a promising strategy for achieving robust skill acquisition with reduced human supervision, paving the way for more scalable real-world multi-task RL.