Optimizing Neurorobot Policy under Limited Demonstration Data through Preference Regret

1. Problem Statement

The paper addresses two critical limitations in current Robot Reinforcement Learning from Demonstrations (RLfD):

• Data Scarcity: Most RLfD algorithms assume an abundance of expert data, which is unrealistic in real-world scenarios due to high collection costs and time constraints.
• Cascading Errors (Covariate Shift): Traditional Imitation Learning (IL) and Behavior Cloning (BC) treat data as independent and identically distributed (i.i.d.). During online execution, small planning errors accumulate, causing the agent to deviate from the training distribution into unobserved states, leading to policy collapse.
• Suboptimal Expert Data: Existing methods often blindly imitate expert demonstrations, even when the expert data is noisy or suboptimal, failing to optimize for the true environmental reward.

2. Methodology: The MYOE Framework

The authors propose the "Master Your Own Expertise" (MYOE) framework, a self-imitation learning approach that combines Active Inference (AIF), Generative World Models, and Preference Regret minimization.

A. Core Architecture: QMoP-SSM

The framework utilizes a Queryable Mixture-of-Preferences State Space Model (QMoP-SSM). This is a generative world model that separates learning into two parallel processes:

1. Representation Learning: Infers the hidden state of the world ($s^o_t$) from observations ($o_t$) to predict future states and rewards.
2. Preference Learning: Predicts future "preferred" state trajectories ($s^p_t$) based on learned goals ($g_\theta$) and expert demonstrations.

The model is trained to minimize Free Energy, which involves:

• Minimizing the Kullback-Leibler (KL) divergence between the approximate posterior and the prior (reducing "surprise").
• Maximizing observation and reward prediction accuracy.
• Minimizing the distance between the Preference Posterior and the Representation Posterior (ensuring the agent's imagined goals align with successful states).

To prevent "mode collapse" (where the model only learns one specific trajectory), the SSM employs a Mixture of Preferences. It estimates a collection of stochastic preference distributions, allowing the agent to explore diverse paths to the same goal while maximizing entropy to encourage exploration.

B. Behavioral Learning: Preference Regret

The core innovation is the introduction of Preference Regret as an intrinsic reward signal. Instead of simply imitating expert actions, the agent optimizes its policy by minimizing the difference between the rewards predicted by the preference model and the actual environment rewards.

The total reward signal ($R_\tau$) is formulated as:
$$R_\tau = \underbrace{r^o_\tau}_{\text{Env Reward}} - \underbrace{(r^p_\tau - r^o_\tau)}_{\text{Preference Regret}} + \alpha H[\text{Entropy}]$$

• Preference Regret Term: $(r^p_\tau - r^o_\tau)$ represents the difference between the reward estimated by the preferred trajectory ($r^p$) and the actual observed reward ($r^o$).
  • If the agent deviates from a high-quality preference, the regret is negative (penalizing deviation).
  • If the expert demonstration is suboptimal (i.e., the actual environment reward $r^o$ is higher than the preference estimate $r^p$), the regret becomes positive. This incentivizes the agent to ignore the suboptimal expert and pursue higher environmental rewards, effectively solving the "blind imitation" problem.
• Entropy Term: Encourages exploration (curiosity) by maximizing the entropy of the state distribution.

The policy is trained using Proximal Policy Optimization (PPO) with Generalized Advantage Estimation (GAE), utilizing the augmented reward signal to update the actor-critic networks.

3. Key Contributions

1. QMoP-SSM: A novel state space model that jointly learns world representations and goal-conditioned preference trajectories, enabling the agent to "imagine" future states and goals.
2. Preference Regret Mechanism: A new intrinsic reward formulation that allows agents to adaptively balance between imitating experts and optimizing for environmental rewards. It mitigates cascading errors by guiding the agent away from suboptimal expert paths when necessary.
3. Mixture of Preferences: An extension to the SSM that uses a mixture of stochastic distributions to prevent mode collapse and ensure diverse exploration of the state space.
4. MYOE Framework: A complete end-to-end system for online RLfD that operates effectively with extremely limited data (only 5 expert episodes).

4. Experimental Results

The authors evaluated MYOE against state-of-the-art baselines (DreamerV3-BC, PPO-BC, LAIfO, and various self-supervised BC variants) across 32 tasks in three simulated environments (Franka Kitchen, Meta-World, Robosuite) and two real-world robot platforms (7bot and PX100).

• Simulation Performance:
  • MYOE achieved significantly higher success rates (SR) than all baselines across most tasks.
  • In tasks with sparse rewards or suboptimal expert data (e.g., "M Button Press Wall"), baseline methods often failed completely (SR $\approx$ 0), while MYOE succeeded.
  • MYOE converged faster, demonstrating superior sample efficiency.
• Real-World Performance:
  • Experiments on physical robots (7bot and PX100) with only 5 demonstration episodes (10 steps each) showed MYOE outperforming baselines in "Reach" and "Block Picking" tasks.
  • MYOE successfully handled noise and cascading errors induced in the real-world setup, whereas baselines suffered from policy divergence.
• Ablation on Preference Regret: Theoretical analysis (Lemma 1) and empirical results confirm that the regret mechanism allows the agent to ignore suboptimal expert demonstrations and maximize true environmental rewards.

5. Significance

This work represents a significant advancement in Neurorobotics and Sample-Efficient RL.

• Practicality: It solves the "data scarcity" problem, making RLfD viable for real-world deployment where collecting thousands of expert demonstrations is impossible.
• Robustness: By moving beyond simple behavior cloning to a "preference-regret" based optimization, the system is robust to noisy, suboptimal, or limited expert data.
• Theoretical Integration: It successfully bridges Active Inference (free energy principle) with modern Deep Reinforcement Learning, providing a biologically plausible mechanism for how agents might learn from limited experience by balancing internal beliefs (preferences) with external reality (rewards).

The paper concludes that while MYOE excels in most scenarios, future work should focus on improving the model's ability to handle extremely fine-grained visual details (e.g., small objects) and utilizing more advanced autoregressive models for better temporal dependency extraction.