Technical Summary: Feedback Manipulation Regularization (FMR)

Problem Statement

Reinforcement Learning (RL) faces a dual challenge: aptitude (learning to solve a task) and alignment (learning to solve it in accordance with human values). While human demonstrations and feedback are critical for alignment, existing approaches predominantly rely on multi-stage pipelines designed for the contextual bandit framing of language generation (e.g., RLHF). These methods struggle to translate to fully sequential decision-making environments.

Furthermore, current preference-based learning methods suffer from fundamental limitations in sequential settings:

1. Relational Ambiguity: Preferences are relative ("which is better?") rather than absolute ("is this good?"). A preferred behavior is not necessarily an aligned one, especially if both options in a comparison pair are misaligned.
2. Sampling Limitations: Randomly sampled comparison pairs make it difficult to target specific behaviors without explicit curation.
3. Data Scarcity: In offline settings with limited data, particularly where expert demonstrations are scarce and imperfect demonstrations contain noisy or misaligned behaviors, existing algorithms often fail to maintain alignment.

The paper addresses the need for a single-stage, offline method that effectively combines demonstrations and evaluative feedback to align policies in fully sequential environments, even under limited data regimes.

Methodology: Feedback Manipulation Regularization (FMR)

The authors propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic framework that integrates evaluative feedback into imitation learning. Unlike methods that treat feedback as a reward signal or a preference score, FMR uses feedback to directly manipulate the policy's probability distribution.

Core Mechanism

FMR operates by treating human feedback $h(s, a) \in \{-1, 0, 1\}$ as a corrective signal that reshapes the learned policy $\pi_\theta$ toward a latent human-aligned policy $\pi_h$.

1. Feedback-Adaptive Temperature:
   The method introduces a temperature parameter $\tau(s, a)$ derived from feedback.

   • Negative feedback ($h < 0$) implies $\tau > 1$, discouraging the selected action.
   • Positive feedback ($h > 0$) implies $\tau < 1$ for non-selected actions (effectively encouraging the selected action relative to others).
   • No feedback ($h = 0$) results in $\tau = 1$, having no impact.
     The temperature is calculated via an exponential map: $\tau_j = \beta^{-h}$ (for negative feedback on the selected action) or $\beta^h$ (for positive feedback on non-selected actions), where $\beta > 1$.

2. Policy Reshaping:
   The human-aligned policy is approximated by applying temperature scaling to the current policy in probability space:
   $$\pi_\tau(a_j | s) := \frac{\pi_\theta(a_j | s) \tau_j^{-1}}{Z_\theta(s)}$$
   where $Z_\theta(s)$ is the normalization constant. This operation redistributes probability mass, suppressing misaligned behaviors and encouraging aligned ones without requiring explicit reward modeling.

3. Regularization Objective:
   FMR formulates alignment as a distribution matching problem using the reverse Kullback-Leibler (KL) divergence between the learned policy $\pi_\theta$ and the feedback-tempered surrogate $\pi_\tau$. This yields a regularization term:
   $$R_{FMR}(\theta) := \mathbb{E}_{s \sim D_U} \mathbb{E}_{a \sim \pi_\theta(\cdot|s)} [\log \tau(s, a)]$$
   The final objective combines the standard imitation learning loss ($L_{IL}$) with this regularizer:
   $$L(\theta) = L_{IL}(\theta) + \alpha R_{FMR}(\theta)$$
   where $\alpha$ controls the strength of alignment.

Data Handling

The method is designed for Learning from Noisy Demonstrations (LfND), handling datasets $D_U = D_E \cup D_I$ where $D_E$ contains aligned expert data and $D_I$ contains imperfect (noisy) data. FMR leverages evaluative feedback to distinguish and suppress misaligned sub-trajectories within $D_I$.

Key Contributions

1. Algorithm-Agnostic Framework: FMR can be applied to any imitation learning algorithm (e.g., Behavioral Cloning, IQL, DemoDICE, ReCOIL) as a regularization term.
2. Single-Stage Offline Training: Unlike multi-stage RLHF pipelines, FMR operates in a single stage, making it suitable for fully sequential decision-making environments where environment access is restricted.
3. Robustness in Limited Data: The method demonstrates robustness even when the ratio of expert to imperfect data is low (e.g., 1:50) and when imperfect data contains little to no helpful behavior.
4. Principled Evaluation: The authors adapt Safety Gymnasium environments to serve as a testbed, using cost signals to provide a principled measure of misalignment distinct from total return.

Experimental Results

The authors evaluated FMR across 3D navigation and velocity-restricted locomotion tasks (Hopper, Swimmer, Walker2D) using various imitation learning baselines.

• Alignment Improvement: FMR consistently reduced misalignment across all tested algorithms. In the PathM navigation task, FMR achieved up to a 92% reduction in misalignment for ReCOIL. In velocity tasks, reductions of up to 98% were observed.
• Aptitude Preservation: Crucially, the reduction in misalignment did not come at the cost of task performance (aptitude). FMR maintained or improved success rates and normalized returns compared to baselines.
• Comparison to Alternatives:
  • vs. DVL (Dual Value Learning): DVL, which substitutes feedback for reward, performed poorly, confirming that naively interpreting feedback as a reward signal is ineffective.
  • vs. CPL (Contrastive Preference Learning): CPL, which uses feedback for pairwise scoring, degraded significantly as data ratios decreased, whereas FMR remained stable.
• Data Overlap: FMR performed best when imperfect demonstrations contained some relevant sub-trajectories (high overlap). However, it still provided significant improvements in low-overlap scenarios where baselines failed completely.
• Feedback Efficiency: Experiments showed that FMR learns effectively from negative feedback alone, which serves as the primary driver for correcting deviations.

Significance and Claims

The paper claims that FMR represents a significant step forward in offline agent alignment by:

1. Bridging Modalities: Successfully integrating evaluative feedback and demonstrations as an interconnected signal for sequential decision-making, moving beyond the limitations of preference-based relative judgments.
2. Addressing the "Alignment-Aptitude" Trade-off: Demonstrating that it is possible to drastically reduce misalignment (up to 98%) without sacrificing the agent's ability to complete the task.
3. Handling Data Scarcity: Providing a viable solution for scenarios where high-quality aligned data is scarce and noisy data is abundant, a common real-world constraint.
4. Modularity: Offering a plug-and-play regularization term that enhances existing imitation learning algorithms without requiring architectural changes or multi-stage training pipelines.

The authors conclude that FMR effectively harnesses the rich information in feedback to modulate policy entropy, discouraging undesirable actions and reinforcing preferred behaviors, thereby enabling more robust and aligned offline learning.