Technical Summary: Freeform Preference Learning for Robotic Manipulation

Problem Statement

Reward design remains a critical bottleneck for autonomous robot policy improvement, particularly in long-horizon manipulation tasks. Existing approaches face significant limitations:

• Sparse Rewards: Binary success/failure labels provide insufficient signal for complex tasks where intermediate progress is crucial.
• Ambiguous Binary Preferences: Traditional preference learning (e.g., Bradley-Terry models) requires annotators to collapse multiple competing notions of quality (e.g., speed vs. safety) into a single "overall" binary label. This creates ambiguity, especially when comparing trajectory segments from different task stages.
• Rigid Reward Functions: Hand-designed scalar rewards often fail to capture the full spectrum of human intent, such as hygiene, formality, or carefulness, leading to unwanted downstream behaviors when optimized.

The paper posits that preserving the multi-dimensional nature of human judgment is essential for learning robust policies that capture all aspects of a user's intent.

Methodology: Freeform Preference Learning (FPL)

FPL introduces a framework for learning robot policies from freeform human preferences defined in natural language, rather than collapsing feedback into a single scalar. The method consists of three core components:

1. Freeform Preference Collection

Instead of asking "Which trajectory is better overall?", annotators are shown pairs of trajectories and asked to:

• Define specific preference axes in natural language (e.g., "speed," "safety," "formality of setup," "smoothness"). These axes can be predefined or specified by the annotator during the process.
• Provide pairwise preferences ($y_k \in \{0, 1\}$) for each defined axis ($l_k$).
  This yields a dataset $\mathcal{P}$ of trajectory pairs annotated with natural language labels and per-axis binary preferences.

2. Language-Conditioned Reward Learning

FPL learns a single reward model $r_\phi$ conditioned on the natural language description of the axis, leveraging pretrained vision-language model (VLM) representations.

• Axis-Conditioned Scoring: For a trajectory pair $(\tau_i, \tau_j)$ and a set of axes $\{(l_k, y_k)\}$, the model is trained to minimize a modified Bradley-Terry loss:
  $$L_{FPL-Reward}(\phi) = -\mathbb{E} \left[ \sum_{k=1}^{K_{ij}} \log \sigma \left( (2y_k - 1) [r_\phi(\tau_i | l_k) - r_\phi(\tau_j | l_k)] \right) \right]$$
• Temporal Modeling: Unlike standard models that score individual observations, FPL conditions the reward on the observation history. The trajectory-level reward is defined as the sum of per-prefix scores:
  $$r_\phi(\tau, l_k) = \sum_{i=1}^{T} g_\phi(o_{1:i}, l_k)$$
  where $g_\phi$ is a multimodal transformer. This allows the model to capture temporal dependencies necessary for evaluating trajectory-level preferences.

3. Multi-Dimensional Policy Extraction

Rather than collapsing the learned multi-dimensional rewards into a single weighted scalar (which risks reward hacking), FPL trains a policy $\pi_\theta$ conditioned on the natural language axes and their corresponding reward scores simultaneously.

• The policy is trained via behavior cloning (or offline RL) on a dataset where actions are conditioned on the set of axes $\mathcal{L} = \{l_1, \dots, l_{K_\pi}\}$ and their associated rewards $r_\phi(\tau | l_k)$.
• Iterative Training: The process can be iterative. As the policy improves, annotators can provide more refined feedback, and the set of axes can expand, creating a natural curriculum.
• Test-Time Steerability: At inference, the policy can be steered toward specific behaviors by varying the target reward values conditioned on specific axes, without retraining.

Key Contributions

1. Freeform Preference Learning (FPL): A novel method for eliciting and learning from multi-dimensional, natural-language human preferences, avoiding the ambiguity of "overall" binary labels.
2. Language-Conditioned Reward Model: A reward function that maps a trajectory and a natural language axis description to a scalar score, enabling the learning of diverse attributes (quality, speed, safety, hygiene) from a single model.
3. Compositionality and Steerability: The framework enables policies to exhibit compositional behaviors (combining traits not seen together in data) and allows for test-time steering of behavior by adjusting reward conditioning.
4. Dense Supervision without Segmentation: FPL learns dense progress signals for long-horizon tasks without requiring explicit subtask segmentation or manual reward shaping.

Experimental Results

The authors evaluated FPL on four real-world tasks (putting a cube in a bowl, folding shorts, plating toast, setting a table) and two simulated tasks (object rearrangement, bimodal square).

• Performance: FPL outperformed baselines (including sparse rewards, binary preference learning, and weighted regression) by an average of 38 percentage points across real-world tasks.
• Simulation: In the "Bimodal Square" task, FPL successfully composed behaviors (placing a nut on a peg quickly) even when the training data only contained fast examples for the other peg. It also achieved high performance on the "inverted" task (left peg) using the same policy by changing the reward conditioning, a capability not present in baseline methods.
• Reward Density: Qualitative analysis showed that FPL's axis-conditioned rewards were temporally localized around specific subtask events (e.g., placing a cup), whereas binary preference rewards produced ambiguous, late-stage spikes.
• Annotation Efficiency: Collecting freeform preferences was approximately 50% faster per label than collecting single binary preferences, as the cost of viewing a video is shared across multiple axis-specific judgments.

Significance and Claims

The paper claims that FPL addresses the fundamental challenge of capturing the "multidimensional" nature of human intent in robotics. By preserving the structure of human feedback rather than collapsing it, FPL:

• Provides denser, less ambiguous supervision for policy optimization.
• Enables compositional generalization, allowing robots to combine learned behaviors (e.g., speed + safety) in ways not explicitly demonstrated.
• Offers test-time steerability, allowing users to adjust robot behavior (e.g., prioritizing speed over delicacy) without retraining.
• Demonstrates that natural language is a viable and effective interface for defining reward axes, moving beyond rigid, predefined reward structures.

The authors conclude that while preference learning is more expensive than unsupervised approaches, FPL significantly improves the quality and flexibility of learned policies, making it a promising direction for real-world robot learning.