🤖 machine learning

Trajectory First: A Curriculum for Discovering Diverse Policies

This paper proposes "Trajectory First," a two-stage curriculum learning framework that leverages a spline-based trajectory prior to generate diverse, high-reward behaviors and subsequently distills them into reactive policies, thereby overcoming the limited exploration and behavioral diversity issues found in existing constrained-diversity reinforcement learning methods for complex tasks like robot manipulation.

Original authors: Cornelius V. Braun, Sayantan Auddy, Marc Toussaint

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cornelius V. Braun, Sayantan Auddy, Marc Toussaint

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to push a heavy box across a room. Most standard robot trainers teach the robot to find one perfect way to do it: "Push from the left, slide it forward, done." If the floor gets slippery or the box shifts, that single strategy might fail, and the robot gets stuck.

This paper argues that a smart robot should learn many different ways to solve the same problem. Maybe one strategy is to push from the left, another from the right, and a third to lift and carry it. Having a "toolbox" of strategies makes the robot much more robust.

However, teaching a robot to be diverse is incredibly hard. If you just tell a robot, "Be different," it often gets confused. It might try to be different by waving its arms in the air (which is different but useless) or it might get stuck trying to find a new way and accidentally crash into things.

The authors propose a new training method called "Trajectory First." Think of it as a two-step curriculum, like a master chef training a new apprentice.

Step 1: The "Flight Simulator" Phase (Exploration)

Instead of teaching the robot to react step-by-step immediately, the authors first take the robot out of the "real world" and put it in a flight simulator where it can plan entire movements at once.

The Analogy: Imagine you are trying to find the best route through a giant, foggy maze. If you just take one step at a time and look around (the usual way), you might get stuck in a dead end.
The Paper's Trick: The authors use a mathematical tool called a B-spline. Think of this as drawing a smooth, flexible wire through the maze. The robot doesn't move step-by-step; it just adjusts the shape of the wire.
The Goal: They use a "search engine" (an evolutionary strategy) to wiggle these wires around until they find many different paths that successfully get the robot to the goal. They aren't looking for the perfect path yet; they just want a big pile of different successful paths.
Why it works: Because they are moving the whole "wire" at once, the robot can jump from one side of the maze to the other instantly, discovering routes that a step-by-step robot would never find because it would be too scared to take the first step.

Step 2: The "Real World" Phase (Distillation)

Now that the robot has a library of successful, diverse "wires" (trajectories), it's time to teach it how to drive in real life.

The Analogy: You take all those perfect flight simulator routes and teach the apprentice how to drive the car reactively. The apprentice learns: "If I see the wall on the left, I turn right. If I see the wall on the right, I turn left."
The Challenge: Usually, when you switch from a simulator to real life, the robot forgets its diversity and reverts to just one safe, boring way of doing things.
The Paper's Fix: They introduce three "stabilizers" to keep the robot diverse:
1. Symmetric Sampling: They mix the old simulator data with new real-world data 50/50. It's like telling the apprentice, "Remember the cool routes we found in the sim? Keep practicing those while you learn the new road."
2. The "Best So Far" Trick: They constantly update the definition of "success." Instead of saying "You must be perfect," they say, "You must be at least as good as the best thing we've seen so far." This prevents the robot from getting too greedy too early and crushing its own diversity.
3. Fast Updates: They make the robot learn from its mistakes much faster than usual. Since the definition of "diverse" changes as the robot learns, the robot needs to update its brain quickly to keep up with the shifting goal.

The Results

The authors tested this on robots that had to push cubes, press buttons, and navigate mazes.

The Old Way: The robots found one or two ways to do the task, often getting stuck in the same "local optimum" (the same safe, boring solution).
The New Way: The robots discovered a wide variety of creative solutions. For the button-pressing task, some robots used their elbow, others used their wrist, and others used their whole body to press the button. They all got the job done, but they did it in totally different ways.

In Summary

The paper claims that to get a robot to be truly diverse and robust, you can't just ask it to "be different" while it learns step-by-step. You first need to let it dream up many different successful plans in a safe, flexible space (the "Trajectory First" phase), and then carefully teach it how to execute those plans in the real world without losing its creativity.

This approach solves the problem of robots getting stuck in local loops and ensures they have a rich "toolbox" of strategies to handle unexpected changes in their environment.

Technical Summary: Trajectory First: A Curriculum for Discovering Diverse Policies

Problem Statement

Reinforcement learning (RL) methods often assume unimodal action distributions, producing a single policy that may be brittle to environmental perturbations and prone to getting trapped in local optima. While discovering a set of diverse policies maximizes robustness, existing constrained diversity optimization methods frequently fail in complex, contact-rich tasks like robot manipulation.

The core challenge identified is under-exploration. In domains such as robot manipulation, object states are mediated by intermittent contacts, creating non-smooth mappings between actions and states dominated by contact discontinuities. Consequently, diversity-seeking objectives often collapse to a few modes because alternative strategies are too rarely discovered to be reinforced. Standard local exploration methods (e.g., Gaussian noise in policy space) are insufficient to bridge disconnected high-reward manifolds in these environments.

Methodology: A Two-Stage Curriculum

The authors propose "Trajectory First," a curriculum designed to overcome exploration bottlenecks by decoupling global trajectory exploration from reactive policy learning. The approach consists of two distinct stages:

Stage 1: Constrained Novelty Search (CNS) in Trajectory Space

Instead of optimizing neural network parameters directly, the first stage explores the space of open-loop trajectories using an Evolutionary Strategy (ES).

Spline-Based Parameterization: Trajectories $\tau$ are parameterized as B-splines defined by control points $\omega$ . This provides a smooth inductive bias and reduces the dimensionality of the search space compared to raw action sequences.
Optimization Objective: The goal is to find a set of trajectory parameters $\Omega_n$ that maximizes diversity subject to a near-optimality constraint ( $v_{\omega_i} \geq \alpha v^*$ ).
Constrained Novelty Search: The authors introduce a Lagrangian objective that balances an intrinsic novelty reward (based on the entropy of the trajectory set, estimated via k-NN distance) and the extrinsic task reward.
Optimization Algorithm: The control points $\omega$ are optimized using CMA-ES, which is highly efficient for the moderately sized parameter space of B-splines. This allows the algorithm to "jump" between modes and discover diverse, high-reward anchors that local policy-space search would miss.

Stage 2: Distillation into Reactive Policies

The diverse trajectories discovered in Stage 1 are distilled into a set of $n$ reactive, skill-conditioned policies $\Pi_n$ using constrained off-policy learning.

Framework: The authors adopt the Domino framework, treating the gradient of the diversity objective as an intrinsic reward.
Stabilization Techniques: To ensure stable distillation from offline trajectory data to online reactive policies in contact-rich tasks, three critical modifications are introduced:
1. Symmetric Sampling: Training batches are composed of 50% online data and 50% filtered CNS trajectories (selected from the top 25% of returns). This prevents diversity collapse during the transition to online RL.
2. The $v_{max}$ -trick: Instead of relying on a single expert policy to estimate the optimal value $v^*$ , the authors update $v^*$ as the maximum performance achieved across all skills and the entire training history ( $v^* \leftarrow \max\{v^*, \max_i v_i\}$ ). This provides a monotonically increasing, stable target for the suboptimality constraint, preventing the diversity objective from being suffocated by premature optimality pressure.
3. High Policy Update-to-Data (UTD): Unlike standard offline-to-online RL where the critic is updated frequently and the policy infrequently, this method increases the policy update frequency (e.g., 4 updates per environment step). This is necessary because the intrinsic reward depends on the state occupancy of the current policy set; frequent policy updates allow the agent to keep pace with the shifting intrinsic landscape.

Key Contributions

Empirical Identification of Exploration Bottlenecks: The work demonstrates that existing constrained diversity optimization methods (e.g., Domino) fail to discover diverse policies in robot manipulation tasks due to the inability of local exploration to bridge disconnected high-reward regions.
Trajectory-First Curriculum: A novel two-stage approach that uses evolutionary search over spline-based trajectories to generate diverse high-reward anchors, which are then distilled into reactive policies.
Stabilization Recipe: A set of techniques (Symmetric Sampling, $v_{max}$ -trick, High Policy UTD) that enable stable offline-to-online learning for diversity objectives, addressing the non-stationarity of intrinsic rewards in constrained settings.

Experimental Results

The method was evaluated on four robotic tasks: Maze Navigation, Cube Pushing, Button Pressing, and Cube Flipping.

Diversity vs. Performance: The proposed curriculum achieved a 1.5x increase in behavioral diversity compared to the strongest baseline (Domino) while maintaining comparable task performance.
Task-Specific Findings:
- In Maze Navigation, the method discovered the highest number of distinct paths, whereas baselines tended to cluster around safe, collision-free routes.
- In Button Pressing, baselines often collapsed to using the same robot link for all skills. The proposed method successfully discovered whole-body manipulation strategies using different robot links.
- In Cube Flipping, the method found diverse contact modes and force profiles, whereas noise-based exploration baselines failed due to frequent collisions and negative rewards.
Robustness: The method remained robust under adversarial reward specifications (e.g., high collision penalties) where baselines failed to interact with objects entirely, as the CNS phase provided a warmstart within high-reward regions.

Significance and Claims

The paper claims that the primary bottleneck in constrained diversity optimization for robotics is not the learning objective itself, but the reliance on local exploration. By introducing an inductive bias through spline-based trajectory priors, the authors enable the discovery of diverse contact modes that are otherwise inaccessible to standard policy-space search.

The authors position this work as a solution to the "exploration-exploitation" trade-off in diversity optimization. They argue that while prior work assumes diverse high-reward behaviors are easily reachable under online policy optimization, this assumption fails in complex manipulation tasks. The proposed curriculum bridges global trajectory-level exploration and reactive control, offering a practical recipe for training robust, diverse agents. The authors modestly note limitations, such as the reliance on manually defined state features for diversity measurement and the fixed transition between stages, suggesting future work could explore adaptive switching and representation learning from raw observations.