💻 computer science

Combined Constrained Sampling and Reinforcement Learning for Robotic Manipulation

This paper proposes a novel approach that combines constrained state sampling with reinforcement learning to effectively train universal, non-prehensile robotic manipulation policies in complex, contact-rich environments by leveraging structured reset strategies and curriculum learning to enhance exploration.

Original authors: Marc Toussaint, Cornelius V. Braun, Armand Jordana, Sayantan Auddy, Eckart Cobo-Briesewitz, Denis Shcherba, Tilman Burghoff, Justin Carpentier

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marc Toussaint, Cornelius V. Braun, Armand Jordana, Sayantan Auddy, Eckart Cobo-Briesewitz, Denis Shcherba, Tilman Burghoff, Justin Carpentier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot arm how to push, slide, and balance a ball or a cube on a table without ever picking it up. This is called "non-prehensile manipulation." It's incredibly hard because the physics of sliding and bouncing are messy, unpredictable, and full of sudden changes (like when a ball hits a wall and bounces back).

The paper introduces a new way to train robots for these tasks by combining two main ideas: Reinforcement Learning (RL) and Constrained Sampling. Here is the breakdown using simple analogies.

The Problem: The Robot is Lost in a Dark Forest

Think of Reinforcement Learning as a robot learning by trial and error. It tries random moves, gets a reward if it succeeds, and learns from its mistakes.

The Issue: In complex tasks (like balancing a cube on its corner), the robot has to find a very specific, rare sequence of moves to succeed. If the robot starts in a random position every time, it might spend years bumping into walls before it ever accidentally finds the "magic spot" where the cube balances. It's like trying to find a specific needle in a haystack by throwing darts blindly.

The Solution: A Smart Map and a Guided Tour

The authors propose a new system called CSRL (Combined Constrained Sampling and Reinforcement Learning). They solve the "lost in the forest" problem with three tricks:

1. The "Physics-Informed" Map (Constrained Sampling)

Instead of letting the robot start in a completely random, chaotic mess, the authors built a special "sampler."

The Analogy: Imagine you want to teach a student how to balance a broom on their hand. You wouldn't start by throwing the broom in the air and hoping it lands in their hand. Instead, you would carefully place the broom in a position where it could be balanced, even if it's not perfectly stable yet.
How it works: The sampler uses math to understand the rules of physics (like friction and gravity). It generates start and goal positions that are physically possible (e.g., the ball is touching the table, not floating in mid-air) and covers a wide variety of interesting contact scenarios (like the ball touching the wall, the floor, or the robot's arm). This ensures the robot always starts in a "teachable" moment.

2. The "Training Wheels" Approach (Curriculum Learning)

Once the robot has a list of valid starting positions, the authors don't throw it into the deep end immediately.

The Analogy: Think of learning to swim. You don't start by jumping into the deep end of the ocean. You start in the shallow end, then the middle, then the deep end.
How it works: The system starts by training the robot to reach goals that are very close to where it starts. As the robot gets better, the system slowly increases the distance between the start and the goal. This is called a "curriculum." It guides the robot from easy tasks to hard ones, preventing it from getting frustrated or stuck.

3. The "Bridge Builder" (Projected Interpolation)

Sometimes, even with a good map, the jump from "Start" to "Goal" is too big.

The Analogy: If you want to walk from your house to a friend's house across a wide river, you can't just jump. You need a bridge.
How it works: The system takes the "Start" and the "Goal" and draws a straight line between them in the robot's mind. However, since a straight line might go through a wall (which is impossible), the system "projects" that line onto the nearest valid, physical path. It creates a series of stepping stones (intermediate goals) that the robot can actually walk on, making the journey much easier to learn.

What Did They Test?

The team tested this method on four different scenarios, ranging from simple to very hard:

Double-Sphere: A robot pushing a ball against a wall.
Panda-Sphere: A complex robot arm (like a Panda robot) using its whole body to scoop and hold a ball.
Sphere-Cube: Balancing a cube on its edge or corner.
Panda-Cube: The complex robot arm balancing the cube.

The Results

Random vs. Smart: When they let the robot start randomly, it failed almost completely on the hard tasks. When they used their "Smart Map" (Constrained Sampling), the robot learned successfully.
Speed: The robot learned much faster with the "Training Wheels" (Curriculum) and "Bridge Builder" (Interpolation) methods.
Real World: They even tested a version of this on a real robot (using a camera to track a ball), and the skills learned in the simulation transferred well to the real world.

The Bottom Line

The paper argues that to teach robots complex physical skills, we shouldn't just let them "guess" randomly. Instead, we should use math to generate smart, physically valid starting points and guide the robot through a step-by-step learning path. This combination allows robots to master difficult tasks like balancing and pushing that were previously too hard for standard AI methods.

Technical Summary: Combined Constrained Sampling and Reinforcement Learning for Robotic Manipulation

Problem Statement

Training non-prehensile manipulation policies in contact-rich environments remains a core challenge in robotics. While Reinforcement Learning (RL) has demonstrated strength in handling contact-rich dynamics by implicitly smoothing underlying discontinuities, it struggles with severe exploration problems. Relevant interaction modes often require reaching rare contact states before receiving informative rewards.

A critical bottleneck lies in the design of reset and goal distributions ( $p(s_0)$ and $p(g)$ ). Existing strategies often rely on heuristics, random scene generation, or task-specific samplers (e.g., grasp-specific). These methods fail to provide a general mechanism to specify the diverse contact configurations necessary for non-prehensile tasks, such as balancing objects, where the state space is constrained by complex contact dynamics and complementarity structures.

Methodology

The authors propose CSRL (Combined Constrained Sampling and Reinforcement Learning), a framework that integrates a physics-informed state sampler with goal-conditioned RL. The approach consists of three main components:

1. Constrained State Sampler

Unlike standard physics simulators that sample dynamics forward, this sampler generates valid start and goal states by solving a non-linear constraint problem. It explicitly models the structure of contact to ensure diverse coverage of contact modes.

Formulation: The sampler defines a constraint space $S_c$ using rigid-body dynamics, collision constraints, and contact force constraints (Coulomb friction cones). It assumes goals can be held in quasi-static equilibrium.
Variables: The decision variables include discrete contact modes ( $c_{ij} \in \{0, 1\}$ ) and continuous state variables ( $s$ ), along with points of attack and linear forces for active contacts.
Sampling Strategy: A hierarchical approach is used. First, a contact mode configuration is sampled (e.g., selecting 1, 2, or 3 support contacts). Then, an Augmented Lagrangian method solves a proximal optimization problem to project a random box-constrained sample $\bar{s}$ onto the feasible manifold $S_c$ . This process is repeated until a dataset $D$ of feasible states is generated.

2. Reset and Curriculum Strategies

To address the exploration challenge in high-dimensional spaces where random starts and goals are likely distant, CSRL employs two additional strategies:

Projected Interpolation: Instead of resetting to random states, the system samples a start $a$ and goal $g$ from the constrained dataset $D$ . It computes a convex interpolation between them in an embedded state space $\phi$ , then projects this interpolated point back onto the constrained manifold $S_c$ (approximated by finding the nearest neighbor in $D$ ). This ensures intermediate states remain physically valid.
Curriculum Learning: A phase variable $\alpha$ is scheduled from near 0 to 1. Initially, interpolation uses $t \sim U[0, \alpha]$ , resetting the agent closer to the goal. The MDP truncation time is also scaled with $\alpha$ , allowing the agent to collect short rollouts near the goal before tackling longer, more complex trajectories.

3. Integration with RL

The constrained sampler provides the distribution $p_0$ for a goal-conditioned MDP. The RL agent (using TD7 as the default, with evaluations of TD3 and HER variants) learns a universal policy $\pi: s, g \mapsto a$ to connect any start state to any goal state within the distribution.

Key Contributions

Novel Constrained State Sampler: A method to generate reset and goal distributions for contact-rich manipulation that explicitly accounts for contact structure, providing a rich covering of diverse contact modes without committing to specific interaction primitives.
Projected Interpolation and Curriculum: A strategy to bridge the gap between sampled states using constraint-projected interpolation and a curriculum that gradually increases task difficulty, mitigating sparse reward challenges.
Experimental Validation: Demonstration that this approach outperforms standard RL with random resets and alternative reset methods across four domains of increasing difficulty.

Experimental Results

The authors evaluated CSRL on four domains:

Double-sphere: A basic test for universal manipulation between quasi-static states.
Panda-sphere: Tests whole-body contact skills using a Panda arm.
Sphere-cube: Tests rotation and orientation control, including balancing on edges/corners.
Panda-cube: Tests controlling a cube with end-effector and finger contacts.

Key Findings:

Superiority of Constrained Sampling: On the double-sphere and panda-sphere domains, using constrained sampling for both starts and goals significantly outperformed random sampling. Random sampling performed particularly poorly on the panda-sphere task, failing to generate valid static holding states.
Impact of Interpolation and Curriculum: Adding projected interpolation and curriculum learning further improved success rates and convergence speed across all domains and RL algorithms (including HER variants).
Comparison to Reverse Curriculum: The proposed interpolation strategy outperformed reverse curriculum generation (which simulates backward from goals), particularly in terms of computational efficiency and success rates.
Real-World Transfer: A sphere-cube policy trained in simulation was successfully transferred to a real-world setup using an end-effector-attached ball and OptiTrack tracking, demonstrating the potential for sim-to-real transfer of fundamental manipulation skills.

Significance and Claims

The paper claims that by complementing physics-based dynamics sampling with a physics-informed constrained state sampler, it is possible to train universal, non-prehensile, and dynamic manipulation policies in contact-rich settings. The approach effectively addresses the exploration bottleneck in RL by ensuring the agent is exposed to diverse, physically valid contact configurations from the outset.

The authors position this work as a step toward generalist non-prehensile manipulation. They note that while the current approach targets "privileged training" (where the policy has full state access), it lays the groundwork for future teacher-student paradigms. The paper acknowledges limitations regarding computational costs for complex objects due to the use of a single-CPU NLP solver and the challenges of sensor-based transfer, suggesting future work in parallelization and dedicated solvers.