Warm-Starting Collision-Free Model Predictive Control With Object-Centric Diffusion

Technical Summary: Warm-Starting Collision-Free Model Predictive Control With Object-Centric Diffusion

Problem Statement
Robotic manipulation in cluttered, non-convex environments requires generating motions that are simultaneously collision-free and dynamically feasible. While classical trajectory optimization and Model Predictive Control (MPC) can enforce physical constraints, they are highly sensitive to initialization and frequently converge to local minima in complex scenes. Conversely, learning-based approaches, particularly diffusion models, excel at capturing the multi-modal distribution of feasible trajectories but historically lack a general, efficient mechanism to condition on scene structure. Furthermore, prior diffusion-based methods often generate open-loop trajectories that fail to strictly enforce dynamic feasibility or safety constraints when deployed on physical hardware.

Methodology
The authors propose a hybrid, two-stage framework that combines a conditional diffusion model for trajectory generation with a constraint-based MPC solver for refinement. The architecture consists of three integrated components:

1. Object-Centric Scene Representation: To condition the diffusion model on the environment, the paper introduces a Slot Attention-based encoder. This module processes single-view RGB images of the workspace to extract a compact set of latent "slots," each representing a distinct object or obstacle. This object-centric representation provides a permutation-invariant, low-dimensional conditioning signal that captures essential collision geometry without the high dimensionality of raw point clouds or images.
2. Conditional Diffusion Warm-Start: A Diffusion Transformer (DiT) is trained to denoise a trajectory sequence conditioned on the robot's initial state, the target end-effector position, and the latent slot representation of the scene. The model learns a prior over feasible motion sequences, generating a diverse set of trajectory proposals that navigate around obstacles.
3. MPC Refinement via Optimal Control: The trajectory sampled from the diffusion model serves as a "warm-start" for a receding-horizon MPC solver. The solver formulates an Optimal Control Problem (OCP) that minimizes a cost function comprising goal tracking, state regularization (penalizing deviation from the diffusion prior), and control effort. Crucially, the OCP enforces hard constraints, including rigid-body dynamics and signed-distance collision avoidance. This step projects the generative prior onto a dynamically feasible and strictly collision-free trajectory in real time.

Key Contributions

• Hybrid Architecture: The paper demonstrates that coupling object-centric diffusion priors with full-dynamics MPC yields reliable motion generation under strict time limits, overcoming the initialization sensitivity of pure optimization and the safety limitations of pure learning-based methods.
• Object-Centric Conditioning: The authors propose and validate a Slot Attention mechanism for scene encoding, showing it provides a more compact and generalizable conditioning signal compared to image-based, occupancy grid, or configuration-space representations.
• Real-Time Closed-Loop Control: Unlike prior works that generate open-loop trajectories tracked by low-level controllers, this approach performs closed-loop replanning on a physical robot, rigorously enforcing dynamic and safety constraints at 100 Hz.
• Systematic Data Generation: A pipeline is introduced to generate high-quality training data by combining sampling-based planning (RRT) with constrained trajectory optimization to produce smooth, feasible expert trajectories.

Experimental Results
The method was evaluated on a Franka Emika Panda robotic arm across three benchmark environments (table, drawer, and shelf) with varying obstacle configurations.

• Performance: The proposed method achieved success rates of 82%, 79%, and 83% across the three environments, significantly outperforming pure OCP baselines (<10% success) and other learning-based methods like Motion-Planning Diffusion (MPD). It also surpassed classical planners (RRT-Connect, cuRobo) and the PRESTO diffusion baseline in terms of trajectory cost and computation time.
• Latency: The system operates with a runtime of under 72 ms, making it suitable for real-time control.
• Ablation Studies: Experiments confirmed that the Slot Attention representation outperforms image-based and occupancy-grid conditioning, particularly in low-computation regimes. Additionally, the study found that cost-gradient guidance during the diffusion process offered diminishing returns when OCP refinement was present, leading to its omission in the final pipeline.
• Real-Robot Validation: Experiments on the physical robot confirmed the system's ability to reactively avoid obstacles detected via AprilTags and re-optimize motions when obstacles were removed dynamically.

Significance
The paper claims that this approach establishes a complete robotics pipeline capable of robust real-robot performance in cluttered, non-convex environments. By leveraging the global reasoning of diffusion models to guide the local optimization of MPC, the method achieves a balance of safety, efficiency, and success rate that neither component can achieve alone. The work highlights the practical viability of combining generative priors with model-based control for collision-aware manipulation, offering a solution that is both computationally efficient and strictly compliant with physical safety constraints.