💻 computer science

Diffusion Policy for Coordinated Control of a Nonholonomic Mobile Base and Dual Arms in Door Opening and Passing

This paper presents a diffusion-based visuomotor control policy that enables a nonholonomic mobile base with dual arms to robustly and coordinately execute the complex, long-horizon task of opening heavy self-closing doors and passing through them, outperforming traditional state-machine approaches in generalization and disturbance handling.

Original authors: Shangqun Yu, Matthew En, Daniel Wu, Sangjun Park, Ziyi Zhou, Seyed Fakoorian, Donghyun Kim

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shangqun Yu, Matthew En, Daniel Wu, Sangjun Park, Ziyi Zhou, Seyed Fakoorian, Donghyun Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to walk through a heavy, self-closing door that you have to pull open. For a human, this is easy: you grab the handle, twist it, pull the door wide, hold it open with one hand while stepping through, and maybe switch hands if you need to. But for a robot, this is a nightmare. It has to coordinate its wheels (which can't move sideways easily), its two arms, and its eyes, all while fighting a door that wants to slam shut on its own.

This paper presents a new "brain" for a robot that solves this problem using a method called a Diffusion Policy. Here is how it works, explained simply:

The Problem: The "State Machine" Struggle

Traditionally, programmers taught robots to do this by writing a strict "recipe" or flowchart (called a state machine). It would say: "Step 1: Twist knob. Step 2: Pull door. Step 3: Walk through."

The Flaw: If the door is heavier than expected, or the robot slips, or the handle is a different color, the robot gets confused. It's like following a recipe that says "add salt" but doesn't tell you what to do if you accidentally drop the salt shaker. The robot just freezes or crashes because the real world is messy and unpredictable.

The Solution: The "Diffusion" Artist

The authors replaced the strict recipe with a Diffusion Policy. Think of this not as a robot following a map, but as an artist learning to paint by watching a master.

Learning by Watching (Imitation): Instead of being told exactly what to do, the robot watched humans (or a computer simulation of a human) perform the task 100 times. It saw how they reached, twisted, pulled, and walked through.
The "Denoising" Process: Imagine the robot is looking at a very blurry, static-filled TV screen. The "Diffusion" part is like the robot slowly clearing up the static, step-by-step, to reveal a clear picture of what action to take next.
- It starts with a random guess (the static).
- It looks at what it sees (the door, the handle, its own arms).
- It "cleans up" the guess to make it smoother and more accurate.
- It repeats this cleaning process very quickly (100 times in training, but only 10 times when actually moving) to decide exactly how to move its arms and wheels.

Why This is Special

The paper highlights three main superpowers this robot gained:

The "Swiss Army Knife" Coordination: Most robots treat walking and grabbing as separate jobs. This robot learned to do them at the exact same time. It's like a juggler who learned to walk a tightrope while juggling, rather than learning to walk first and then learning to juggle. It coordinates its wheels and two arms seamlessly to pull the door and step through.
The "Recovery Reflex": This is the most impressive part. In the real world, things go wrong. The researchers tested this by manually pushing the door shut while the robot was trying to open it.
- Old Robot: Would likely panic, stop, or try to pull a door that is now closed, leading to a crash.
- This Robot: It noticed the door moved (using its eyes), realized its plan was wrong, and immediately "re-painted" its action. It stopped pulling, adjusted its grip, and started the opening motion all over again. It didn't need a human to hit the "reset" button; it fixed itself.
Adapting to Change: The robot was trained on doors of different colors and in different lighting. Because it learned the concept of opening a door rather than memorizing a specific door, it worked just as well on a red door as a blue one, or in bright sun as in dim light.

The Result

The team built a real robot with two arms (called a "RealMan" platform) and tested it.

In Simulation: The robot succeeded 10 out of 10 times, even when the door color changed or the lighting shifted.
In Real Life: The robot successfully opened heavy, self-closing doors and walked through them. When the researchers messed with the door mid-task, the robot recovered and finished the job.

The Bottom Line

This paper proves that you don't need to write a complex, brittle set of rules to teach a robot a difficult physical task. Instead, by using a "diffusion" method that learns from examples and constantly refines its actions like an artist cleaning up a sketch, a robot can learn to handle complex, real-world challenges—like opening a stubborn door—on its own, even when things go wrong.

Technical Summary: Diffusion Policy for Coordinated Control of a Nonholonomic Mobile Base and Dual Arms in Door Opening and Passing

Problem Statement
The paper addresses the complex challenge of enabling a dual-arm robot mounted on a nonholonomic mobile base to open heavy, self-closing (damped) pull doors and traverse through them. This task is characterized by long-horizon planning, tight coordination between locomotion and manipulation, and contact-rich interactions. Specifically, the robot must rotate a handle, widen the door gap, hold the door against a self-closing mechanism, switch arms as necessary, and navigate through the opening while maintaining clearance.

Traditional model-based approaches rely on state machines and hand-crafted trajectories that transition between manually defined stages (e.g., rotating then pulling). These methods lack robustness and fail to generalize to diverse real-world conditions without extensive engineering. While recent learning-based methods (behavior cloning, reinforcement learning) offer alternatives, they often depend on abstracted state representations, require multiple sub-modules, or are limited to simplified scenarios (e.g., undamped push doors). No existing visual-action model has successfully demonstrated simultaneous coordination of a nonholonomic base and dual arms for the complete, damped pull-door traversal task using only visual and proprioceptive inputs.

Methodology
The authors propose an end-to-end diffusion-based visuomotor control policy ( $\pi_\theta(a|s)$ ) that maps raw observations (RGB images and proprioceptive data) directly to continuous actions.

Data Collection:
- Hardware: Expert demonstrations were collected using a RealMan dual-arm platform equipped with a teleoperation kit, recording joint positions and multi-view RGB images.
- Simulation: A stage-based motion planner was developed to generate diverse reference trajectories. This system combines Inverse Kinematics (IK) for dual-arm manipulation with Model Predictive Control (MPC) for nonholonomic base motion. A task-space scheduler guides the robot through keyframes (reaching, twisting, pulling, holding).
- Randomization: To ensure robustness, the training data includes domain randomization of lighting conditions, door/handle colors, and the robot's initial base pose relative to the door.
Policy Architecture:
- Input: The policy conditions on a stack of the most recent observations ( $T_o = 3$ ), which include latent vectors from three independent ResNet-18 encoders (one per camera view) concatenated with proprioceptive state data.
- Backbone: A 1D U-Net with FiLM (Feature-wise Linear Modulation) conditioning serves as the diffusion backbone.
- Output: The model predicts a future action sequence ( $T_p = 16$ steps). During execution, the robot performs the first $T_a = 8$ steps before replanning based on new observations.
- Inference: The policy operates by denoising a Gaussian noise vector into an action sequence. While trained with 100 denoising steps, the authors found that inference could be performed with as few as 10 steps without performance loss, significantly reducing latency.
Optimization: To achieve real-time control on the onboard Jetson AGX Orin, the authors employed Low-Rank Adaptation (LoRA) with quantization and optimized the system's power configuration to prevent frequency throttling.

Key Contributions

Stage-Based Trajectory Generation: A pipeline combining IK and MPC with randomized test scenarios to automate the generation of diverse reference trajectories for training.
End-to-End Diffusion Controller: The development of a unified image-to-action coordination controller based on diffusion policies, balancing model complexity and inference time to handle long-horizon tasks.
Real-World Validation: Experimental validation on a physical dual-arm mobile manipulator, demonstrating the policy's ability to execute coordinated door-opening tasks and recover from external disturbances using only vision and proprioception.

Results

Simulation: The diffusion policy achieved a 100% success rate (10/10) in opening and traversing damped pull doors in simulation.
- Ablation Studies: Using separate encoders for each camera view was critical; a unified encoder reduced success to 0/10. Reducing denoising steps to 30 or even 10 maintained high performance, whereas further reduction degraded results.
- Baselines: The proposed diffusion policy outperformed baselines. ACT (Action Chunking with Transformers) achieved 8/10 success but failed to reach 100%, likely due to overfitting to dominant visual patterns in randomized environments. SmolVLA failed completely (0/10), attributed to its pretraining on smaller manipulator datasets.
Hardware Deployment: The policy successfully executed the full task sequence on the RealMan platform.
- Disturbance Recovery: A key finding was the policy's robustness to external disturbances. When the door was manually re-closed during the opening sequence, the robot detected the change via visual feedback, halted the current motion, re-adjusted its posture, and re-initiated the opening sequence rather than failing or diverging.
- Latency: Through model and hardware optimization, the system achieved low-latency execution sufficient for real-time base control and traversal.

Significance and Claims
The paper claims that diffusion-based policies offer a promising paradigm for complex real-world manipulation tasks that require tight coordination between perception, manipulation, and navigation. Unlike traditional methods that rely on brittle state machines or require extensive engineering for specific door types, this approach learns a unified policy directly from demonstrations. The authors highlight that the resulting system not only achieves high success rates in difficult, damped pull-door scenarios but also demonstrates a critical capability for real-world deployment: the ability to detect and recover from external disturbances autonomously. This resilience, achieved without explicit disturbance modeling, distinguishes the proposed method from both traditional model-based approaches and other learning-based frameworks that often lack recovery mechanisms once perturbed.