🤖 machine learning

ObstaDiff: Generalizable Diffusion Policy Learning via Obstacle-aware Representations

ObstaDiff is a generalizable diffusion policy framework that leverages obstacle-aware visual representations to enable robust robotic manipulation in cluttered, unstructured environments, achieving superior task success and reduced collision rates in real-world agricultural trials compared to existing baselines.

Original authors: Jiawen Wang, Kevin Yao, Khalid Jawed

Published 2026-09-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiawen Wang, Kevin Yao, Khalid Jawed

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long been masters of the factory floor, where they move with precision among fixed parts and predictable paths. But step outside that controlled world into a real garden, a cluttered workshop, or a busy home, and the task becomes far more difficult. In these unstructured environments, a robot arm must reach for a specific object, like a ripe fruit, while navigating a maze of leaves, stems, and tools that block its path. The challenge is not just seeing the object, but understanding the space around it well enough to move without crashing. For years, researchers have tried to teach robots to learn from human demonstrations, a method called imitation learning. However, most of these systems were trained in clean, empty settings and struggle when faced with the visual chaos of the real world. They often fail to distinguish between what they should touch and what they should avoid, leading to clumsy movements or collisions.

A team of researchers at the University of California, Los Angeles, has developed a new approach to help robots navigate these crowded spaces, which they call ObstaDiff. Instead of feeding the robot a standard video feed that mixes the target object, the obstacles, and the background into one confusing image, the system breaks the scene down into three distinct layers: the target, the obstacles, and the background. This separation allows the robot to clearly see which parts of the scene it needs to reach for and which parts it must steer clear of. By training a learning model on this structured view, the robot learns to generate smooth, safe paths that weave through narrow gaps in the foliage, rather than plowing straight into the nearest leaf.

The researchers tested this system in a real-world greenhouse setting, a particularly demanding environment filled with dense plant growth and changing light. They set up a task where a robotic arm had to reach a specific pepper plant hidden among other plants. To see if the system could truly generalize, they did not just repeat the same movement over and over. Instead, they ran hundreds of trials where they changed the position of the target pepper, rearranged the surrounding obstacle plants, and even swapped the training pepper for a different variety of green pepper that the robot had never seen before. Across 366 real-world executions, the new system succeeded in completing the task 75.41% of the time. In comparison, other leading methods that did not use this specialized way of seeing the world succeeded less than half the time. Perhaps more importantly, the new system crashed into obstacles only 8.20% of the time, a significant improvement over the much higher collision rates of the other methods.

The key to this success lies in how the robot processes what it sees. Traditional systems often treat the camera image as a single, flat picture, forcing the robot to figure out the difference between a leaf and a pepper on its own. The new system, however, uses a lightweight encoder that explicitly labels the image channels as target, obstacle, and background before the robot even begins to plan its move. This gives the robot a clear map of the situation: here is the goal, here are the walls, and here is the empty space. The robot then uses a learning process that generates a sequence of movements to guide its arm to a safe "bottleneck" position just before the target. Once it reaches this safe zone, it switches to a pre-recorded motion to finish the job, such as gently touching the pepper. This two-step strategy allows the robot to focus its learning on the difficult part of the journey—getting through the clutter—while relying on simple, proven motions for the final contact.

The study also revealed that simply adding more data or depth information to standard systems was not enough to solve the problem. When the researchers tried feeding the same structured image data into older, standard learning models, the performance did not improve, and in some cases, it got worse. This suggests that the way the robot interprets the visual information is just as important as the information itself. The new system works because the visual encoder and the learning model are designed to work together, specifically tuned to understand the spatial relationship between the target and the obstacles. Without this tailored design, the robot cannot effectively use the structured view to avoid collisions.

These findings offer a promising path forward for robots that need to operate in complex, natural environments. By teaching robots to see the world in terms of what to reach and what to avoid, rather than just a jumble of pixels, the researchers have created a system that is more robust and reliable. The experiments showed that the robot could handle changes in the layout of the plants and even adapt to new types of peppers without needing to be retrained from scratch. While the system still relies on humans to specify the target and cannot yet plan complex tasks on its own, it demonstrates a significant step toward making robotic manipulation practical in the messy, unpredictable real world. The results suggest that with the right way of seeing, robots can learn to move through a crowded garden as carefully as a human would.

Technical Summary: ObstaDiff

Problem Statement

Robot manipulation in cluttered, real-world environments, particularly in agricultural settings like greenhouses, faces two intertwined challenges: navigating unstructured obstacles (e.g., foliage, stems, tools) that form narrow, irregular passages around a target, and handling substantial visual variability due to changing viewpoints, illumination, and target appearances.

Existing imitation learning (IL) approaches, including diffusion-based policies, often assume clean backgrounds or rely on generic visual encoders. These methods tend to entangle target, obstacle, and background cues into a single latent feature. Consequently, downstream policies lack explicit signals regarding which scene elements to reach and which to avoid, leading to trajectories that ignore nearby obstacles, especially under distribution shifts (e.g., new obstacle layouts or target appearances). Traditional pipeline-based methods require extensive manual calibration and struggle with adaptability in such dynamic, cluttered scenes.

Methodology

ObstaDiff is a decomposed diffusion-policy framework designed for imitation learning in cluttered, unstructured scenes. It addresses the limitations of generic encoders through two primary design choices: a structured visual encoder and a modular execution scheme.

1. Structured Target–Obstacle–Background (TOB) Representation

Instead of processing raw RGB-D images, ObstaDiff transforms observations into a structured Target–Obstacle–Background (TOB) representation.

Semantic Segmentation: Using a fine-tuned YOLOv8s-Worldv2 detector, the system identifies target and obstacle regions within the eye-in-hand RGB-D frame.
Channel Construction: The input is converted into a four-channel tensor:
1. Background mask ( $m_{bg}$ )
2. Obstacle mask ( $m_{obs}$ )
3. Target mask ( $m_{tar}$ )
4. Normalized depth image ( $\bar{d}$ )
Lightweight Encoder: A specialized StructuredObsEncoder processes this TOB input. Unlike generic backbones (e.g., ResNet-18) that treat inputs as natural images, this encoder splits the input into a semantic branch (masks) and a depth branch. Both branches utilize small convolutional networks followed by spatial softmax to extract compact, keypoint-like coordinates. This design explicitly preserves the spatial relationships between the target and obstacles while suppressing irrelevant background variations.

2. Decomposed Policy Framework

The manipulation task is decomposed into two stages:

Alignment Stage: A conditional diffusion policy ( $\pi_{align}$ ) generates end-effector trajectories to reach a "bottleneck pose" ( $p^*$ ). This stage is conditioned on the TOB representation and the current end-effector pose. The policy learns to navigate toward the target while reasoning about surrounding obstacles to avoid collisions.
Interaction Stage: Once the robot reaches the bottleneck pose, control switches to a replay-based interaction policy ( $\pi_{inter}$ ). This stage executes task-specific close-range behaviors (e.g., harvesting, touching) using a library of demonstrated trajectories.
Switching Mechanism: The transition between stages is governed by a data-driven bottleneck switching criterion. The system compares the current target observation with initial target states stored in a replay library. When the current state sufficiently matches a library entry, the system switches to replaying the corresponding interaction trajectory. This avoids reliance on potentially unreliable depth sensors during close-range contact.

Key Contributions

Obstacle-Aware Structured Representation: The authors introduce the TOB representation and a lightweight encoder that explicitly separates semantic roles (target, obstacle, background). This provides the diffusion policy with high-level spatial structure, enabling obstacle-aware approach motions rather than treating the scene as a generic image.
Modular Decomposed Policy: The framework decouples obstacle-aware alignment from task-specific interaction. This allows the same learned alignment policy to support diverse interaction objectives (e.g., harvesting vs. inspection) simply by changing the replay library, without retraining the diffusion model.
Real-World Validation: The method is evaluated on a 7-DoF Sawyer robot in an indoor greenhouse testbed, demonstrating generalization across target poses, obstacle layouts, and target appearances.

Experimental Results

The authors conducted 366 real-robot executions (61 trials per method across 6 methods) in a greenhouse environment. The evaluation covered three generalization settings:

Target-Pose Generalization (TPG): Varying target positions and orientations.
Obstacle-Layout Generalization (OLG): Varying the arrangement of obstacle plants.
Target-Appearance Generalization (TAG): Using unseen green pepper varieties instead of the training purple peppers.

Performance Metrics:

Task Success Rate (TSR): ObstaDiff achieved a mean TSR of 75.41%, significantly outperforming baselines such as ACT (RGB) at 45.90% and standard Diffusion Policy (DP) with RGB at 49.18%.
Obstacle Collision Rate (OCR): ObstaDiff reduced the mean collision rate to 8.20%, compared to 14.75% for ACT (RGB) and 26.23% for DP (RGB).
Ablation Studies:
- Simply adding depth to a standard DP encoder (DP-RGB-D) or feeding the TOB representation to a standard encoder (DP-TOB) did not yield significant improvements, confirming that the structured encoder is critical for leveraging the TOB representation.
- The TOB representation improved trajectory smoothness and reduced collisions even in non-diffusion baselines (ACT), though success rates varied.

Significance and Claims

The paper claims that ObstaDiff demonstrates that explicitly structuring visual representations to separate targets from obstacles is essential for robust generalization in cluttered environments. By providing the diffusion policy with clear semantic cues about what to reach and what to avoid, the system generates safer and more reliable approach trajectories.

The authors emphasize that their contribution lies in the coupling of a task-structured representation with a specialized encoder, rather than the novelty of the encoder components themselves. The decomposed design is highlighted as a practical solution for real-world deployment, allowing for reliable close-range interaction without depending on unstable depth sensors.

The work is modest in its scope, acknowledging that it addresses low-level obstacle-aware alignment rather than autonomous target selection or high-level task planning. Furthermore, the interaction stage relies on replaying demonstrated trajectories, meaning the results specifically validate the obstacle-aware approach behavior rather than novel contact-rich manipulation strategies. The evaluation is confined to a specific greenhouse setting, and the authors note the difficulty of adapting point-cloud-based methods (like DP3) to moving eye-in-hand cameras in dense foliage.