🤖 machine learning

Dynamic Object Masks as Goal Representations for Visual Goal-Conditioned Reinforcement Learning

This paper proposes a dynamic object mask-based goal representation for visual goal-conditioned reinforcement learning that eliminates the need for privileged state information by using standard image processing or pretrained detectors to generate object-agnostic visual cues, thereby achieving high success rates, strong generalization to unseen objects, and successful sim-to-real transfer on robotic arms.

Original authors: Fahim Shahriar, Cheryl Wang, Alireza Azimi, Gautham Vasan, Hany Hamed, Abhishek Naik, A. Rupam Mahmood, Colin Bellinger

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fahim Shahriar, Cheryl Wang, Alireza Azimi, Gautham Vasan, Hany Hamed, Abhishek Naik, A. Rupam Mahmood, Colin Bellinger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to learn how to do a new job, like picking up a specific toy or finding a hidden door. To learn, the robot needs a "goal"—a clear picture of what success looks like. In the world of robotics, this is called Goal-Conditioned Reinforcement Learning. Think of it like teaching a dog a trick: you don't just say "be good"; you say "sit" or "fetch the ball." The robot is the dog, and the goal is the command.

For a long time, scientists taught robots by giving them very precise, mathematical coordinates, like "move your arm to X, Y, and Z." It's like giving a GPS address to a dog; it works if the dog has a supercomputer brain, but it's hard to get that exact address in a messy, real-world room where lights change and objects move. Other methods tried to show the robot a picture of the final result, but that's like showing a dog a photo of a finished puzzle without telling it how the pieces fit together. The big question is: How do we give a robot a simple, visual "target" that works in the real world without needing expensive, perfect sensors?

This paper proposes a clever, visual solution: Dynamic Object Masks. Instead of giving the robot complex numbers or a full photo of the future, the researchers give the robot a simple, black-and-white "sticker" or mask that highlights the target object. Imagine playing a game of "Marco Polo" where, instead of shouting, you hold up a glowing sticker that only covers the person you are looking for. As the robot gets closer, the sticker on its "vision screen" gets bigger. If it moves away, the sticker shrinks. This mask acts as both the goal (the target) and a reward system (the size of the sticker tells the robot how well it's doing).

The researchers tested this idea using robot arms in computer simulations and on real robots in a lab. They found that this "sticker" method is incredibly effective. In their simulations, the robots learned to reach for objects faster and more reliably than when using traditional methods. Even more impressively, when they tried this on real robots—specifically a Franka Panda arm and a UR10e arm—the system worked beautifully. The robots learned to reach for objects they had never seen before, achieving a 99% success rate in reaching both training and novel objects.

The paper also tackled a tricky problem: how to tell the robot it's doing a good job before it actually finishes the task. Usually, robots only get a "good job" reward at the very end, which makes learning slow. The authors discovered that the size of the mask could serve as a constant "progress bar." As the robot gets closer, the mask grows, giving the robot a steady stream of positive feedback. This helped the robots learn complex tasks, like picking up and lifting objects, much faster than before.

One of the most exciting parts of the study is that the robots could learn these skills from scratch in the real world, without needing a human to program every single movement. They used pre-trained AI models (like Detic and Grounding DINO) to automatically create these "stickers" (masks) for any object the robot saw. While one model (Grounding DINO) had some trouble with false alarms in the real world, another (Detic) performed exceptionally well, allowing the robot to learn to reach for a pear in just 60,000 steps (about 449 minutes of real-time training).

The authors suggest that this approach is a major step forward because it doesn't rely on "privileged" information—like knowing the exact 3D coordinates of an object—which is often impossible to get in a real, messy environment. Instead, it relies on simple visual cues that any camera can see. However, they also note that the system's success depends on the quality of the object detector; if the detector misses the object or gets confused, the robot's performance drops. They didn't solve every problem in robotics, but they showed that a simple, dynamic visual mask is a powerful, flexible tool for teaching robots to see and reach in the real world.

Technical Summary: Dynamic Object Masks as Goal Representations for Visual Goal-Conditioned Reinforcement Learning

Problem Statement

Goal-Conditioned Reinforcement Learning (GCRL) offers a unified framework for robots to solve diverse tasks, yet current methods face significant barriers in real-world deployment. Most existing approaches rely on "privileged" goal representations, such as vectorized 3D positions, orientations, or future observation images. In practical settings like warehouses, agriculture, or laboratories, robots rarely have access to these privileged states, precise object coordinates, or future observations. Furthermore, obtaining 3D target positions often requires specialized depth cameras and complex processing (e.g., feature matching), which are sensitive to environmental factors like lighting, reflectivity, and distance.

While image-based (e.g., VAE-encoded) or text-based (e.g., CLIP embeddings) representations offer alternatives, the authors argue they often limit an agent's ability to generalize to new targets. Additionally, GCRL faces the "sparse reward" challenge, where agents receive feedback only upon task completion, leading to poor sample efficiency. While dense rewards (e.g., distance-to-goal) are commonly used, they can introduce local optima and still rely on difficult-to-compute 3D metrics.

Methodology

The paper proposes a Dynamic Object Mask-based Goal Representation and a corresponding Mask-Based Reward System designed for vision-based navigation and manipulation.

1. Dynamic Goal Representation

Instead of static goal images or vectorized coordinates, the method utilizes a binary mask updated at every timestep.

Generation: At each timestep $t$ $t$ , a mask $m_t$ $m_{t}$ is generated from the camera image $I_{cam}^t$ $I_{c am}^{t}$ . This can be achieved via:
- Privileged positional information (in simulation).
- Pretrained open-vocabulary object detectors (e.g., Detic, Grounding DINO).
- Simple color/shape filtering.
Input Integration: The mask is appended to the camera image as a new channel. To capture temporal dynamics, the system stacks the current and two previous masked images ( $I_t = [I'_t, I'_{t-1}, I'_{t-2}]_C$ ), resulting in a 12-channel input (3 RGB + 3 mask channels $\times$ 3 frames).
Object-Agnostic Nature: The mask provides spatial cues without requiring specific object class knowledge, facilitating generalization to unseen objects.

2. Mask-Based Reward Formulation

To address sparse rewards without relying on 3D distance calculations, the authors propose using the size of the target mask as a dense reward signal.

Mechanism: As the robot (in an egocentric view) approaches the target, the target object occupies more pixels in the camera frame, increasing the area of the activated mask.
Calculation:
1. Compute the normalized ratio of activated pixels ( $R'$ ) in the mask.
2. Apply a scaled sigmoid function: $R_{mask} = \frac{2}{1 + e^{-10 R'}} - 1$ .
3. This scaling amplifies feedback when the robot is far from the target (small mask size), providing a stronger gradient signal for early learning stages.
Region of Interest (ROI): For grasping tasks, the reward is only calculated if the target mask falls within the gripper's ROI, encouraging proper alignment for grasping rather than just proximity.

3. Learning Architectures

The method is evaluated using two standard RL algorithms:

Soft Actor-Critic (SAC): Used for most experiments (Franka and UR10e in simulation). Enhancements include random shifts, ensemble Q-networks, and weight clipping to stabilize pixel-based learning.
Proximal Policy Optimization (PPO): Used for Sim-to-Real transfer on the UR10e. A novel image interpolation technique was introduced during training, mixing simulation images with real-world images to mitigate visual domain shifts.

Key Contributions

Object-Masking as Goal Representation: Introduction of dynamic, object-agnostic masks as a primary goal conditioning strategy in vision-based GCRL, eliminating the need for privileged 3D state information.
Mask-Based Dense Reward: A novel reward signal derived from mask size that provides dense feedback without computing Euclidean distances or relying on depth sensors.
Visual Pick-and-Place: Demonstration of learning complex pick-up tasks using Region-of-Interest (ROI) based masks to guide grasping alignment.
Real-World Validation: Successful deployment in both Sim-to-Real transfer (UR10e) and Learning from Scratch (Franka Panda) tasks using pretrained open-vocabulary detectors (Detic, Grounding DINO).

Experimental Results

The authors evaluated their approach across three simulated environments (Franka-MuJoCo, UR10e-MuJoCo, Create3-Isaac Sim) and two real-world setups.

Learning Efficiency: In simulation, mask-based agents achieved faster convergence than one-hot encoding, CLIP embeddings, and target state images. They matched the performance of 3D-position-based agents in reaching tasks.
Generalization: Mask-based agents achieved a 99.9% average reaching accuracy on both training and novel (unseen) objects in simulation, outperforming 3D-position baselines (98.9% training, 98.8% test).
Reward Comparison: The mask-based reward system performed comparably to distance-based dense rewards but demonstrated higher training stability.
Complex Tasks: In a pick-up task, 9 out of 11 seeds using mask-based GC and rewards achieved over 90% success, whereas only 2 out of 11 seeds succeeded with 3D-position-based GC and distance rewards. The mask-based approach successfully guided the arm to align with the target for grasping.
Real-World Performance:
- Sim-to-Real (UR10e): Using Grounding DINO, the system achieved a 96% success rate on in-distribution objects and 92% on out-of-distribution objects, with no significant degradation from simulation.
- Learning from Scratch (Franka): Using Detic, the agent learned an optimal reaching policy in 60,000 steps. The system successfully tracked a moving unseen object (a plastic banana), demonstrating robustness. Note: Grounding DINO struggled in the Franka learning-from-scratch task due to false positives, highlighting the dependency on detector quality.

Significance and Claims

The paper claims that dynamic object masks provide a simple, effective, and practical alternative to privileged state representations for real-world robotics. By removing the dependency on 3D position estimation and specialized depth sensors, the approach lowers the barrier to entry for deploying GCRL in unstructured environments.

The authors emphasize that their method enables strong generalization to unseen objects and supports learning from scratch on physical robots. They position the mask-size reward as a viable solution to the sparse reward problem that avoids the local optima issues often associated with distance-based shaping. While the system's performance is bounded by the quality of the underlying object detector, the authors note that the detector is a replaceable component, meaning the framework benefits directly from advancements in open-vocabulary detection models.

The work concludes that this approach is particularly effective for egocentric tasks where the robot moves toward a target, as the mask naturally grows in size, providing a continuous progress signal.