💻 computer science

ActiveGrasp: Information-Guided Active Grasping with Calibrated Energy-based Model

This paper proposes ActiveGrasp, a novel framework that combines a calibrated energy-based model for generating multi-modal SE(3) grasp distributions with an information-guided active view selection strategy to effectively grasp objects in densely cluttered environments with limited view budgets.

Original authors: Boshu Lei, Wen Jiang, Kostas Daniilidis

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Boshu Lei, Wen Jiang, Kostas Daniilidis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to pick up a specific bottle from a messy table covered in other objects. This is a classic "cluttered environment" problem. If the robot only looks at the table from one angle, it might not see the bottle clearly, or it might think a spot is safe to grab when it's actually blocked by another object.

The paper ActiveGrasp introduces a smarter way for robots to solve this. Instead of just guessing where to grab, the robot actively moves its camera to find the best new angle before it tries to pick anything up.

Here is how their system works, explained with everyday analogies:

1. The Problem: The "Guessing Game"

Previous methods were like a person trying to find a lost key in a dark room by shining a flashlight randomly. They would look at what was visible (visibility) or where objects looked "grabbable" (affordance), but they often missed the mark because they didn't truly understand the uncertainty of whether a grab would actually succeed.

2. The Solution: A "Calibrated Energy Map"

The authors built a special brain for the robot called a Calibrated Energy-based Model.

The Energy Map: Imagine the robot creates a 3D map of the table. On this map, every possible way to grab the bottle has an "energy" score.
- Low Energy = A great, safe grab (like a smooth, flat path).
- High Energy = A bad, risky grab (like a steep cliff or a dead end).
The Calibration: This is the secret sauce. In many AI systems, the "score" the computer gives doesn't match reality (e.g., it says a 90% chance of success, but it only works 50% of the time). The authors "calibrated" their model so that the energy score perfectly matches the real probability of success. If the model says a grab is low-energy, it really is a high-probability success.

3. The Strategy: Reducing "Confusion" (Entropy)

The core of their method is deciding where to look next.

The Old Way: Previous robots looked for places that were "interesting" or "hidden" just to see more of the scene. It's like a detective looking at a wall just because it's dark, even if the clue isn't there.
The ActiveGrasp Way: This robot asks, "Where am I most confused about whether I can grab the object?"
- They measure this confusion using Entropy (a fancy word for uncertainty).
- The robot calculates: "If I move my camera to this spot, will I learn enough to know for sure if a grab will work?"
- It picks the view that reduces its confusion the most. It's like a detective moving to a spot where they can finally see the suspect's face clearly, rather than just looking at a shadow.

4. The Process: A Loop of Learning

The robot follows a cycle:

Look: It takes a few initial photos and builds a 3D model of the messy table.
Calculate: It uses its "Calibrated Energy Model" to guess where it could grab the object and how uncertain it is about those guesses.
Decide: It picks the single best new camera angle that will clear up the most confusion.
Move & Repeat: The robot moves, takes a new photo, updates its 3D model, and repeats until it has used its "view budget" (the number of times it is allowed to move).
Grab: Finally, it picks the best, most certain grab spot and executes the move.

5. The Results

The authors tested this in two ways:

In Simulation: A virtual robot in a computer game.
In Real Life: A physical robot arm in a lab.

They found that their method was significantly better than previous state-of-the-art methods. Even with a limited number of camera moves (a tight "budget"), their robot successfully grabbed objects in cluttered environments more often.

Key Takeaway:
Instead of just "looking more," ActiveGrasp teaches the robot to "look smarter." By using a mathematically calibrated model to measure exactly how much it doesn't know, the robot knows exactly where to look to turn a risky grab into a successful one.

Technical Summary: ActiveGrasp

Problem Statement

Robotic grasping in densely cluttered environments remains a fundamental challenge. Initial views often lack sufficient information to predict successful grasps. While previous approaches attempted to solve this by actively gathering multiple views (Next-Best-View or NBV selection) before generating grasp poses, they suffer from critical limitations:

Misguided Information Gain: Many methods estimate information gain based on scene visibility or affordance maps, which biases the robot toward regions with rich visual features but may not be feasible for grasping.
Manifold Mismatch: Some methods project grasp distributions onto 2D or 3D grids, ignoring the intrinsic structure of grasp poses on the SE(3) manifold.
Lack of Calibration: Existing grasp distributions are often uncalibrated, meaning the predicted success probabilities do not align with real-world success rates, leading to biased information gain estimation.

Methodology

The authors propose ActiveGrasp, a framework that integrates a calibrated energy-based model (EBM) with an information-theoretic approach to NBV selection. The pipeline consists of three main stages:

1. Scene Representation and Grasp Distribution

The system reconstructs the 3D scene $w$ using 3D Gaussian Splatting (3DGS) from initial views. To model the grasp distribution on the SE(3) manifold, the authors utilize an Energy-Based Model (EBM) adapted from SE(3) Diffusion Fields.

The EBM maps a scene representation $w$ and a grasp pose $g$ to a scalar energy $E_\theta(g, w)$ .
The conditional distribution of grasp poses is defined as $p(g|w) \propto e^{-E_\theta(g,w)}$ .
To ensure the model captures the multi-modality of grasps on the SE(3) manifold, it is trained using denoised score matching with noise added to ground-truth successful grasps.

2. Calibrated Grasp Generation

A core innovation is the calibration of the EBM so that its energy levels correspond directly to the probability of grasp success.

Dual Classification: The model is trained to output logits for both success ( $a_S$ ) and failure ( $a_F$ ) categories, allowing the energy to be interpreted as a success probability ( $p_S = e^{a_S} / (e^{a_S} + e^{a_F} + 2)$ ).
Dual Score Matching: The model employs two score matching losses: one aligning the gradient with perturbations of successful grasps and another with failed grasps. This allows the model to learn both successful regions and failure modes.
Average Precision (AP) Loss: To enforce calibration without disrupting the delicate energy structure learned by score matching, an AP loss is used. This provides "lazy" supervision, ensuring positive samples rank higher than negatives without over-constraining the model.
Learnable Temperature: A learnable temperature parameter is introduced to stabilize the energy scale and ensure numerical convergence during training.

3. Information-Guided Active View Selection

Instead of relying on heuristics like visibility, ActiveGrasp defines information gain as the reduction in the entropy of the grasp distribution.

Entropy Definition: The entropy of the grasp distribution is defined as the expected entropy of a single grasp, where the success of a single grasp follows a Bernoulli distribution parameterized by the success rate $s(g, w)$ .
Estimation: The system approximates the scene posterior using a Gaussian Approximation of the Posterior (GAP). The information gain for a candidate view is derived by estimating the reduction in the conditional entropy $H[S|G, W]$ given the new observation.
Selection: The next best view is selected by maximizing this estimated entropy reduction, effectively guiding the robot to observe regions where the grasp outcome is most uncertain.

Key Contributions

Information-Theoretic Definition: The paper provides a rigorous definition of grasp distribution entropy and derives the information gain for NBV selection specifically for grasping tasks, moving beyond visibility-based metrics.
Calibrated EBM for SE(3): The authors introduce a method to calibrate an energy-based model on the SE(3) manifold for grasp pose detection, ensuring the predicted success probabilities match real-world success rates. This enables accurate entropy estimation.
Active Grasping Benchmark: The work presents a reproducible benchmark using YCB objects and a physically informed simulator (PyBullet), facilitating future research in active grasping.

Experimental Results

Experiments were conducted in both simulated environments (PyBullet) and on a real-world Kinova 7 DoF robot arm.

Simulation: In a cluttered environment with a limited view budget (2 initial views + 2 active views), ActiveGrasp achieved a 79.00% success rate, outperforming state-of-the-art methods (e.g., Se3diff + ActiveNGF at 74.00%, Contact Graspnet + ActiveNGF at 32.50%). It also achieved the lowest Expected Calibration Error (ECE) of 0.02.
Real-World: On a real robot, ActiveGrasp demonstrated superior performance in selecting views that focus on potential grasp locations rather than just obstacle coverage, achieving higher success rates compared to methods like Breyer et al. and ACE.
Efficiency: The view selection process takes approximately 9.61 seconds, which is significantly faster than competing active learning methods (e.g., 18.24s for ACE, 27.66s for ActiveNGF).
Ablation Studies: Results confirm that both the scene augmentation and the calibration (specifically the learnable temperature and AP loss) are critical for achieving high success rates and low calibration errors.

Significance and Claims

The authors claim that ActiveGrasp offers a more principled and task-aware approach to active perception. By defining information gain through the lens of grasp distribution entropy and utilizing a calibrated EBM, the system avoids the biases of previous methods that prioritize visual richness over grasp feasibility. The paper posits that this framework successfully drives robots to explore affordable parts of target objects, enabling robust grasping in clutter with limited view budgets. The authors also note that their simulated environment serves as a reproducible platform for the community.

Limitations: The authors acknowledge that the method relies on a second-order approximation ( $\nabla^2_w \eta(w)$ ) for information gain and requires model calibration, which may be sensitive to domain shifts.