💻 computer science

MS-MEM: Multi-Skill Manipulation-Enhanced Mapping via Uncertainty- and Disturbance-Aware Action Selection

This paper proposes MS-MEM, an evidential framework that integrates active viewpoint selection, object pushing, and grasping with a collateral disturbance constraint to enhance mapping accuracy in cluttered environments while minimizing unnecessary scene changes.

Original authors: Yitian Shi, Jesper Mücke, Nils Dengler, Sicong Pan, Rania Rayyes, Maren Bennewitz

Published 2026-09-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yitian Shi, Jesper Mücke, Nils Dengler, Sicong Pan, Rania Rayyes, Maren Bennewitz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots that work in our homes and warehouses face a problem that is deceptively simple: they cannot see everything. In a cluttered room, objects hide behind one another, creating blind spots that confuse a robot's sensors. To find a specific item, a robot must do more than just look; it must interact with the world. It might push a box aside to see what is behind it, or move to a new angle to get a better view. This field of study, known as active perception, relies on the idea that a robot learns best by touching and moving things, not just by staring at them. However, there is a catch. Every time a robot pushes an object, it changes the scene. If it pushes too hard or too often, it might scatter a carefully arranged shelf, making it harder to understand the layout later. The challenge for engineers is to teach a robot how to be curious enough to find hidden things, but careful enough not to make a mess in the process.

Researchers at the Karlsruhe Institute of Technology, the University of Bonn, and the Technical University of Darmstadt have developed a new system to solve this balancing act. They call it MS-MEM, a framework designed to help robots map out tight, cluttered spaces like shelves while keeping the scene as undisturbed as possible. The system allows a robot to choose between three different types of actions: moving to a new viewpoint to see more, pushing objects to reveal what is hidden, or picking up and removing an object that is blocking the view. The core innovation is that the robot does not just pick the action that reveals the most information; it also calculates the cost of that action. It asks itself, "If I push this box, how much will I disturb the rest of the shelf?" and "Is the new information worth the mess I might make?"

The team built their system on a foundation of uncertainty. Instead of treating the map of the room as a fixed picture, the robot maintains a "belief" about what is where, complete with a measure of how unsure it is. When the robot is very confident about an area, it knows not to touch it. When it is unsure, it knows it needs to investigate. To make these decisions, the researchers introduced a new way for the robot to understand grasping. Previous systems often struggled to predict how a robot's gripper would interact with an object when the view was partial or the object was in a tight spot. This new system teaches the robot to estimate not just where to grab, but how confident it is in that grab, and how the orientation of the object might be uncertain. This allows the robot to fuse information from multiple angles, refining its understanding of a grasp over time as it moves around the object.

In their experiments, the researchers tested this system in a simulated environment that mimicked a real-world shelf filled with randomly placed objects. They compared their new multi-skill approach against older methods that relied on only one type of action, such as only pushing or only grabbing. The results showed that combining these skills was far more effective. By using pushing to separate objects and create space, and then using grasping to remove specific blockers, the robot was able to build a more accurate map of the scene. However, the most significant finding came from the system's ability to limit disturbance. When the researchers added a specific constraint to the robot's decision-making process—a rule that penalized unnecessary changes to the scene—the robot became much more careful. It still found the hidden objects, but it moved far fewer items than the systems that did not have this rule. In simulations, the new system reduced the total distance objects were moved by a significant margin compared to methods that ignored scene disturbance, while still achieving high accuracy in mapping the shelf.

The team also tested the system in the real world using a physical robot arm equipped with a camera and a gripper. They placed the robot in front of a shelf containing 69 objects across five different challenging setups. The robot had to find and identify as many objects as possible. The system that combined pushing, grasping, and careful movement found more objects than the systems that used only one skill. It successfully identified 44 objects, compared to 40 for the pushing-only system and 38 for the grasping-only system. Crucially, it did this while causing less physical displacement of the items on the shelf. The grasping-only system moved the least, but it failed to find many hidden items because it was too conservative. The pushing-only system found many items but scattered the shelf significantly. The new system struck the best balance, finding the most objects while keeping the scene relatively orderly.

This work demonstrates that for robots to operate effectively in human spaces, they must be able to reason about the consequences of their actions. It is not enough to simply be able to pick things up or push them; the robot must understand the value of the information it gains versus the cost of the change it creates. By teaching robots to weigh these factors, the researchers have taken a step toward machines that can navigate our cluttered world with the same care and adaptability that humans use. The system does not just see the world; it understands the difference between a helpful interaction and a disruptive one, allowing it to learn about its environment without destroying the very order it is trying to map.

Technical Summary: MS-MEM

Problem Statement
Service robots operating in confined, cluttered environments (e.g., shelves) face significant challenges in achieving accurate scene understanding due to severe occlusions and restricted accessibility. Passive sensing is often insufficient for reliable interpretation, necessitating a tight coupling between perception and manipulation. While existing approaches like Manipulation-Enhanced Mapping (MEM) utilize active viewpoint selection and non-prehensile pushing to reduce uncertainty, they are limited by their reliance on pushing as the sole interaction primitive. Pushing can be suboptimal as it may unnecessarily disturb object configurations, invalidate accumulated map evidence, and miss opportunities where more selective actions, such as grasping, could yield efficient information gain with less collateral damage. Furthermore, current frameworks often lack a unified mechanism to compare heterogeneous actions (viewing, pushing, grasping) under a common uncertainty-aware objective that explicitly penalizes unnecessary scene changes.

Methodology
The authors propose Multi-Skill Manipulation-Enhanced Mapping (MS-MEM), an evidential framework that integrates active viewpoint selection, object pushing, and grasping into a unified decision-making process. The system operates on a shared metric-semantic evidential belief map $\Phi_t$ , updated via Calibrated-Neural Accelerated Belief Updates (CNABU).

Key methodological components include:

Full-Evidential Grasp Representation (FE-vMF): To support grasp planning under partial observability, the authors extend the vMF-Contact model to a comprehensive evidential representation. This models both grasp affordance (using a Beta distribution) and orientation uncertainty (using von Mises–Fisher distributions for both the baseline and approach directions). This allows the system to reason about grasp quality and confidence, particularly in ambiguous or collision-sensitive regions.
Full-Evidential Uncertainty-guided Multi-view Grasp Fusion (FE-UMGF): Since grasp estimates are derived from partial observations, MS-MEM employs a temporal fusion mechanism. It maintains a global grasp buffer, aggregating evidence from new viewpoints while decaying historical evidence. This ensures consistent grasp selection as the map belief evolves across diverse viewpoints.
Disturbance- and Occlusion-aware Information Gain (DOIG): The core decision metric is a unified objective function that evaluates candidate actions (viewpoint, push, grasp) based on their expected contribution to reducing map uncertainty. Crucially, DOIG incorporates a Collateral Disturbance Constraint (CDC). This constraint explicitly penalizes semantic changes in regions of the scene that are already mapped with high confidence. For manipulation actions, the system predicts the post-action belief and calculates the number of "disturbed confident voxels." This ensures that the robot selects actions that effectively reduce occlusion while preserving the reliability of previously accumulated map evidence.
Multi-Skill Action Selection (MSAS): The framework generates candidates for Next-Best View (NBV), pushing, and grasping. These are evaluated in parallel using the DOIG objective. The system selects the single best action $a^*_t$ (or a grasp followed by an NBV) that maximizes information gain while adhering to the disturbance constraints.

Key Contributions
The paper outlines three primary contributions:

MS-MEM Framework: A multi-skill manipulation-enhanced mapping system for cluttered, confined scenes that enables uncertainty-informed action selection across active viewpoint changes, pushing, and grasping.
Evidential Grasp Learning: A comprehensive framework (FE-vMF) that jointly models uncertainty in grasp affordance and orientation, employing uncertainty-guided multi-view fusion to ensure consistent grasp selection over time.
Unified Decision Objective: The design of the Disturbance- and Occlusion-aware Information Gain (DOIG) objective, which enables direct comparison across perception and manipulation actions while penalizing unnecessary scene changes via the CDC.

Experimental Results
The authors evaluated MS-MEM through simulation and real-world experiments involving a UR5 manipulator with a Robotiq 2F-85 gripper and a RealSense L515 camera.

Mapping Accuracy: In simulation, combining pushing and grasping (MS-MEM) outperformed single-skill baselines ("Grasp Only" and "Push Only") in both occupancy and semantic Intersection over Union (IoU). While the "w/o CDC" variant achieved the highest occupancy IoU (0.905), the full MS-MEM system achieved a competitive occupancy IoU of 0.899 and improved semantic IoU (0.767) compared to single-skill methods.
Scene Preservation: The inclusion of the CDC penalty significantly reduced scene disturbance. Compared to the "Push Only" baseline, MS-MEM reduced the average object position change from 1.571 m to 0.707 m in simulation. In real-world tests, MS-MEM reduced position change by 28.0% compared to "Push Only" (0.408 m vs. 0.567 m) while maintaining superior object recognition (44 objects found vs. 40 for Push Only).
Synergy: The results demonstrate that pushing effectively creates grasp opportunities by separating clutter, while grasping allows for the selective removal of occluders. The "w/o CDC" variant performed more grasps than the "Grasp Only" baseline, indicating that pushing actively facilitates grasping. However, the full MS-MEM with CDC achieved a better balance between mapping accuracy and scene preservation.
Uncertainty Calibration: The FE-vMF model demonstrated accurate directional uncertainty estimation, with higher predicted uncertainty in collision-sensitive or ambiguous regions and lower uncertainty when obstacles constrained feasible approach directions.

Significance and Claims
The paper claims that MS-MEM highlights the synergistic effects of active viewpoint selection, pushing, and grasping. By moving beyond single-skill primitives and unconstrained manipulation, the framework achieves higher mapping accuracy while substantially reducing scene disturbance. The authors emphasize that the complementary roles of pushing (separating objects, revealing regions) and grasping (selectively removing reachable occluders) improve mapping performance in confined spaces. The proposed DOIG objective allows the robot to select highly informative actions while explicitly preserving the reliability of previously mapped scene evidence, addressing a key limitation in prior manipulation-enhanced mapping works.

Technical Summary: MS-MEM

More like this