ManiDreams: An Open-Source Library for Robust Object Manipulation via Uncertainty-aware Task-specific Intuitive Physics

1. Problem Statement

Robotic manipulation relies heavily on dynamics models (simulators or learned world models), but most existing approaches focus on minimizing prediction error rather than addressing the fundamental reality that real-world manipulation is inherently uncertain.

• The Gap: Current pipelines treat uncertainty as a training-time concern (e.g., via Domain Randomization) or discard explicit uncertainty structures once a policy is deployed.
• The Challenge: Three distinct sources of uncertainty occur simultaneously during execution:
  1. Perceptual Uncertainty: Noisy and delayed observations.
  2. Parametric Uncertainty: Inaccurate physical parameter estimation (mass, friction, geometry).
  3. Structural Uncertainty: Imperfections in the dynamics model itself (model mismatch).
• The Need: A unified framework that represents, propagates, and constrains these uncertainties jointly within the planning loop, rather than suppressing them during training.

2. Methodology: The ManiDreams Framework

ManiDreams is an open-source, modular Python framework that implements a Sample-Predict-Constrain paradigm. It wraps any base policy with a loop that evaluates candidate actions against distributional outcomes, adding robustness without retraining the underlying policy.

The framework is built on four composable abstractions organized in a three-layer architecture (Abstract Interfaces, Concrete Implementations, Task Integration):

A. Core Abstractions

1. Domain-Randomized Instance Set (DRIS):

   • Function: Represents the state not as a single point, but as a distribution.
   • Mechanism: It bundles an observation with a set of $m$ context instances ($\hat{C}$), where each instance has randomized physical parameters (mass, friction, etc.).
   • Output: A collection of state-context pairs $D_t = \{(s^{(i)}_t, c^{(i)})\}$, allowing the system to reason about a set of possible futures simultaneously.

2. Task-Specific Intuitive Physics (TSIP):

   • Function: The forward dynamics predictor.
   • Mechanism: It propagates the entire DRIS forward under candidate actions. It is backend-agnostic, supporting both:
     • Simulation-based: Wrapping physics engines (e.g., ManiSkill3) to run parallel rollouts.
     • Learning-based: Wrapping learned world models (e.g., Diffusion models) operating in latent spaces.
   • Key Feature: It converts stochastic dynamics over individual contexts into a deterministic mapping over distributions.

3. Caging Constraints:

   • Function: Bounds the growing uncertainty to ensure task safety.
   • Mechanism: Unlike traditional state validity checkers that verify a single configuration, caging constraints verify the spread of the distribution. They ensure that the predicted set of future states ($S_{t+1}$) remains within a safe region (e.g., a geometric cage or trajectory corridor).
   • Types: Includes geometric, trajectory-based, plate-based, and pixel-space constraints.

4. Solver:

   • Function: Action selection.
   • Mechanism: Combines a Sampler (proposing $N$ candidate actions from a policy or distribution) with an Optimizer (scoring candidates via caging constraints).
   • Process: It selects the best valid action (e.g., via N-best selection or MPPI) that satisfies the constraints, effectively acting as a safety filter on top of existing policies.

B. Execution Pipeline

The system operates via a two-level loop:

1. Planning Loop (Inner): Given current DRIS $D_t$, the solver samples actions, TSIP predicts outcomes for all candidates, and constraints filter/select the optimal action.
2. Execution Loop (Outer): The selected action is dispatched to the executor (simulator or real robot). The executor observes the result, updates the DRIS, and feeds it back into the planning loop.

3. Key Contributions

1. Composable Framework: An open-source library realizing the sample-predict-constrain paradigm through plugin-based abstractions, allowing independent swapping of TSIP backends, solvers, and constraints.
2. Unified Distributional Dynamics: A novel approach (DRIS + TSIP + Cages) that explicitly represents, propagates, and constrains perceptual, parametric, and structural uncertainties via a standardized interface.
3. Comprehensive Evaluation: Extensive validation including runnable examples, robustness benchmarks, runtime analysis, and zero-shot sim-to-real deployment on a Franka Panda arm.

4. Experimental Results

The framework was evaluated on ManiSkill tasks (PushCube, PickCube, PushT) and real-world scenarios.

• Robustness Benchmarks:

  • ManiDreams significantly outperformed a standard PPO baseline under escalating perturbations (observation noise, observation delay, and physics parameter randomization).
  • The advantage was most pronounced at moderate perturbation levels. At extreme levels, both methods degraded, suggesting a need for adaptive DRIS configurations.
  • Task Sensitivity: The framework adapted well to different task dynamics (e.g., contact-rich pushing vs. vertical grasping).

• Ablation Studies:

  • DRIS Instances ($m$): Performance improved with more instances but saturated beyond $m=16$.
  • Solver Samples ($N$): Gains were consistent up to $N=16$.
  • Distribution Width: An "inverted-U" pattern was observed; too narrow underestimated variability, while too wide introduced excessive conservatism. A "Medium" width yielded optimal results (82% success vs. 74-76% for extremes).

• Real-World Deployment:

  • Successfully deployed on a Franka Panda arm for cluttered picking and scooping tasks using a diffusion-based TSIP and pixel-based DRIS (derived from SAM2 segmentation).
  • Achieved zero-shot sim-to-real transfer without fine-tuning the policy, simply by swapping the executor.

• Runtime Overhead:

  • The sample-predict-constrain loop added computational cost (e.g., ~50-80ms per step vs. ~20ms for baseline PPO).
  • While this limits ultra-high-frequency dynamic tasks, it remains practical for standard manipulation (10–20 Hz) and plan-then-execute modes.

5. Significance

ManiDreams shifts the paradigm from "training robust policies" to "planning robustly with uncertainty."

• Modularity: It decouples the dynamics model from the planning logic, allowing researchers to mix and match simulators, learned models, and constraint types.
• Practicality: It provides a general solution for handling uncertainty in real-world robotics, demonstrating that robust manipulation can be achieved by wrapping existing policies with an uncertainty-aware planning layer rather than retraining them from scratch.
• Open Source: By releasing the code and documentation, it lowers the barrier for the community to implement uncertainty-aware manipulation, potentially accelerating the deployment of reliable robots in unstructured environments.