CoEnv: Driving Embodied Multi-Agent Collaboration via Compositional Environment

1. Problem Statement

Multi-agent embodied systems hold significant promise for complex collaborative manipulation tasks (e.g., assembly, synchronized lifting). However, deploying these systems faces critical challenges:

• Spatial and Temporal Coordination: Agents must avoid spatial conflicts, reason about temporal dependencies in sub-tasks, and maintain awareness of shared workspace dynamics.
• Safety and Cost: Physical execution of multi-agent strategies carries high risks of costly collisions and hardware damage.
• Sim-to-Real Gap: Traditional approaches often struggle to bridge the gap between virtual planning and physical execution, particularly when dealing with heterogeneous robots and complex occlusions.
• Data Scarcity: Generating high-quality, large-scale training data for multi-agent policies via manual teleoperation is inefficient and difficult to scale.

Existing methods often rely on disconnected textual representations or process agent viewpoints in isolation, lacking a unified space for real-time collision reasoning and intention perception.

2. Methodology: The CoEnv Framework

The authors propose CoEnv, a framework built on the concept of a Compositional Environment. This environment synergistically integrates real-world physical components with simulation components, creating a unified decision-making space. The framework operates through three core stages:

A. Real-to-Sim Scene Reconstruction

To ensure spatial consistency between the physical and digital worlds, CoEnv converts real-world observations into a simulator-ready state ($s_t$):

• 3D Asset Generation: Uses text/image-to-3D models (Meshy) to create simulation-ready meshes with standardized scales and pre-defined physical properties (mass, friction).
• Multi-View Object Localization:
  • Detection: Uses Grounded SAM 2 to detect and segment objects in RGB images.
  • Disambiguation: Leverages GPT-5 as a visual reasoning module to resolve ambiguities in cluttered scenes using context (spatial layout, relative positions).
  • Pose Estimation: Employs FoundationPose for 6-DoF pose estimation.
  • Fusion: Aggregates poses from multiple views ($K'$) using arithmetic averaging for translation and quaternion averaging for rotation to mitigate occlusion and calibration errors.
• Iterative Calibration: The system runs in a loop within the ManiSkill/SAPIEN simulator, comparing rendered views against real captures to progressively refine camera extrinsic calibration.

B. Simulation-Conditioned Action Synthesis

Once the scene is reconstructed, CoEnv generates actions for $N$ agents using a Hierarchical Task Planning approach driven by Vision-Language Models (VLMs). It supports two complementary execution modes:

1. Interactive Mode (Real-time Planning):

   • Planner: Uses GPT-5 to decompose high-level goals into sub-goals and execution plans.
   • Adaptive Camera Control: The planner can dynamically request new camera viewpoints to resolve occlusions caused by other agents or objects before committing to a plan.
   • Closed-Loop Execution: Executes actions sequentially (Execute $\to$ Observe $\to$ Verify $\to$ Adapt).
   • Checkpoints: Inserts non-action verification steps (e.g., "verify gripper alignment") to catch errors early. If a checkpoint fails, the system re-plans.

2. Iterative Mode (Code-Based Planning):

   • Agent: Uses Claude Code to generate a complete Python program encoding the entire multi-agent trajectory in a single pass.
   • Refinement Loop: The code is executed in simulation. If it fails, the VLM analyzes logs and images to generate feedback, which is fed back to the code agent for iterative refinement (up to $M_{max}$ iterations).
   • Advantage: Excels at tasks requiring precise, coherent long-horizon trajectory control.

C. Sim-to-Real Transfer with Collision Awareness

To safely deploy simulated strategies to real robots:

• Trajectory Interpolation: Converts discrete simulation waypoints into smooth, dense trajectories for real-world controllers using linear interpolation in joint space.
• Collision Volume Verification: Before execution, the system performs a forward kinematics pass to compute the "swept volume" of each robot link along the path. An action is only executed if the collision volumes of all agent pairs remain disjoint ($\mathcal{V}^{(i)}_l \cap \mathcal{V}^{(j)}_l = \emptyset$). If a violation is detected, the system triggers re-planning.

3. Key Contributions

1. Compositional Environment: A novel paradigm unifying real-world perception and physics simulation into a shared decision-making space, enabling agents to perceive intentions and coordinate safely.
2. CoEnv Framework: A system that integrates real-to-sim reconstruction with VLM-driven action synthesis (Interactive and Iterative modes) and validated sim-to-real transfer.
3. Scalable Data Generation: Demonstrates that CoEnv can automatically generate high-quality multi-agent demonstration data without manual teleoperation, addressing a major bottleneck in robot learning.
4. Robust Mechanisms: Introduces Adaptive Camera Control to handle occlusions and Checkpoint Verification to prevent error propagation in long-horizon tasks.

4. Experimental Results

The framework was evaluated on five challenging multi-arm manipulation tasks involving up to three heterogeneous robots (Franka Research 3 and AgileX Piper):

• Tasks: Cube Stacking, Ball Pickup, Transfer Cylinder, Place Cucumber, and Brush Box.
• Performance:
  • Overall Success Rate: CoEnv achieved an average success rate of 49% across all tasks and modes.
  • Mode Comparison:
    • Iterative Mode excelled at tasks requiring precise trajectory control (e.g., Cube Stacking: 90% success, Ball Pickup: 60%).
    • Interactive Mode outperformed in tasks requiring complex, reactive coordination and handovers (e.g., Transfer Cylinder: 40% vs. 10% for iterative; Brush Box: 70% vs. 50%).
  • Ablation Studies: Removing Adaptive Camera Control dropped success rates by 20% (avg), while removing Checkpoint Verification caused a drastic drop to 20% (avg), proving both are essential for robustness.
• Data Efficiency: The iterative mode generated significantly more training episodes per session (e.g., 17.5 episodes vs. 1.5 for interactive) with minimal token overhead for environment resets.

5. Significance

• New Paradigm for Collaboration: CoEnv shifts the focus from isolated agent policies to a unified cognitive space where heterogeneous agents can jointly plan, verify, and refine strategies before physical execution.
• Safety and Reliability: By performing collision reasoning and strategy validation in a high-fidelity simulation with a "safety net" (collision volume verification) before real-world deployment, it significantly reduces the risk of hardware damage.
• Scalability: The framework provides a practical, automated pipeline for generating large-scale, high-quality multi-agent datasets, which is crucial for training next-generation generalist embodied AI models.
• Bridging the Gap: It effectively addresses the sim-to-real gap through rigorous scene reconstruction and iterative calibration, enabling reliable transfer of complex collaborative behaviors.

In conclusion, CoEnv establishes a principled methodology for multi-agent embodied AI, demonstrating that combining simulation-based cognitive planning with real-world verification is a viable path toward robust, scalable, and safe robotic collaboration.