🤖 machine learning

Moving Out: Physically-grounded Human-AI Collaboration

This paper introduces "Moving Out," a new benchmark for physically-grounded human-AI collaboration that addresses the limitations of existing discrete or non-physical datasets, and proposes the BASS method to enhance agents' ability to adapt to diverse human behaviors and complex physical constraints.

Original authors: Xuhui Kang, Sung-Wook Lee, Haolin Liu, Yuyan Wang, Yen-Ling Kuo

Published 2026-06-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Xuhui Kang, Sung-Wook Lee, Haolin Liu, Yuyan Wang, Yen-Ling Kuo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Why Moving a Sofa is Hard for Robots

Imagine you and a friend are trying to move a heavy, awkward sofa out of an apartment. You can't just push it in a straight line. You have to tilt it, rotate it around a narrow hallway corner, and coordinate your strength so you don't drop it or get stuck.

This is what the paper calls "physically-grounded collaboration." It's about robots (AI) working with humans in the real world, where physics matters. Things have weight, shape, and friction. If you push a round table, it rolls; if you push a square box, it slides. If you push a heavy object, you need two people.

The problem is that most current AI training happens in "grid worlds" (like a chessboard or a simple video game). In those games, you just click a square to move. There is no weight, no momentum, and no awkward angles. The authors argue that to build a robot that can actually help you move furniture, we need to train it in a world that feels like the real world.

The Solution: "Moving Out" (The Benchmark)

The authors created a new test environment called Moving Out. Think of it as a 2D video game simulator that runs on a realistic physics engine.

The Game: Two agents (one AI, one human or another AI) must move various objects (stars, circles, polygons) from a starting point to a "goal zone."
The Twist: The objects aren't just icons; they have mass (some are heavy, some are light), shapes (some get stuck in corners, some roll), and sizes (some need one person, some need two).
The Goal: The AI must learn to figure out: "Oh, this box is heavy, I need to ask my partner for help," or "This hallway is narrow; we need to turn the table sideways to fit through."

The Two Big Challenges

The paper sets up two specific tests to see if the AI is smart enough to handle real life:

The "New Partner" Test (Adapting to Humans):
Imagine you play a game with a friend who always walks on the left side of the room. Then you play with a different friend who always walks on the right. A good teammate adapts.
- The Test: The AI is trained on data from 36 different human players. Then, it is tested against a human it has never met before. Can it figure out how to work with this stranger's unique style?
- The Result: Most AIs fail here because they memorize the specific moves of their training partners. They get confused when the partner does something slightly different.
The "New Object" Test (Adapting to Physics):
Imagine you practiced moving a small wooden chair. Then, suddenly, you have to move a giant, heavy steel safe.
- The Test: The AI is trained on objects with specific weights and shapes. Then, it is tested on objects with new weights and shapes it has never seen before.
- The Result: The AI must understand the concept of "heaviness" or "shape," not just memorize how to move a specific box.

The New Method: BASS (Behavior Augmentation, Simulation, and Selection)

To solve these problems, the authors invented a new method called BASS. Here is how it works, using a cooking analogy:

1. Behavior Augmentation (The "What-If" Chef)
Usually, if you teach a robot by showing it videos of humans moving furniture, the robot just copies exactly what it sees. But humans are messy; sometimes we stumble, sometimes we push hard, sometimes we wait.

BASS Trick: The AI takes the training videos and creates "fake" new videos. It swaps parts of one human's movement with parts of another's, but only if the start and end points match perfectly.
Analogy: Imagine you are learning to dance. You watch a video of a dancer. BASS takes the dancer's footwork from one song and their arm movements from another song, but only if the beat matches. This creates thousands of new, valid dance routines so the AI learns to handle many different styles, not just one.

2. Simulation (The "Mental Rehearsal")
Before the AI makes a move, it doesn't just guess. It runs a "mental simulation."

BASS Trick: The AI asks itself: "If I push this box this way, and my partner pushes that way, what will happen?" It predicts the future state of the objects.
Analogy: Before you try to squeeze a suitcase into an overhead bin, you do a quick mental check: "If I tilt it left, will it fit? If I push it down, will it hit the light?" BASS does this calculation instantly for every possible move.

3. Selection (The "Best Choice")
After simulating the future, the AI picks the move that looks most likely to succeed.

BASS Trick: It chooses the action that gets the object closest to the goal without getting stuck.
Analogy: You have three ways to open a jammed door. You mentally try all three, see which one works best in your imagination, and then physically do that one.

The Results: Did it Work?

The authors tested BASS against other standard AI methods (like simple copying or standard game-playing algorithms).

Against Unseen Humans: BASS was much better at working with strangers. It didn't get confused when the human partner acted differently than expected.
Against New Objects: BASS handled new shapes and weights much better. It understood that a heavy object needs more force, even if it had never seen that specific heavy object before.
Human Feedback: When real humans played with the BASS AI, they reported that it felt more helpful and "human-like." The humans felt the AI understood when they needed help and when to let them work alone.

Summary

The paper argues that to make robots that can truly help us in the real world (like moving furniture), we can't just train them on simple grid games. We need to train them in physics-based environments where weight and shape matter.

They built a new game called Moving Out to test this and created a new AI method called BASS. BASS works by "imagining" different scenarios and practicing with many different styles of partners. The result is an AI that is much better at collaborating with humans, even when the humans are unpredictable or the objects are weird.

Technical Summary: Moving Out: Physically-grounded Human-AI Collaboration

Problem Definition

Existing benchmarks for human-AI collaboration, such as Overcooked-AI, largely rely on discrete, symbolic, or task-level action spaces. These environments simplify interaction dynamics, failing to capture the complexities of the physical world where agents operate in continuous state-action spaces. In physical environments, collaboration is constrained by continuous variables (position, orientation), object attributes (mass, shape, size), and contact dynamics (friction, collisions). These factors create a high-dimensional configuration space where minor variations in human actions (e.g., rotation angles, applied forces) can drastically alter outcomes.

The paper identifies a critical gap: current methods struggle to adapt to diverse human behaviors and unseen physical constraints simultaneously. Specifically, agents trained via self-play often fail to generalize to human partners with different behavioral patterns, and models trained on fixed physical attributes cannot generalize to new object properties (e.g., different masses or shapes). The authors define the problem as a decentralized Markov Decision Process (Dec-MDP) where the AI agent must optimize its policy against an unknown distribution of human policies ( $D$ ) while navigating physical constraints ( $\Gamma$ ) that limit feasible state transitions.

Methodology: BASS

To address these challenges, the authors propose BASS (Behavior Augmentation, Simulation, and Selection), a novel framework designed to enhance agent diversity and action understanding in continuous physical environments.

1. Behavior Augmentation

Standard trajectory augmentation (e.g., adding noise) is insufficient for multi-agent collaboration because it can break the consistency required for cooperation. BASS employs two specific strategies to generate valid augmented data:

Pose Perturbation: Small Gaussian noise is added to the partner's pose ( $\tilde{p}_{partner} = p_{partner} + \epsilon$ ) while keeping other state variables unchanged. This mimics natural human motion variations and improves robustness to small deviations.
Sub-trajectory Recombination: The method identifies segments in two different demonstrations where Agent A starts and ends in nearly identical states (within a tolerance $\epsilon_{pose}$ ). It then swaps the corresponding sub-trajectories of Agent B between the two demonstrations. This ensures that Agent A's intent remains consistent while exposing the model to diverse partner behaviors, effectively increasing the diversity of collaborative strategies without creating physically invalid states.

2. Simulation and Action Selection

Since real-world deployment lacks access to a physics engine for trial-and-error, BASS trains a latent dynamics model to predict future states.

Architecture: The system uses two autoencoders (VAE) to encode states into a latent space. A dynamics model ( $f$ ) takes the current latent state ( $z_t$ ), the agent's action ( $a_t$ ), and a predicted partner action ( $a^{(p)}_t$ ) to predict the next latent state ( $z_{t+1}$ ).
Partner Action Prediction: A dedicated predictor (sharing architecture with the policy) anticipates the partner's action based on the current state.
Action Selection: At inference time, the policy generates multiple candidate actions. The dynamics model simulates the outcome of each candidate (considering the predicted partner action) to estimate the resulting future state. The agent then selects the action that maximizes a reward function based on the Normalized Final Distance (NFD)—the reduction in distance between objects and the goal region. This allows the agent to choose the most effective action for collaboration without needing a real-time simulator.

Key Contributions

Moving Out Benchmark: The introduction of a continuous, physics-based environment built on a 2D rigid-body engine. Unlike previous benchmarks, it features realistic physical constraints (friction, mass, shape) and requires agents to navigate narrow passages and coordinate forces.
Two Novel Challenges:
- Challenge 1 (Adapting to Diverse Human Behaviors): Evaluates generalization to unseen human strategies using a dataset of 1,000+ human-human demonstrations from 36 participants.
- Challenge 2 (Generalizing to Unseen Physical Constraints): Evaluates generalization to new object attributes (mass, size, shape) using 720 demonstrations from 4 experts on maps with randomized physical properties.
BASS Framework: A method that significantly outperforms baselines (MLP, GRU, Diffusion Policy, MAPPO) by combining behavior augmentation for diversity and simulation-based action selection for physical reasoning.

Experimental Results

The authors evaluated BASS against several baselines (MLP, GRU, Diffusion Policy, MAPPO) across AI-AI and Human-AI settings.

Adaptation to Unseen Behaviors (Challenge 1): BASS demonstrated superior robustness. When tested against unseen human behaviors (cross-group evaluation), BASS showed the smallest performance drop in Task Completion Rate (TCR) and Normalized Final Distance (NFD) compared to baselines. For instance, while the standard Diffusion Policy (DP) dropped 20.72% in TCR, DP/BASS only dropped 14.07%.
Generalization to Unseen Constraints (Challenge 2): BASS consistently outperformed baselines in TCR and NFD. The simulation-based selection allowed the agent to better adapt to variations in object mass, size, and shape, whereas baselines struggled with the new physical configurations.
Human-AI Collaboration: In user studies with 32 human participants, BASS significantly improved task completion (TCR: 0.6512 vs. 0.3855 for DP) and reduced waiting time. Participants rated BASS higher on "Helpfulness" and "Physics understanding," indicating better alignment with human intentions.
Ablation Studies: Removing the multi-agent simulation component or the behavior augmentation significantly degraded performance, confirming that modeling partner dynamics and augmenting diverse behaviors are both critical for success.

Significance and Claims

The paper claims that Moving Out provides the first benchmark with human-collected datasets specifically designed to study continuous, low-level motion control under physical constraints. It argues that existing models are insufficient for physically grounded collaboration because they fail to account for the continuous state-action space and the impact of physical attributes on interaction dynamics.

The authors position BASS as a foundational step toward improving model adaptability. They claim that by explicitly modeling the outcome of actions through simulation and augmenting behavior to cover diverse partner strategies, agents can collaborate more effectively with both unseen AI partners and humans. The work suggests that future research must move beyond symbolic or discrete environments to address the complexities of real-world physical interaction, including the need for better theoretical understanding of human-AI collaboration and the integration of explicit communication channels.

The paper concludes modestly, acknowledging that while BASS reduces failure rates (e.g., dropping failure cases from ~0.8 to ~0.34), effectively addressing all failure patterns in continuous physical collaboration remains a substantial challenge for future research.