Technical Summary: ATOM-Bench

Problem Statement

Generalist manipulation policies are increasingly positioned as foundation models for robotic control, promising the ability to acquire reusable physical and semantic skills that can be recombined to solve novel tasks. However, verifying this compositional generalization in real-world settings remains a significant challenge. Existing benchmarks often rely on simulation, where high-capacity policies can rapidly memorize task distributions without learning truly reusable skills, or they evaluate broad task suites that obscure specific failure modes. Consequently, it is difficult to diagnose whether a policy's failure stems from weak atomic motor execution, poor instruction grounding, or an inability to recombine learned skills into new task structures.

Methodology

To address this diagnostic gap, the authors introduce ATOM-Bench, a real-world benchmark designed to evaluate both atomic skill acquisition and compositional generalization in manipulation policies.

Task Factorization

ATOM-Bench decomposes tabletop manipulation into two distinct types of "atoms":

1. Motor Atoms: These define the physical operations required (e.g., pick-and-place, reorientation, pushing, stacking, pouring, accessing articulated objects).
2. Instruction Atoms: These define the semantic or logical constraints in language (e.g., color, shape, size, source relation, goal destination, counting, exclusion).

Benchmark Structure

The benchmark consists of two paired real-world robot tracks:

• Single-Arm Track: Utilizing a Franka Panda robot.
• Dual-Arm Track: Utilizing an Agilex Cobot Magic robot.

Each track includes:

• 30 Atomic Tasks: Divided into a Motor Set (isolating physical skills) and an Instruction Set (isolating language constraints under simple carrier actions).
• 24 Held-Out Compositional Tasks: These tasks combine one or more motor atoms with multiple instruction atoms. Crucially, these are not seen during the fine-tuning phase, serving as a test for generalization.

Data Collection

The authors collected 3,000 human demonstrations (100 per atomic task) via teleoperation across both platforms. This data supports reproducible real-world evaluation and atomic fine-tuning.

Evaluation Metrics

To distinguish between different failure modes, ATOM-Bench introduces two novel metrics alongside standard Success Rate (SR) and Process Success Rate (PSR):

• Atomic Score (AS): The mean PSR of the constituent atoms of a compositional task, measured in isolation. This establishes a ceiling for how well the policy performs the required components individually.
• Compositional Failure Share (CFS): A metric quantifying the portion of compositional failure that cannot be explained by weak atomic skills. A high CFS indicates that while the policy knows the atoms, it fails to recombine them effectively.
• Transfer Gap (TG): The difference in performance between single-task fine-tuning and joint atomic fine-tuning.

Key Contributions

1. ATOM-Bench Benchmark: A real-world evaluation framework with 3,000 demonstrations across two robotic platforms (Franka Panda and Cobot Magic), featuring 30 atomic and 24 held-out compositional tasks.
2. Diagnostic Metrics: The introduction of Atomic Score (AS) and Compositional Failure Share (CFS) to separate failures caused by weak execution from those caused by limited compositional reuse.
3. Comprehensive Evaluation: An evaluation of five representative manipulation policies (Pi0.5, Motus, LingBot-VLA, GROOT N1.6, and SmolVLA) through 2,700 physical rollouts.
4. Paired Task Design: A unique structure where dual-arm tasks are designed as bimanual counterparts to single-arm tasks, allowing for the study of coordination and hand-role assignment.

Experimental Results

The authors evaluated five policies under an atomic-transfer fine-tuning protocol (jointly fine-tuned on all 15 atomic tasks) and tested them on both atomic and held-out compositional tasks.

• Atomic Skill Acquisition: Policies generally acquire simple instruction-grounding skills (e.g., color, shape) more easily than fine-grained motor skills. Motor atoms requiring precise control (pouring, reorientation, pushing) and instruction atoms requiring set-level reasoning (counting, exclusion) remain challenging. Pi0.5 demonstrated the strongest performance across both platforms.
• Compositional Generalization: A significant gap exists between atomic proficiency and compositional success. Even models with high Atomic Scores (e.g., Pi0.5 with AS > 80%) achieved low Success Rates on held-out compositional tasks (SR < 17%).
• Failure Modes: Analysis revealed that failures are dominated by motor execution errors, reference errors (wrong object selection), and timeouts. The CFS metric showed that for strong models, a large portion of failure is due to compositional reuse issues rather than a lack of atomic knowledge.
• Robustness: While atomic skills showed robustness to visual and language perturbations, compositional skills were significantly more brittle, with success rates dropping sharply under paraphrased instructions.
• Training Protocols: Atomic-transfer fine-tuning generally outperformed single-task fine-tuning for instruction grounding and single-arm tasks. However, for complex bimanual motor skills, single-task fine-tuning sometimes yielded better results, suggesting that specialized practice is beneficial for complex coordination.

Significance

The paper claims that ATOM-Bench provides a critical diagnostic testbed for the robotics community. It demonstrates that current manipulation policies, while capable of learning simple atomic skills, struggle with fine-grained motor control and the logical recombination of these skills in real-world settings. The benchmark allows researchers to pinpoint whether failures arise from weak motor execution, poor instruction grounding, or limited compositional reuse, moving beyond aggregate task success rates to a more nuanced understanding of policy limitations. The authors emphasize that strong atomic performance does not reliably transfer to held-out compositional tasks, indicating that real-world generalization in manipulation remains an unsolved challenge.