Technical Summary: DeMiAn – Dense Multi-Aspect Annotation for Robot Policy Learning

1. Problem Statement

The scaling of robot policy learning is currently bottlenecked by the high cost of collecting skilled-operator demonstrations. While modern datasets require thousands of hours of teleoperation on dedicated hardware, the language paired with these demonstrations often remains underutilized. Typical datasets label clips with short, high-level instructions (e.g., "open the drawer") that fail to explicitly capture the rich spatial relations, contact transitions, embodiment states, and subgoal structures already present in the visual pixels.

Existing approaches to "dense language" in robotics typically commit to a single caption style (e.g., motion primitives, chain-of-thought, or scene grounding) and apply it uniformly. This paper argues that no single annotation style dominates across all tasks; rather, the optimal language structure depends on the specific demands of the manipulation primitive. The challenge is to leverage the low cost of generating dense language to amplify signal in fixed demonstration corpora without incurring the latency or cost of new data collection.

2. Methodology: DeMiAn

The authors propose DeMiAn (Dense Multi-aspect Annotation), an end-to-end framework operating in two stages: a training-time annotation pipeline and an inference-time learned instructor.

2.1 Four Complementary Annotation Aspects

DeMiAn re-labels existing video segments along four distinct dimensions, each surfacing structural information omitted by standard task labels:

1. Physical Motion: Describes temporal action and contact grounding (e.g., reaching, grasping, lifting, object interactions).
2. Scene Composition: Provides spatial grounding (e.g., workspace layout, visible objects, distractors, and spatial relations).
3. Arm Pose: Focuses on embodiment and contact-state grounding (e.g., arm posture, gripper state, orientation relative to the environment).
4. Reasoning: Captures causal and subgoal grounding (e.g., the purpose of the segment, dependencies on neighboring segments, and preparation vs. execution phases).

2.2 Training-Time Pipeline

An automatic pipeline using a Vision-Language Model (VLM), specifically Qwen3-VL-30B, re-annotates existing datasets (RoboCasa 365, MolmoBot, and EgoVerse). For each video segment, the VLM is prompted with sampled frames and context to generate a two-sentence caption for each of the four aspects. This process is applied to over 1 million clips without collecting new demonstrations.

2.3 Inference-Time Learned Instructor

To address the fact that different tasks benefit from different annotation types, DeMiAn employs a small learned instructor (trained via Supervised Fine-Tuning on Qwen3.5-2B).

• Training: The instructor is trained to predict the optimal annotation aspect for a given task. The target distribution is derived from a "reward table" constructed by running the action policy with each of the four ground-truth aspects and recording the success rate (SR). The instructor learns to approximate a per-task oracle, sampling targets via a softmax over these rewards.
• Deployment: The instructor runs asynchronously alongside the action policy. It takes the initial scene snapshot and task description as input, generates the appropriate dense annotation, and injects it into the policy's prompt at the next action-chunk boundary. This design hides generation latency behind the rollout, ensuring no added delay to the robot's control cadence.

2.4 Policy Architectures

The method is evaluated on two distinct architectures:

• DeMiAn VLA: Built on openpi 0.5 (PaliGemma backbone with flow-matching action expert), utilizing dense annotations as an auxiliary co-training signal.
• DeMiAn WAM: A video-based World-Action Model using Cosmos-Predict 2.5 (DiT backbone), where dense language conditions the video prediction objective during mid-training and the action head during post-training.

3. Key Contributions

1. Automatic Multi-Aspect Pipeline: A VLM-based system that surfaces four complementary structural aspects (motion, scene, pose, reasoning) from existing robot and human demonstrations.
2. Learned Asynchronous Instructor: A small model that selects the task-appropriate annotation aspect at deployment, closing the gap between fixed-aspect baselines and a per-task oracle without introducing inference latency.
3. Generalization Evidence: Empirical demonstration that this approach generalizes to composite tasks (via subgoal-driven prompt switching) and out-of-distribution (OOD) scenes and objects.
4. Compute-Performance Frontier: A controlled comparison showing that dense re-annotation improves the compute-performance trade-off in both WAM mid-training and VLA post-training, even when the FLOPs for annotation generation are charged against the budget.

4. Experimental Results

Evaluated on RoboCasa 365 (17 atomic kitchen tasks) and MolmoSpaces (9 benchmarks across 4 task families):

• Oracle Gap: No single fixed annotation aspect dominates. A per-task oracle (selecting the best aspect per task) improves the average success rate (SR) by 8 percentage points over the task-only baseline on RoboCasa VLA (reaching 52% vs. 44%).
• Instructor Performance: The learned instructor raises the average SR to 49%, closing the gap to within 2–3 points of the oracle. This represents a +5 percentage point lift over the baseline, significantly outperforming random aspect selection.
• Generalization:
  • On composite tasks (e.g., "Prepare Coffee"), the DeMiAn-VLA policy with the instructor achieves 18% full-task success, outperforming the baseline (13%) and approaching the performance of ground-truth atomic prompts (22%).
  • The system shows stronger OOD generalization on MolmoSpaces scenes and objects compared to task-only baselines.
• Compute Efficiency:
  • In WAM mid-training (on 50K EgoVerse clips), dense annotations allow the model to reach higher downstream SR at lower compute budgets compared to the baseline.
  • In VLA post-training (on 1M MolmoBot clips), DeMiAn matches the baseline performance on specific tasks (NextTo, Color) with ~62% less compute (saving ~1.3 × 10²⁰ FLOPs).

5. Significance and Claims

The paper posits that language density is a cheap, practical lever for scaling robot policy learning. By treating annotation type as a learnable design choice rather than a fixed prompt format, DeMiAn extracts significantly more signal from existing datasets.

The authors claim that:

• Re-annotation is viable: Generating dense annotations is orders of magnitude cheaper than collecting new demonstrations, and the computational cost is recovered by improved sample efficiency.
• Task-dependence is critical: The "one-size-fits-all" approach to dense language is suboptimal; the ability to dynamically select the right annotation aspect per task is essential for reaching peak performance.
• Scalability: The method scales effectively across different model families (VLA and WAM) and data scales (from 11k clips to 1M clips), suggesting it is a complementary strategy to demonstration scaling.

Limitations: The authors note that the annotations inherit biases from the frozen VLM, the current evaluation is simulation-only (RoboCasa/MolmoSpaces), and the four-aspect schema is heuristic. They also acknowledge that the instructor does not yet fully close the oracle gap on all tasks, particularly in the final phases of composite tasks.