Technical Summary: Closed-Loop Distillation of Reading Heuristics for Exploratory Manipulation Trace QA

1. Problem Definition: Exploratory Manipulation Trace QA (EMT-QA)

The paper addresses a specific failure mode in robot manipulation: exploratory traces. In these scenarios, a robot attempts a task, encounters resistance (a "failed" attempt), and must infer a latent precondition to succeed. For example, a robot pulling a locked drawer fails, revealing the precondition that the lock must be opened first. The successful action chain is therefore [lock-open, drawer-pull], not just [drawer-pull].

The authors formalize this as Exploratory Manipulation Trace QA (EMT-QA):

• Input: A synchronized stream of (1) manipulation video and (2) proprioceptive trajectory (end-effector pose and gripper state) from an exploratory trace.
• Task: Predict the minimal-success action chain (the fewest actions required to complete the task) implied by the latent precondition revealed in the trace.
• The Challenge: State-of-the-art Vision-Language Models (VLMs) and embodied multimodal Large Language Models (LLMs) fail to reliably recover this chain from raw video, raw proprioception, or their combination. They often miss subtle discriminative signals (e.g., a small change in end-effector orientation or a phase-split in the trajectory) that a human observer can easily detect.

2. Methodology: Closed-Loop Trace Distillation

To address the inability of raw models to "read" these traces, the authors propose Closed-Loop Trace Distillation, a pipeline that generates a Distilled Reading Heuristic (DRH).

Core Concept

Instead of fine-tuning model weights or invoking agents at inference, the system distills the logic required to interpret a trace into a single-line natural language prompt (the DRH). This heuristic acts as a "reading instruction" for a frozen downstream model.

The Training Pipeline

1. Coding Agent: A per-task coding agent (e.g., Claude Code) iterates over labeled training traces. It has access to a fixed toolset (video reader, trajectory analysis, coordinate transforms) but sees only training data and ground-truth chain labels.
2. Candidate Generation: The agent proposes candidate DRHs (e.g., "read the sign of the pitch component of the target orientation").
3. Closed-Loop Evaluation: Candidates are evaluated on held-out groups of traces. A candidate is committed only if it clears a chain-accuracy gate (e.g., 0.85 accuracy) across $K$ consecutive training groups.
4. Artifact: The result is a single, task-specific DRH. No model weights are updated during this process.

Inference

At inference time, the agent is absent. A frozen base VLM receives:

1. The raw trace (video + proprioception).
2. The standard chain prompt (task description + candidate options).
3. The DRH as a prompt entry.

The model then outputs the minimal-success action chain. The authors also demonstrate that the same DRH can serve as the sole specification for a coding agent to one-shot generate programmatic classifiers (e.g., scripted rules or standard ML classifiers) that match the VLM's performance.

3. Key Contributions

1. Formalization of EMT-QA: Defining a multimodal QA task where the probe segment of an exploratory trace reveals a latent precondition determining the minimal-success action chain.
2. Closed-Loop Trace Distillation: A training pipeline that converts labeled traces into a Distilled Reading Heuristic (DRH). This approach requires no model weight updates and no agent calls at inference.
3. Performance Lift: Demonstrating that the DRH lifts chain accuracy by +0.38 to +0.47 over the best raw-modality baseline (Naked-Modality VLM) across three simulator and two real-robot tasks.
4. Dual-Consumer Utility: Showing that the DRH serves as a sufficient specification for both a frozen VLM and programmatic baselines, decoupling the "reading procedure" from the specific inference engine.

4. Experimental Results

The system was evaluated on five tasks: three in simulation (safe, lamp, door) and two on real robots (bottle, cabinet).

• Chain Accuracy:
  • Naked-Modality VLM (Best Baseline): Achieved 0.53–0.62 accuracy across tasks.
  • Distilled-Prompt VLM (With DRH): Achieved 0.93–1.00 accuracy.
  • Improvement: Consistent gains of +0.38 to +0.47 absolute accuracy.
• Modality Ablation:
  • Experiments confirmed that neither the frozen base VLM (GPT-5.5) nor a recent embodied multimodal LLM (HY-Embodied-0.5-X) could recover the performance lift from raw video, raw proprioception, or their combination alone.
  • In some cases (e.g., the "safe" task), the embodied LLM collapsed to below-chance performance (0.048) when combining modalities without the DRH.
• Programmatic Baselines:
  • Using the DRH as the sole specification, a coding agent generated programmatic baselines (scripted rules and standard classifiers) that matched or exceeded the performance of the prompted VLM on held-out splits (e.g., 63/63 accuracy on simulator tasks).

5. Significance and Claims

The paper argues that the primary bottleneck in EMT-QA is not the modality bandwidth (the ability to see or sense) but the reading procedure (knowing how to interpret the trace).

• Decoupling Training and Inference: The DRH is a portable, human-auditable artifact. It allows any inference-time consumer (a frozen VLM or a programmatic script) to inherit the task-level readout without retraining or agent invocation.
• Nature of the Heuristic: The DRH does not provide the answer directly; it provides the procedure (e.g., "check the sign of the delta"). This leaves the perception and final selection to the base model.
• Limitations: The authors note the approach assumes ground-truth labels at training time and relies on heuristics expressible in a single line of natural language. The current scope is limited to single-arm, stationary-base, short-horizon tasks. The paper does not claim applicability to label-free regimes or tasks where the discriminator lies primarily in complex visual features beyond the current empirical grammar.

In summary, the paper posits that for exploratory manipulation, explicitly teaching a model how to read the evidence (via a distilled heuristic) is more effective than relying on the model's implicit ability to infer it from raw data.