Technical Summary: CLAW – A Vision-Language-Action Framework for Weight-Aware Robotic Grasping

Problem Statement

Vision-Language-Action (VLA) models have emerged as a powerful paradigm for robotic control, enabling end-to-end policies that map natural language instructions and visual observations to motor actions. However, current VLA architectures often struggle to satisfy precise, numerically constrained task requirements. Specifically, they lack explicit mechanisms for monitoring sensor feedback, such as digital weight thresholds, to determine when to terminate an action. Because VLA action generation is implicitly shaped by statistical correlations in training data rather than explicit condition monitoring, these models often fail to stop grasping at exact weight targets, instead relying on heuristic patterns (e.g., the number of grasp attempts) observed during training. This limitation prevents VLAs from executing tasks requiring rigorous, real-time quantitative control.

Methodology

The authors propose CLAW (CLIP-Language-Action for Weight), a framework that decouples condition evaluation from action generation to address the limitations of end-to-end VLAs in weight-aware manipulation. The system integrates two primary components:

1. Lightweight Prompt Generator (CLIP)

CLAW employs a fine-tuned CLIP model as a specialized, lightweight perception module.

• Function: This module continuously monitors the digital readout of a weight scale via a fixed camera.
• Mechanism: It takes the scale image and the human task instruction (e.g., "load 40g garlic") as input. Based on the visual reading, it outputs a discrete symbolic prompt: either continue or stop.
• Training: The authors constructed a dataset of 2,000 scale images paired with synthetic instructions. CLIP was fine-tuned to classify whether the current weight ($w^*$) is below or above the target weight ($k$), mapping this comparison to the binary prompts. This approach avoids the latency of large generative VLMs, ensuring high-frequency inference suitable for real-time control.

2. Visuomotor Policy ($\pi_0$)

The action generation is handled by $\pi_0$, a state-of-the-art flow-based VLA policy.

• Function: $\pi_0$ receives multi-view camera observations of the workspace and the discrete prompts generated by the CLIP module.
• Mechanism: It conditions its continuous action distribution on both the visual context and the symbolic prompt. The policy generates high-frequency (up to 50Hz) motor commands to manipulate objects.
• Training: The authors collected demonstration episodes where the robot performed grasping and bowl-removal tasks. Crucially, during data collection, frames were manually annotated with the corresponding clip_prompt labels (simulating the CLIP output). $\pi_0$ was then fine-tuned using a flow-matching loss conditioned on these prompts, teaching the policy to distinguish between "continue grasping" and "stop and remove" phases based on the provided language directive.

System Architecture

The framework forms a closed loop:

1. Perception: CLIP analyzes the scale image and the task instruction.
2. Decision: CLIP generates a prompt ($m_t \in \{continue, stop\}$).
3. Action: $\pi_0$ integrates $m_t$ with multi-view observations to generate the next action chunk.
   This design allows the system to combine symbolic reasoning (weight thresholds) with continuous visuomotor control.

Key Contributions

The paper outlines four primary contributions:

1. Framework Design: Introduction of CLAW, which augments standard VLA architectures with a lightweight, task-specialized VLM for explicit condition monitoring, enabling weight-aware manipulation.
2. CLIP Adaptation: A fine-tuning procedure that transforms CLIP into a reliable prompt generator, capable of translating raw scale readings into VLA-understandable language directives (continue/stop).
3. Policy Integration: A fine-tuning strategy for $\pi_0$ using prompt supervision, ensuring the policy effectively integrates symbolic prompts with multi-view observations to produce precise actions.
4. Empirical Validation: Comprehensive evaluation across three scenarios (single-object candy picking, single-object garlic picking, and mixed-object dual-arm manipulation), demonstrating that CLAW outperforms both raw and fine-tuned $\pi_0$ baselines in respecting weight constraints.

Experimental Results

The authors evaluated CLAW against two baselines: Raw-$\pi_0$ (pre-trained without task-specific tuning) and Fine-tuned $\pi_0$ (trained on the specific task but without the CLIP prompt module).

• Success Rates: In single-object tasks (picking 20g, 30g, or 40g of candy or garlic), CLAW achieved a 100% success rate for both the action execution and the correct stop point.
• Baseline Performance:
  • Raw-$\pi_0$ failed to stop at the correct weight (0% stop point success) and had low action success rates (15–35%).
  • Fine-tuned $\pi_0$ achieved 100% action success but failed to consistently stop at the target weight (stop point success ranged from 0% to 35%). The authors observed that this model relied on the number of grasps seen in training rather than the actual scale reading, leading to stochastic stopping behavior.
• Robustness: In mixed-object and disturbance tests (where extra weight was added mid-grasp), CLAW successfully adapted. When the scale reading exceeded the threshold, CLIP immediately switched the prompt to stop, causing $\pi_0$ to abort the grasping motion and initiate bowl removal. This demonstrated the system's ability to handle unexpected input variations and coordinate dual-arm tasks.

Significance and Claims

The paper claims that CLAW addresses a critical gap in current VLA research: the inability to enforce precise, numerically defined constraints in continuous control tasks.

• Decoupling of Concerns: The authors argue that by separating condition monitoring (handled by a lightweight, specialized VLM) from action generation (handled by a flow-based VLA), the system achieves both flexibility and quantitative accuracy.
• Prompt Sensitivity: The results highlight that $\pi_0$ is highly sensitive to language prompts. Even with nearly identical instructions, changing the target object or the continue/stop directive allowed the model to reliably disambiguate and execute distinct behaviors.
• Efficiency: The use of a fine-tuned CLIP model rather than a large generative VLM ensures the system operates at frequencies compatible with real-time robotic control, avoiding the latency issues associated with larger models.

The authors conclude that while VLA models offer strong semantic grounding, they benefit significantly from external, lightweight modules that provide explicit, task-specific constraints. This approach enables robots to move beyond "completing an action" to "completing the task" with rigorous adherence to physical parameters.