CompliantVLA-adaptor: VLM-Guided Variable Impedance Action for Safe Contact-Rich Manipulation

1. Problem Statement

Current state-of-the-art Vision-Language-Action (VLA) models (e.g., RDT, Pi0, OpenVLA) excel at semantic understanding and high-level task planning but fundamentally operate as rigid position-tracking systems. They output desired end-effector positions or trajectories without considering interaction forces.

• The Limitation: In contact-rich tasks (e.g., inserting a peg, closing a drawer, wiping a surface), this rigid execution leads to catastrophic failures when the robot encounters unexpected resistance. Without force adaptation, robots apply excessive force, damaging objects or the robot itself.
• The Gap: There is a disconnect between the high-level semantic reasoning of VLAs and the low-level physical compliance required for safe interaction. Existing Variable Impedance Control (VIC) methods lack the semantic understanding to automatically adapt stiffness and damping based on visual context and task requirements.

2. Methodology: CompliantVLA-adaptor

The authors propose a plug-and-play modular adaptor that augments existing VLA models with a VLM-guided Variable Impedance Control (VIC) system. The architecture consists of three core components:

A. Hybrid Control Architecture

The system operates on three temporal scales to balance reasoning speed with control stability:

1. VLM-Informed Impedance Generation (~1 Hz): A Vision-Language Model (VLM) processes visual observations, natural language instructions, and real-time force/torque feedback to generate context-aware impedance parameters (Stiffness $K$ and Damping $D$).
2. VLA Action Chunk (~3 Hz): The base VLA model generates the desired end-effector displacement ($x_d$) based on the current state and instruction.
3. Low-Level VIC Controller (1000 Hz): A classical VIC controller tracks the VLA's desired pose using the dynamically generated $K$ and $D$ parameters, ensuring safe physical interaction.

B. VLM-Based Impedance Parameter Generation

The system utilizes a frozen VLM (e.g., ChatGPT-4o-mini) as an "in-context impedance coach."

• Multimodal Input: The VLM receives the task description, visual inputs (wrist and global cameras), and real-time force/torque data ($F$).
• Contact Phase Recognition: The VLM classifies the current interaction phase into four categories: Free-motion, Approaching, Contact, and Retreat. This allows the system to apply different impedance hierarchies (e.g., high stiffness for precision in free motion, low stiffness for compliance during contact).
• Anisotropic Parameter Generation: The VLM outputs specific stiffness ($K_x, K_y, K_z$) and damping ($D_x, D_y, D_z$) matrices. It applies logic such as reducing impedance along the primary motion axis while maintaining higher impedance on constraint axes for alignment.

C. Real-Time Force-Regulated Safety Layer

To ensure safety even if the VLM's semantic reasoning is imperfect, a dual-layer safety mechanism is implemented:

• Force Scaling Factor ($\alpha_{force}$): The final stiffness is calculated as $K_{final} = K_{VLM} \cdot \alpha_{force}$. If measured forces exceed safe thresholds, $\alpha_{force}$ reduces the stiffness dynamically.
• Critical Damping: Damping is computed to ensure critical damping ($\zeta = 0.7$) relative to the effective mass and stiffness.
• Safety Threshold: If contact forces exceed 30N for three consecutive readings, the task is terminated to prevent damage.

3. Key Contributions

1. CompliantVLA-adaptor Framework: A novel architecture that bridges high-level semantic reasoning and low-level compliant control, enabling VLA models to generate context-aware impedance parameters without retraining the base VLA.
2. Multimodal Impedance Coaching: A method where a VLM acts as an impedance controller, translating visual-linguistic context and force feedback into anisotropic stiffness and damping values.
3. Safety-Enhanced Execution: The integration of a real-time force-regulated safety layer that ensures interaction forces remain within safe thresholds, fundamentally changing failure modes from "catastrophic force violations" to "graceful degradation" (e.g., minor misalignment).
4. Zero-Shot Generalization: The approach leverages pre-trained VLMs to adapt to novel objects and scenarios without requiring task-specific training data for impedance tuning.

4. Experimental Results

The method was evaluated in both simulation (LIBERO and ManiSkill benchmarks) and real-world hardware (Franka Emika Panda).

• Simulation Performance:

  • Task Suite: 8 contact-rich tasks (peg insertion, drawer manipulation, object placement).
  • Success Rates: The baseline VLA models (RDT, Pi0, OpenVLA-oft) exhibited highly unstable performance under force constraints, with success rates as low as 0% for difficult tasks and an average of 9.86%.
  • Improvement: With the CompliantVLA-adaptor, the average success rate increased to 17.29%, with specific tasks showing dramatic relative improvements (e.g., drawer tasks rising from 0% to ~30-40%).
  • Failure Modes: Baselines failed due to force violations (>30N). The adaptor significantly reduced these violations, shifting failures to minor alignment issues rather than unsafe forces.

• Real-World Performance:

  • Tasks: Pushing a 3kg box and wiping a whiteboard.
  • Force Regulation: The robot successfully modulated stiffness between 400–600 N/m for gentle whiteboard wiping and 800–1000 N/m for pushing the heavy box.
  • Safety: Interaction forces remained within safe limits, with smooth transitions and no abrupt spikes, validating the system's ability to handle physical contact safely.

5. Significance and Conclusion

This work addresses a critical barrier in deploying VLA models in the real world: the lack of physical safety during contact.

• Paradigm Shift: It moves VLA systems from rigid position control to adaptive, force-aware compliance, making them suitable for delicate and uncertain environments.
• Practicality: The "plug-and-play" nature of the adaptor means it can be applied to existing VLA models without the need for massive, task-specific retraining datasets.
• Future Impact: By bridging semantic understanding with physical intelligence, this approach paves the way for safer, more generalizable robotic assistants capable of performing complex manipulation tasks in unstructured human environments.

Limitations Noted: The authors acknowledge that VLM inference latency (approx. 1 Hz) is slower than the control loop (1000 Hz), meaning the VLM provides a "prior" while the low-level force controller handles immediate safety. Additionally, reliance on API-based VLMs incurs cost and latency, and generalization to completely unseen object dynamics remains a challenge.