Technical Summary: Inference-time Policy Steering via Vision and Touch (ViTaL)

Problem Statement

Inference-time steering adapts pre-trained generative robot policies (e.g., diffusion policies) during deployment without re-training by verifying or refining candidate actions before execution. While recent methods successfully utilize visual observations for long-horizon task verification, vision alone is often insufficient for contact-rich manipulation. In tasks such as pipetting, wiping, or insertion, success depends on both global semantic progress (observable via vision) and subtle local physical interactions, such as contact force and grasp stability (observable via touch).

Current approaches face two primary challenges in integrating these modalities:

1. Modality-dependent Observability Across Temporal Scales: Visual outcomes often require long-horizon predictions to distinguish semantic modes (e.g., moving toward the correct cup), whereas tactile outcomes are local and transient (e.g., force application during a squeeze), often obscured in long-horizon visual predictions.
2. Multi-modal Reward Modeling: Defining a unified reward function is difficult because the relative importance of visual and tactile outcomes shifts across task phases. A single reward function struggles to balance global task progress with fine-grained contact requirements (e.g., distinguishing gentle grasping from dropping).

Methodology: ViTaL Framework

The authors propose ViTaL (Visuo-Tactile Latent Steering), a framework that formulates multimodal guidance as a bi-level optimization problem. This approach decomposes steering into two stages that align with the natural timescales of vision and touch:

1. Bi-level Optimization Formulation

The objective is to select an action sequence $a^*_{t:t+h}$ that maximizes a task-conditioned reward while adhering to a visual prior.

• Inner Optimization (Visual Mode Selection): The system samples long-horizon action sequences ($H$ steps) from the base policy. These are rolled out using a visuo-tactile latent world model to predict future states. A visual verifier scores these predictions based on semantic progress toward the task goal. The sequence with the highest visual reward, $\bar{a}_{t:t+H}$, is selected as the global behavior mode.
• Outer Optimization (Tactile Refinement): The first $h$ steps of the visually selected sequence serve as an "action anchor." The system then performs short-horizon ($h$ steps) diffusion editing to refine this anchor. A tactile verifier scores the predicted tactile futures, and the policy is reweighted to favor actions that satisfy local contact constraints while remaining close to the visual plan.

2. Visuo-Tactile Latent World Model

To support outcome-based steering, ViTaL employs a learned latent world model ($p_\phi$) that predicts future visual and tactile states directly in latent space.

• Architecture: It uses frozen encoders (DINOv3 for vision, AnyTouch for tactile) to map observations to latent states ($z_v, z_\tau$). An action-conditioned transformer predicts future joint latents recursively.
• Efficiency: By operating in latent space, the model avoids the computational cost of decoding full images or tactile maps for every candidate action, enabling efficient text-conditioned guidance.

3. Semantically-Aligned Verifiers

ViTaL uses language as a shared task representation to bridge visual and tactile reasoning. A Vision-Language Model (VLM) decomposes high-level instructions into phase-level textual objectives ($\ell^v_p, \ell^\tau_p$).

• Visual Verifier: Decodes predicted visual latents into RGB images and uses a zero-shot reward model (ROBOMETER) to evaluate semantic progress (e.g., "is the robot moving toward the blue cup?").
• Tactile Verifier: Computes the cosine similarity between the predicted tactile latent embedding and the text embedding of the contact objective (e.g., "grasp lightly"). This constitutes a text-conditioned tactile reward, allowing the system to score tactile futures without hand-designing task-specific reward functions or decoding tactile images.

4. Inference-time Optimization

The steering process combines sampling-based visual selection with diffusion editing for tactile refinement:

• Visual Stage: Samples $N$ long-horizon trajectories, rolls them out in the world model, and selects the best based on the visual verifier.
• Tactile Stage: Treats the selected visual anchor as a condition for diffusion editing. It partially noises the anchor and biases the denoising process toward actions with higher tactile rewards, effectively refining the contact execution.

Key Contributions

1. Bi-level Multi-modal Guidance: A framework that uses vision for long-horizon semantic mode selection and touch for targeted short-horizon contact refinement, addressing the temporal scale mismatch between modalities.
2. Language-Conditioned Tactile Reward: The introduction of a novel tactile verifier that scores predicted tactile futures in an aligned latent space using text descriptions. This enables scalable, demonstration-free tactile verification that is flexible to different task phases.
3. Latent World Modeling: The integration of a visuo-tactile latent world model that allows for efficient, direct scoring of multimodal outcomes in latent space without decoding.

Experimental Results

The authors evaluated ViTaL on three real-world contact-rich manipulation tasks using a Franka Emika robot with GelSight Mini sensors: pipette transfer, wiping, and peg insertion.

• Performance Gains: ViTaL improved overall task success by 51% over the base policy. It outperformed unimodal steering (vision-only or touch-only) by at least 33% and exceeded naive multimodal fusion (simple reward summation) by at least 20%.
• Failure Modes: Vision-only steering often succeeded in global alignment but failed due to local contact errors (e.g., spilling liquid). Tactile-only steering improved contact but often missed the global goal. ViTaL successfully addressed both failure modes.
• Reward Accuracy: The proposed verifiers achieved high preference-order accuracy (approx. 85% for visual, 81% for tactile) when comparing predicted futures against ground truth, demonstrating the reliability of the latent world model and semantic verifiers.
• Efficiency: The bi-level approach provided a better trade-off between success and inference speed compared to naive fusion, adding only 0.05 seconds per inference step while significantly reducing failure rates.

Significance and Claims

The paper claims that ViTaL represents a significant step forward in making pre-trained robot policies robust for contact-rich tasks without requiring re-training or extensive task-specific reward engineering. By decomposing the steering problem into visual mode selection and tactile refinement, the method leverages the complementary strengths of vision (global context) and touch (local precision).

The authors highlight that their text-conditioned tactile reward is the first of its kind for robotic manipulation, enabling flexible, language-driven contact reasoning. They acknowledge limitations, noting that the system's fidelity depends on the latent world model (where compounding errors can affect verification) and that tactile encoders are currently pretrained on smaller datasets compared to modern VLMs. However, the results suggest that multimodal inference-time steering is a viable path toward more adaptable and reliable robotic manipulation.