Technical Summary: Splash – Mask-Isolated Tactile Alignment Learning in MLLMs

1. Problem Statement

Multimodal Large Language Models (MLLMs) are increasingly used for embodied robotics, where understanding physical properties like friction and compliance is crucial. However, equipping compact MLLMs (sMLLMs, e.g., ≤3B parameters) with tactile reasoning capabilities presents a significant challenge: catastrophic forgetting.

Existing approaches that integrate tactile modalities often require updating the entire model or large parameter subsets. In resource-constrained sMLLMs, this optimization for tactile signals distorts pretrained visual-language (VL) features, causing the model to lose its general-purpose reasoning abilities. As demonstrated in the paper, even small amounts of tactile training data can degrade VL performance, creating a zero-sum trade-off between acquiring new sensory modalities and preserving established reasoning capabilities.

2. Methodology: The Splash Framework

The authors propose Splash, a mask-isolated tactile alignment framework designed to integrate tactile sensing into sMLLMs without additional inference overhead or catastrophic forgetting. The core philosophy is to confine all tactile adaptation to a "dormant" (less important) parameter subspace while keeping "critical" parameters frozen.

2.1 Locating the Dormant Subspace

Splash leverages the hypothesis of natural sparsity in pretrained models, where only a subset of parameters is critical for maintaining function.

• Importance Metric: Using a small calibration set from vision-language data (e.g., CC12M), the method computes a visual-relative importance score ($S_{i,j}$) for each weight in the LLM's linear layers. The score is defined as the product of the weight magnitude and the $\ell_2$-norm of the corresponding input activation:
  $$S_{i,j} = |W_{i,j}| \cdot \|x_i\|_2$$
• Mask Generation: Based on a sparsity ratio ($s\%$), a binary mask ($\mathbf{M}$) is generated. Parameters with scores below a threshold are marked as "dormant" (trainable), while those above are "critical" (frozen).
• Stability Constraints: The first and last transformer blocks are excluded from masking to preserve input grounding and high-level semantic representations, ensuring training stability.

2.2 Mask-Isolated Tactile Alignment

Unlike previous two-stage approaches (aligning encoders first, then fine-tuning the LLM), Splash employs a unified, single-stage training process.

• Architecture: The model consists of a frozen visual front-end, a trainable tactile front-end (a ViT-Tiny initialized from ImageNet), and the LLM backbone.
• Training Objective: The model minimizes the cross-entropy loss for generating tactile descriptions given RGB images, contact images, and text prompts.
• Parameter Updates:
  • Critical Parameters: Frozen ($\nabla \mathcal{L} = 0$). These act as stable anchors for the pretrained VL manifold.
  • Dormant Parameters: Updated via gradient descent using the mask ($\theta \leftarrow \theta - \eta (\mathbf{M} \odot \nabla_\theta \mathcal{L})$).
  • Tactile Front-end: Fully trainable.
• Mechanism: By strictly isolating updates to the dormant subspace, Splash prevents the perturbation of the entire parameter manifold, effectively avoiding catastrophic forgetting without introducing external modules like adapters (e.g., LoRA) that add inference latency.

3. Key Contributions

1. Splash Framework: A novel parameter-isolated adaptation method that integrates tactile sensing into sMLLMs with zero additional inference overhead, mitigating vision-related catastrophic forgetting.
2. Unified Training: A single-stage training scheme that concurrently optimizes the tactile front-end and the dormant LLM subspace, eliminating the need for multi-stage alignment procedures.
3. Performance Preservation: Demonstrated ability to achieve state-of-the-art (SOTA) tactile reasoning on small-scale models (1B and 3B) while preserving the models' original general-purpose VL capabilities.

4. Experimental Results

The authors evaluated Splash on three Visuo-Tactile-Language (VTL) benchmarks (SSVTP, TVL, TacQuad) and five general-purpose Vision-Language (VL) benchmarks (MMMU, MathVista, MME, MMBench).

• Tactile Reasoning (VTL Benchmarks):

  • Splash-3B (based on Qwen2.5-VL-3B) achieved an average score of 4.91, outperforming the strongest baseline, TVL-LLaMA (7B backbone), which scored 4.31.
  • Splash-1B (based on InternVL2.5-1B) achieved an average of 5.01, surpassing UniTouch (7B backbone, 3.74) and TVL-Qwen (4.50).
  • Splash demonstrated superior robustness under distribution shifts (e.g., TacQuad), where other methods like (IA)3 suffered significant performance drops.

• Capability Preservation (VL Benchmarks):

  • Splash-3B maintained or improved performance on general VL tasks compared to its zero-shot baseline and other adapted models. For instance, it improved MMMUval from 50.0 (TVL-Qwen) to 55.3 and MathVista from 52.8 to 65.3.
  • In contrast, methods like (IA)3 that preserved VL ability often failed to learn tactile dynamics effectively (under-fitting), while full fine-tuning or LoRA often degraded VL performance.

• Ablation Studies:

  • Sparsity Ratio: A 60% sparsity ratio (60% of parameters frozen) provided the optimal trade-off between tactile learning and VL preservation. 100% sparsity (full tuning) caused catastrophic forgetting, while very low sparsity limited tactile learning.
  • Calibration Data: The method proved robust to the choice of calibration dataset, performing well even with unpaired noise data, suggesting it captures intrinsic architectural redundancies rather than overfitting to specific semantic alignments.

5. Significance and Claims

The paper claims that Splash offers a practical path toward high-level tactile intelligence for resource-constrained edge robots. By identifying and reactivating underutilized "dormant" weights, the framework allows compact models to internalize complex contact dynamics without sacrificing their foundational reasoning abilities.

The authors emphasize that Splash achieves non-destructive modality expansion. Unlike additive parameter-efficient fine-tuning (PEFT) methods that can perturb the entire parameter space, Splash's structural isolation ensures that pretrained VL reasoning pathways remain fundamentally untouched. This enables the deployment of sophisticated visuo-tactile reasoning on small models (≤3B) that are suitable for real-world robotic edge deployment, overcoming the computational bottlenecks of larger 7B+ models.