💻 computer science

Multisensory Continual Learning: Adapting Pretrained Visuomotor Policies to Force

This paper proposes MuSe, a multisensory continual learning framework that effectively adapts pretrained vision-only robot policies to new force-torque modalities using multi-stage fusion and experience replay, thereby enhancing performance on contact-rich tasks without forgetting original capabilities.

Original authors: Jaden Clark, Changhao Wang, Yihuai Gao, Seongheon Hong, Hojung Choi, Mark Cutkosky, Yifan Hou, Shuran Song

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jaden Clark, Changhao Wang, Yihuai Gao, Seongheon Hong, Hojung Choi, Mark Cutkosky, Yifan Hou, Shuran Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have trained a robot to be a master chef, but you only taught it using eyes. It learned to chop, stir, and plate food by watching thousands of videos. It's great at seeing what to do, but it doesn't know how hard to press or when something is slipping.

Now, imagine you want to teach this robot a new skill: feeling. You want it to use a force sensor (like a sensitive touch) to handle delicate tasks, like wiping a vase without breaking it or screwing a lightbulb in without cracking it.

Here's the problem: You don't have a massive library of videos that include both the robot's eyes and its new "touch" sensors. You only have a tiny, new dataset with these new sensors. If you just try to teach the robot with this small new data, it might get confused and forget everything it learned about seeing. This is called "catastrophic forgetting."

This paper introduces a solution called MuSe (Multisensory Continual Learning). Think of MuSe as a clever study guide that helps the robot learn a new sense without forgetting its old ones.

Here is how MuSe works, using simple analogies:

1. The "Two-Way Street" Connection (Multi-Stage Fusion)

Usually, when you add a new sense to a robot, you just tack it on at the end, like adding a side dish to a meal. MuSe is different. It creates a two-way street between the robot's eyes and its new touch sensors.

Early Fusion: It mixes the "touch" data with the "sight" data right at the beginning, like mixing flour and eggs before baking. This lets the robot understand how touch and sight relate to each other immediately.
Late Fusion: It also checks in later in the process, like a taste-tester adjusting the seasoning halfway through cooking.
The Result: The robot doesn't just "see" or "feel"; it learns a unified understanding where sight and touch help each other.

2. The "Future Crystal Ball" (Multisensory Future Prediction)

To make sure the robot really understands the new sense, MuSe doesn't just ask it to "do the task." It asks the robot to predict the future.

The robot is trained to guess: "What will I see in the next second? What will I feel in the next second? What action should I take?"
The Analogy: Imagine a driver learning to drive in the rain. Instead of just driving, they are asked to predict, "If I turn the wheel now, will the car skid?" and "Will the windshield wipers clear the water?"
By forcing the robot to predict both the visual scene and the force signals, it builds a deep, internal map of how the physical world works. This helps it generalize, meaning it can use its new "touch" skills even on tasks it hasn't seen before.

3. The "Flashcard Review" (Experience Replay)

This is the most critical part for preventing the robot from forgetting. When the robot learns the new "touch" skills, MuSe forces it to keep practicing its old "sight" skills at the same time.

The Analogy: Imagine a student learning a new language (French) while trying to keep their Spanish skills sharp. If they only study French for a month, they might forget Spanish. MuSe is like a teacher who says, "For every hour you study French, you must spend 30 minutes reviewing Spanish."
In the robot's case, it mixes the tiny new "touch" data with the huge old "sight" data. When the robot sees a task where it doesn't have touch data (because the old data didn't have it), MuSe simply covers up the "touch" part of the question and lets the robot answer based on sight alone. This keeps the old skills alive.

What Did They Find?

The researchers tested this on a real robot arm.

Forward Transfer (Learning New Things): The robot became much better at contact-heavy tasks (like wiping a vase or inserting a peg) using the new touch sensors. It didn't just learn the task; it learned how to feel its way through it.
Backward Transfer (Not Forgetting): Surprisingly, after learning to use the touch sensors, the robot actually got better at its original vision-only tasks. It seems that learning to "feel" helped it understand the physics of the world better, which made its "sight" skills sharper too.
Cross-Modal Generalization: Even on tasks where they never showed the robot the touch data during training, the robot could still predict what the forces would have been. It learned a general concept of "force" that it could apply anywhere.

The Bottom Line

MuSe shows that you don't need a massive, perfect dataset of every possible sensor to train a robot. You can take a robot that is already an expert at seeing, give it a small amount of new "touch" data, and use this special training method to upgrade it. The robot learns the new sense, keeps its old skills, and actually becomes smarter overall.

Technical Summary: Multisensory Continual Learning (MuSe)

Problem Statement

Robot manipulation, particularly in contact-rich scenarios, often relies on sensory data beyond vision, such as force-torque (F/T), tactile, or audio feedback, to infer interaction states like slippage or contact forces that are not directly visible. However, large-scale multisensory datasets are scarce, and sensors are often hardware- or task-specific, making it impractical to pretrain policies with every potential sensor modality.

This paper formalizes the challenge as multi-sensory continual learning: adapting a pretrained robot policy (trained on vision-only data) to incorporate new sensory modalities for new tasks while preserving performance on the original tasks. The goal is to achieve three specific objectives:

Forward Transfer: Improve performance on new tasks using the new modality compared to training from scratch.
Backward Transfer: Retain or improve performance on pretraining tasks after learning new tasks with new sensory data.
Cross-Modal Generalization: Understand and utilize a newly introduced modality on tasks where it was never observed during training.

Naive fine-tuning on new multisensory data often leads to catastrophic forgetting of pretraining skills and fails to properly integrate the new input dimensions without over-relying on them.

Methodology: MuSe Framework

The authors propose MuSe (Multi-Sensory continual learning), a framework designed to efficiently integrate new sensory modalities into pretrained vision-action policies. The framework is instantiated by adding force-torque sensing to a pretrained Unified Video-Action (UVA) model. MuSe relies on three critical components:

1. Multi-Stage Fusion

To balance deep cross-modal interaction with the preservation of pretrained representations, MuSe employs a dual-pathway fusion architecture:

Early Fusion: The new modality (e.g., F/T) is encoded by a modality-specific encoder and projected into the shared embedding space. These tokens are concatenated with the existing observation and action tokens before being processed by the backbone transformer. This allows the backbone to perform cross-modal attention throughout all layers.
Late Fusion: Features from the new modality are injected into intermediate transformer layers via lightweight cross-attention adapters.
This combination ensures the new modality has direct access to the sequence-processing pathway while also receiving modality-specific conditioning, preventing the new sensor from overwriting visual-action knowledge.

2. Multisensory Future Prediction (World Modeling)

MuSe is trained with a flexible future prediction objective that encourages a unified, physically grounded latent space. Instead of only predicting actions, the model predicts:

Future actions ( $L_{act}$ ).
Future original observations (e.g., video frames, $L_{obs}$ ).
Future values of the newly introduced modality ( $L_{new}$ , e.g., F/T signals).

The total loss is a weighted sum: $L = \lambda_a L_{act} + \lambda_o L_{obs} + \lambda_{n+1} L_{new}$ . Predicting the new modality forces the representation to encode sensor data, while predicting future images anchors the representation to visual dynamics shared across datasets.

3. Experience Replay

To mitigate catastrophic forgetting during fine-tuning, MuSe utilizes experience replay. Training batches are sampled from a mixture of the new multisensory dataset ( $D_{new}$ ) and the original pretraining dataset ( $D_{pre}$ ).

For $D_{new}$ samples, the model receives full multisensory input and is supervised on all prediction losses.
For $D_{pre}$ samples, the new modality is absent. The input is replaced with a learned "modality-mask" token, and the loss for the new modality ( $L_{new}$ ) is masked out, while supervision for actions and original observations is retained.

Experimental Setup and Results

The authors evaluated MuSe on real-world manipulation tasks using a UVA backbone pretrained on 1,271 episodes from 21 vision-only tasks. The model was then fine-tuned on 434 episodes from five contact-rich tasks with F/T supervision.

Key Results

Cross-Modal Generalization: MuSe demonstrated the ability to predict F/T signals on pretraining tasks where F/T was recorded but never used for supervision. MuSe achieved the lowest L2 prediction error (8.42) compared to ablations (e.g., removing experience replay increased error to 12.38).
Forward Transfer: On contact-rich fine-tuning tasks, MuSe significantly outperformed baselines:
- Vase Wiping: 11.5/15 success (MuSe) vs. 5/15 (No F/T) and 8/15 (No Pretrain).
- Peg Insertion: 13/15 success (MuSe) vs. 9/15 (No F/T) and 6/15 (No Pretrain).
  MuSe effectively used F/T to regulate contact forces and recover from jams, whereas the "No F/T" baseline failed due to poor contact regulation.
Backward Transfer: MuSe preserved or improved performance on original pretraining tasks compared to the vision-only baseline:
- Whiteboard Wiping: 12.5/15 (MuSe) vs. 8.5/15 (Pretraining Only).
- Peg Insertion: 10/15 (MuSe) vs. 8/15 (Pretraining Only).
  Crucially, the "No Experience Replay" (No ER) baseline suffered severe forgetting (e.g., 0.5/15 on whiteboard wiping), highlighting the necessity of replay for retaining visual skills.

Significance and Claims

The paper claims that MuSe demonstrates that a modest multisensory dataset can enhance general model capability beyond the fine-tuning distribution. Specifically:

Generalizability of F/T: Force-torque sensing, even when learned from limited data, can be predicted and exploited across a broad range of tasks, including those where it was never explicitly supervised.
Enhanced Representation: Conditioning on F/T enables richer representation learning for contact-rich behaviors, while predicting future F/T provides an estimate of the compliance profile required for precise interaction.
Practical Deployment: The approach allows robot policies to be adapted post-hoc to new sensors without requiring large-scale multisensory pretraining from the start, addressing the hardware and data scarcity constraints of real-world robotics.

The authors note limitations, including the focus on a single sensor expansion (F/T) and the reliance on access to original pretraining data for replay, suggesting future work should explore other modalities and methods that do not require storing prior data.