Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalities

The paper introduces KARMMA, a lightweight multimodal knowledge distillation framework for egocentric action recognition that maintains high accuracy and robustness against missing modalities without requiring modality alignment during training or inference, making it ideal for on-robot deployment.

The Gist

Imagine you are trying to teach a robot to understand what a human is doing in a kitchen, like "cutting an onion" or "pouring coffee." This is called Egocentric Action Recognition (seeing the world through the robot's eyes).

To do this well, the robot usually needs to "see" (video) and "hear" (audio). But here's the problem: in the real world, things go wrong. The camera might get covered by a hand, the microphone might get muted for privacy, or the battery might die. Most current robot brains are like a student who only knows how to take a test if they have both a pencil and a ruler. If you take away the ruler, they freeze and fail.

The authors of this paper, KARMMA, have built a new way to teach robots that is much more flexible and tough. Here is how they did it, explained simply:

1. The Problem: The "All-or-Nothing" Trap

Most robots today are trained assuming they will always have all their sensors working perfectly.

• The Analogy: Imagine a chef who can only cook a perfect soup if they have fresh tomatoes, basil, and garlic. If the garlic is missing, they throw the whole pot away and say, "I can't cook this."
• The Reality: In robotics, sensors fail constantly. If the robot loses its audio, it shouldn't stop working; it should just rely more on its eyes.

2. The Solution: The "Master Chef" and the "Apprentice"

The team created a two-step learning process called Knowledge Distillation. Think of it like a Master Chef (the Teacher) training a fast, efficient Apprentice (the Student).

• The Teacher (The Master Chef): This is a huge, powerful, and slow computer brain. It has already learned from massive amounts of data. It knows how to combine sight and sound perfectly. However, it's too heavy and slow to run on a small robot.
• The Student (The Apprentice): This is a tiny, lightweight brain designed to fit on a robot. It needs to be fast and use very little battery.

The Magic Trick: Instead of just copying the Master Chef's answers, the Apprentice learns how the Master thinks. But here is the twist: The Master Chef is trained to cook even when ingredients are missing.

3. The Secret Sauce: "Modality Dropout"

To make the Apprentice tough, they don't just let the Master teach with all ingredients present. They play a game of "hide and seek" during training.

• The Analogy: Imagine the Master Chef is teaching the Apprentice. Every few minutes, the Chef says, "Okay, I'm hiding the garlic!" or "I'm hiding the tomatoes!"
• The Result: The Apprentice learns to make a delicious soup using only what is currently available. If the audio is missing, the Apprentice learns to rely on the video. If the video is blurry, it leans on the sound.
• The Innovation: Unlike other methods that require the robot to be retrained every time a sensor changes, this Apprentice learns once and can handle any combination of sensors (Video only? Audio only? Both? Neither?) without needing a software update.

4. Making it Fast: The "Token Reduction"

Robots have limited memory. Processing a video frame by frame is like reading every single word in a book to understand the plot. It takes too long.

• The Analogy: The authors invented a "summary strategy." Instead of reading every single word, the robot learns to group sentences and read the main idea of each paragraph.
• The Result: This cuts the robot's workload in half (using 50% less memory) without making it less smart. It's like reading a "CliffsNotes" version of the book but still getting an A on the test.

5. Why This Matters for Robots

This system, KARMMA, is a game-changer for Human-Robot Interaction (like a robot butler or a helper robot).

• Robustness: If a robot is helping a doctor and the camera gets blocked by a patient's arm, the robot doesn't crash. It just switches to listening to the doctor's voice to keep working.
• Efficiency: It runs on small, cheap chips, meaning you don't need a supercomputer in the robot's head.
• Flexibility: You can swap sensors (e.g., change the camera model) without having to retrain the whole robot brain from scratch.

Summary

The paper introduces KARMMA, a smart training method that teaches a small, fast robot brain how to be a "multitasking superhero." It learns from a giant, powerful teacher but practices with "missing ingredients" so it never fails when a sensor goes offline. It's like teaching a student to solve a math problem whether they have a calculator, a pencil, or just their brain—making robots ready for the messy, unpredictable real world.

Technical Summary

1. Problem Definition

Egocentric action recognition involves interpreting human actions from a first-person perspective (e.g., wearable cameras on robots). While multimodal approaches (combining RGB video, audio, optical flow, etc.) generally outperform unimodal ones, they face two critical challenges in real-world robotics:

1. Missing Modalities: Existing multimodal models assume all sensors are available at inference time. In practice, sensors fail, are occluded, or are muted (e.g., privacy constraints), causing significant accuracy drops or total system failure.
2. Computational Cost: Processing multiple modalities simultaneously is computationally expensive, making it difficult to deploy on resource-constrained edge devices or robots.
3. Training Rigidity: Most methods require strict modality alignment during training (i.e., every sample must have all modalities), limiting flexibility when sensor configurations vary.

2. Methodology: The KARMMA Framework

The authors propose KARMMA (multimodal Knowledge distillation framework for egocentric Action Recognition robust to Missing ModAlities), a multimodal-to-multimodal distillation pipeline.

A. Architecture

The framework consists of a Teacher and a Student, both designed to handle arbitrary subsets of modalities:

• Feature Extractors (FEs):
  • Teacher: Uses frozen, pre-trained unimodal encoders (e.g., Swin-B for video, AST for audio) to avoid retraining costs.
  • Student: Uses smaller, trainable variants of these encoders (e.g., Swin-T, AST-T) initialized from pre-trained weights.
• Fusion Block (FB): A transformer-based module that fuses tokens from different modalities. It outputs a [CLS] token (aggregating cross-modal info) and averaged tokens per modality.
• Token Reduction (Θ-Average): To mitigate the quadratic computational cost of self-attention, the authors introduce a parameter-free strategy. If a modality produces $k$ tokens, they are partitioned into $\Theta$ groups and averaged. This caps the token count without learnable parameters.

B. Key Mechanisms for Robustness

1. Modality Dropout: Applied to both Teacher and Student during training. Entire modalities are randomly dropped with probability $p$ (ensuring at least one remains). This forces the model to learn robust representations without relying on a specific sensor.
2. Missing Modality Strategy: To handle missing inputs effectively, the Student's embedding layer introduces two types of learnable tokens:
   • Modality-specific tokens ($\breve{t}_m$): Act as positional encodings to distinguish which modality is present.
   • Token-specific tokens ($\bar{t}^m_i$): Compensate for the absence of a modality by providing learned context when that modality is missing.
3. Knowledge Distillation:
   • Stage 1: Train the Teacher using Cross-Entropy loss on available modalities.
   • Stage 2: Freeze the Teacher and distill knowledge to the Student using Kullback-Leibler (KL) divergence between their class probability distributions, combined with Cross-Entropy loss.
   • Loss Function: $L_S = \alpha L_{CE} + (1-\alpha) L_{KL}$, where $\alpha$ balances task supervision and teacher guidance.

3. Key Contributions

1. Modality-Agnostic Training: The framework does not require modality alignment across samples. It can train on datasets where different samples have different subsets of modalities.
2. Robust Student Model: The resulting student is lightweight, fast, and capable of inferring on any subset of the trained modalities without retraining.
3. Efficient Integration: By using frozen pre-trained encoders for the teacher, the system simplifies the integration of new encoders as they become available.
4. Parameter-Free Efficiency: The $\Theta$-Average token reduction strategy significantly lowers memory and compute costs without sacrificing accuracy.

4. Experimental Results

The method was evaluated on Epic-Kitchens-100 and Something-Something V2.

• Accuracy vs. Robustness:
  • The KARMMA Student (KARMMAS) outperformed both a standard baseline and a baseline with dropout strategies across most modality combinations.
  • Missing Modality Performance: When modalities were dropped at inference (simulating sensor failure), KARMMAS showed significantly less performance degradation than baselines. For example, on Something-Something with only Object Detection annotations, KARMMAS achieved a 36.74% absolute gain over the baseline with dropout.
  • Runtime Dropouts: Simulating 90% random sensor dropouts at inference, KARMMAS maintained high accuracy, whereas baselines suffered massive drops (up to 32%).
• State-of-the-Art Comparison:
  • Compared to the SOTA multimodal-to-unimodal distillation method (Radevski et al.), KARMMA's multimodal student achieved higher accuracy (43.00% vs. 41.81% on Epic-Kitchens with full modalities) while supporting flexible inference on any modality subset.
• Resource Efficiency:
  • The Student uses approximately 50% fewer computational resources (memory and GFLOPs) than the Teacher.
  • The token reduction strategy reduced memory usage by 81.45% with only a negligible accuracy drop (0.27%).

5. Significance

KARMMA addresses a critical gap in robotics and human-robot interaction (HRI): the need for perception systems that are both accurate and resilient.

• Real-World Deployment: It enables robots to operate reliably even when sensors fail or are intentionally disabled (privacy), without requiring retraining for every possible sensor configuration.
• Edge Computing: By distilling a heavy teacher into a lightweight student and optimizing token usage, the model is suitable for deployment on resource-constrained edge devices.
• Scalability: The use of frozen encoders and modality-agnostic training makes the framework highly adaptable to new datasets and sensor types.

In conclusion, KARMMA provides a robust, efficient, and flexible solution for egocentric action recognition, ensuring that multimodal systems remain functional and accurate in the unpredictable environments typical of robotics.