🤖 machine learning

COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition

This paper introduces COMODO, a cross-modal self-supervised distillation framework that transfers semantic knowledge from video to IMU sensors without requiring labels, thereby enabling efficient, privacy-preserving, and high-performance egocentric human activity recognition on wearable devices.

Original authors: Baiyu Chen, Wilson Wongso, Zechen Li, Yonchanok Khaokaew, Hao Xue, Flora Salim

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Baiyu Chen, Wilson Wongso, Zechen Li, Yonchanok Khaokaew, Hao Xue, Flora Salim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Rich but Clumsy" vs. the "Poor but Agile"

Imagine you are trying to teach a robot to understand human activities (like cooking, dancing, or fixing a bike). You have two teachers to choose from:

The Video Teacher (The Rich, Clumsy Giant):
- Pros: This teacher has seen millions of videos. It understands everything. It knows the difference between chopping onions and chopping wood just by looking at the motion. It's incredibly smart.
- Cons: It's a giant. It eats a lot of electricity (battery), it's heavy, and it's a bit of a privacy nightmare (it's always recording you). You can't carry this giant in your pocket to run a marathon or wear it while sleeping.
The IMU Teacher (The Poor, Agile Athlete):
- Pros: This is a tiny sensor (like the accelerometer in your smartwatch). It's super light, uses almost no battery, and is private (it just feels movement, it doesn't "see" you). It's perfect for wearing 24/7.
- Cons: It's "illiterate." It hasn't seen many examples of human movement. It's like a student who has only read three books. It struggles to tell the difference between complex activities because it lacks the "big picture" knowledge.

The Dilemma: We want the intelligence of the Video Teacher but the efficiency of the IMU Teacher. But usually, you can't have both.

The Solution: COMODO (The "Smart Tutor" System)

The authors created a system called COMODO. Think of it as a master tutoring program that lets the "Poor but Agile" athlete learn directly from the "Rich but Clumsy" giant, without the giant ever needing to be present during the actual game.

Here is how it works, step-by-step:

1. The Training Camp (The "Shadow" Phase)

Imagine the Video Teacher and the IMU Student are in a gym together.

The Video Teacher watches a video of someone baking a cake. It says, "Ah, I see the specific rhythm of whisking and the heat of the oven."
The IMU Student feels the vibrations of the same action.
The Magic: The Video Teacher doesn't just say "Good job." Instead, it creates a map of relationships. It tells the student: "This movement feels like 'whisking,' which is similar to 'stirring soup,' but very different from 'typing on a keyboard'."

2. The "FIFO Queue" (The Infinite Blackboard)

Usually, when you train a student, you show them one example at a time. But COMODO uses a special trick called a FIFO Queue (First-In, First-Out).

Imagine a massive blackboard that never gets erased. Every time the Video Teacher sees a new activity, it writes a note on the blackboard.
As the blackboard fills up, the oldest notes slide off the back to make room for new ones.
Why this matters: This gives the IMU Student a huge, diverse library of examples to compare against. It helps the student understand that "walking" looks different from "running," even if the sensor feels similar. It stabilizes the learning process, like a teacher who keeps a consistent curriculum rather than jumping randomly between topics.

3. The "Soft" Lesson (Not Just Right or Wrong)

Old methods tried to force the student to match the teacher exactly (like a rigid drill sergeant). COMODO is more like a mentor.

Instead of saying "This is 100% baking," COMODO says, "This movement shares 40% of the DNA with baking, 30% with cooking, and 10% with cleaning."
It teaches the IMU sensor to understand the structure of human movement, not just the labels. This makes the sensor much smarter at guessing new things it hasn't seen before.

4. The Graduation (Real-World Use)

Once the training is done, the Video Teacher is sent home. It's too big and expensive to keep around.

The IMU Student graduates. It now carries all the "wisdom" of the Video Teacher in its tiny brain.
Now, you can wear the sensor on your wrist. It runs on a tiny battery, respects your privacy, and can still tell you, "You are currently gardening," or "You are fixing a leak," with high accuracy.

Why This is a Big Deal

No More "Labeling" Hell: Usually, to teach a sensor, humans have to watch hours of footage and manually type "This is walking, this is running." This is expensive and boring. COMODO does this automatically using the video data as a guide, so we don't need to manually label the sensor data.
It Works Everywhere: The paper tested this on different datasets (different people, different cameras, different sensors). It worked great even when the student was tested on a completely new group of people it had never met. It's like a student who learns the principles of math rather than just memorizing the answers to one specific test.
Efficiency: It allows us to have "Super AI" on our wrists without draining our batteries in an hour.

The Bottom Line

COMODO is a bridge. It takes the massive, expensive, high-quality knowledge of video AI and distills it down into a tiny, efficient, privacy-friendly sensor. It's like taking a library of encyclopedias and compressing them into a single, smart pocket watch that knows exactly what you're doing, all while you sleep, run, or cook.

1. Problem Statement

The paper addresses a fundamental trade-off in Egocentric Human Activity Recognition (HAR) for wearable systems:

Video-based Models: Offer rich semantic information and high accuracy but suffer from high power consumption, privacy concerns, dependence on lighting, and frequent interruptions, making them impractical for continuous, on-device deployment.
IMU-based Models: Are energy-efficient, privacy-preserving, and robust to lighting/occlusion, making them ideal for wearables. However, they suffer from data scarcity. Unlike computer vision, there are no large-scale annotated IMU datasets. Annotating IMU data is labor-intensive and costly, leading to poor generalization and weak performance in deep learning models.

Goal: To bridge this gap by transferring the rich semantic knowledge from large-scale video data to lightweight IMU sensors without requiring labeled IMU data, thereby enabling efficient, high-performance, and privacy-preserving HAR.

2. Methodology: COMODO

The authors propose COMODO (CrOss-MOdal video-to-imu DistillatiOn), a self-supervised cross-modal knowledge distillation framework.

Core Architecture

Teacher (Video): A pre-trained, frozen video encoder (e.g., TimeSformer or VideoMAE) processes video windows. It acts as the source of rich semantic knowledge.
Student (IMU): A pre-trained but trainable time-series encoder (e.g., Mantis or MOMENT-small) processes IMU signals. Its goal is to learn to mimic the semantic structure of the video embeddings.
Projection Heads: Both encoders are followed by MLP projectors to map features into a shared embedding space, followed by L2 normalization.

Key Mechanism: Cross-Modal Distribution Distillation

Unlike standard contrastive learning (InfoNCE) which aligns specific positive pairs and treats all negatives equally, COMODO aligns the similarity distributions between modalities.

Dynamic Instance Queue (FIFO): A First-In-First-Out queue stores a large pool of video embeddings from previous batches. This provides a stable, diverse reference distribution and introduces an implicit temporal prior, ensuring the target distribution evolves smoothly during training.
Similarity Distribution Alignment:
- For a given IMU sample, the model computes similarity scores against all video embeddings in the queue.
- It calculates a probability distribution $P_x(i|Q)$ for the IMU sample over the queue.
- It calculates a corresponding distribution $P_v(i|Q)$ for the video sample over the same queue.
- Loss Function: The model minimizes the Cross-Entropy between these two distributions:
  $L_{CE} = -\sum_{i=1}^{N} P_v(i|Q) \log P_x(i|Q)$
- Theoretical Advantage: This is equivalent to minimizing the KL-divergence between the teacher's similarity space and the student's. Unlike "hard" contrastive learning (one-hot targets), this "soft" distillation preserves the semantic structure of the teacher. If two video samples are semantically similar (even if not the exact same instance), the IMU student is encouraged to reflect that relationship, rather than pushing all non-matching samples away uniformly.

Inference

During inference, the video encoder, projectors, and queue are discarded. Only the lightweight IMU encoder is used, ensuring low latency and high efficiency on wearable devices.

3. Key Contributions

Self-Supervised Cross-Modal Transfer: A novel framework that transfers semantic knowledge from video to IMU without requiring any labeled IMU data, solving the data scarcity problem.
Effective Cross-Modal Queuing: Introduction of a FIFO queue mechanism that maintains a diverse reference distribution and stabilizes the optimization process via a temporal prior.
Model-Agnostic Design: The framework is compatible with various pre-trained video and time-series foundation models, allowing for flexible teacher-student configurations.
Superior Generalization: Demonstrates strong cross-dataset generalization, outperforming supervised baselines on unseen datasets with different activity classes and sensor configurations.

4. Experimental Results

The authors evaluated COMODO on three egocentric HAR datasets: Ego4D, EgoExo4D, and MMEA.

Performance vs. Supervised Models: COMODO consistently matches or surpasses fully supervised time-series models (e.g., Mantis, Attend and Discriminate) and significantly outperforms other self-supervised methods (e.g., CrossHAR, IMU2CLIP).
- Example: On Ego4D, COMODO achieved 59.13% A@1, outperforming the best supervised baseline (Mantis) by 0.77% and the previous SOTA self-supervised method (IMU2CLIP) by 4.3%.
Cross-Dataset Generalization: In zero-shot transfer scenarios (e.g., training on Ego4D, testing on MMEA), COMODO significantly outperformed supervised models fine-tuned on the source data, proving it learns transferable semantic structures rather than dataset-specific correlations.
Efficiency:
- Inference: COMODO achieves a massive reduction in computational cost compared to video models. It requires 1.70G FLOPs and 75.7ms latency (vs. 196G FLOPs and 3.5s for TimeSformer), while retaining ~79% accuracy.
- Training: Training is efficient because the video encoder is frozen and features are pre-computed; only the lightweight IMU encoder is updated.
Ablation Studies:
- Distillation Strategy: COMODO (distribution distillation) outperformed L2 distance minimization and standard InfoNCE contrastive learning, validating the importance of preserving the teacher's similarity structure.
- Queue Size: Optimal queue sizes vary by dataset complexity (e.g., 16k for Ego4D, 1k for EgoExo4D), showing that larger is not always better; the queue must balance diversity with the dataset's class distribution.
- Sampling: The FIFO queue strategy provided more stable optimization paths (higher gradient continuity) compared to random sampling.

5. Significance and Implications

Scalable Wearable AI: COMODO offers a practical path to deploying high-performance HAR on resource-constrained, privacy-sensitive wearables (like smart glasses) without the need for massive, manually annotated IMU datasets.
Decoupling Training and Deployment: It separates the "rich but heavy" training phase (using video) from the "lean but efficient" deployment phase (using IMU).
Future of Ubiquitous Computing: The framework is a stepping stone for "always-on" human-centered computing, enabling systems to understand complex, long-tail human activities in the wild by leveraging the vast progress in video foundation models.
Privacy: By eliminating the need for continuous video recording during operation, it addresses major privacy concerns associated with camera-based wearables.

In summary, COMODO successfully leverages the semantic richness of video to "teach" IMU sensors how to recognize activities, creating a highly efficient, accurate, and privacy-preserving solution for next-generation wearable AI.