💻 computer science

MMGait: Towards Multi-Modal Gait Recognition

This paper introduces MMGait, a comprehensive multi-modal gait recognition benchmark featuring 12 modalities from five heterogeneous sensors and 725 subjects, alongside the OmniGait baseline model designed to unify single-modal, cross-modal, and multi-modal recognition tasks within a shared embedding space.

Original authors: Chenye Wang, Qingyuan Cai, Saihui Hou, Aoqi Li, Yongzhen Huang

Published 2026-04-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chenye Wang, Qingyuan Cai, Saihui Hou, Aoqi Li, Yongzhen Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to identify a friend walking down a busy street. In the past, security cameras relied mostly on RGB (standard color) video. It's like looking at a friend in a photo: you recognize them by their face, clothes, and how they walk. But what happens if it's pitch black? Or if they wear a disguise? Or if a heavy fog rolls in? Standard cameras struggle because they rely on light and clear visuals.

This paper introduces a new, super-powered way to recognize people by their walk (gait), called MMGait. Think of it as upgrading from a single-lens camera to a Swiss Army Knife of sensors.

Here is the breakdown of their breakthrough in simple terms:

1. The Problem: One Tool Isn't Enough

Current security systems are like a person trying to identify someone using only their eyes. If the lights go out, or the person puts on a mask, the system fails. The researchers realized that to truly recognize someone in the real world (rain, night, crowds), we need to use all our senses, not just sight.

2. The Solution: The "Five-Sensor Orchestra" (MMGait)

The team built a massive new database called MMGait. Instead of just recording video, they set up a "concert" of five different sensors, all recording the same people walking at the same time:

The RGB Camera: The standard "eye" (sees color and texture).
The Infrared (IR) Camera: The "night vision" eye (sees heat, works in the dark).
The Depth Camera: The "3D ruler" (measures how far body parts are, ignoring clothes).
The LiDAR Scanner: The "laser mapper" (creates a precise 3D skeleton using laser beams, like a bat's sonar).
The 4D Radar: The "motion detector" (sees movement through fog, rain, and even walls).

The Scale: They recorded 725 different people walking in 12 different ways (normal, with a backpack, with different clothes) from 10 different angles. That's over 334,000 video sequences! It's the largest and most diverse "walking library" ever created.

3. The Big Experiment: Testing the Sensors

The researchers used this library to run three types of tests:

Single-Modal (The Soloist): Can the system recognize someone using only the radar? Or only the infrared?
- Result: Some sensors (like LiDAR) are great at ignoring clothes, while others (like standard video) are great at seeing details but fail in the dark. No single sensor is perfect.
Cross-Modal (The Translator): Can the system recognize someone if it sees them on a Radar but is looking for them in a Video database?
- Result: This is hard! It's like trying to match a shadow to a color photo. It works okay in good weather, but gets messy when people change clothes.
Multi-Modal (The Chorus): What if we combine them?
- Result: Magic. When you mix the "shape" from LiDAR with the "texture" from the RGB camera, the system becomes incredibly tough to fool. Even if someone changes their coat, the 3D shape of their skeleton stays the same, and the system catches them.

4. The Masterpiece: "OmniGait" (The Universal Translator)

The researchers asked a bold question: "Why do we need five different computer programs to handle five different sensors? Why not just one super-brain that understands all of them?"

They created a new task called Omni Multi-Modal Gait Recognition and built a model named OmniGait.

The Analogy: Imagine a universal translator that can understand English, French, and Japanese simultaneously. If you speak English, it listens. If you write in French, it reads. If you hum a tune in Japanese, it recognizes the melody.
How it works: OmniGait is a single AI model that can take any input (a radar scan, a video, a heat map) and match it against any other input.
The Benefit: Instead of training five separate, expensive AI models, you only need one. It's lighter, faster, and can handle any situation the sensors throw at it.

5. Why This Matters

This isn't just about better security cameras. This is about making technology robust.

Real-world reliability: It works in the rain, at night, and when people are hiding their faces.
Privacy: You don't need to see a person's face to know who they are; their unique walking style is enough.
Future-proofing: As we add new sensors to our phones and cars, this "Omni" approach means the software won't need a total rewrite every time hardware changes.

In a nutshell: The authors built the ultimate "walking library" using five different types of sensors to prove that combining them makes recognition super strong. Then, they built a single "super-brain" (OmniGait) that can use any of those sensors to identify anyone, anywhere, anytime. It's the difference between having a flashlight and having a full-spectrum night-vision suit.

1. Problem Statement

Gait recognition is a powerful non-contact biometric technique, but current state-of-the-art methods rely heavily on RGB-derived modalities (e.g., silhouettes, 2D poses). These approaches suffer from significant limitations in real-world scenarios:

Vulnerability: They are easily degraded by occlusion, low illumination, and adverse weather (rain, fog).
Lack of 3D Perception: RGB data lacks geometric structural information.
Limited Scope: Existing benchmarks typically focus on single modalities (RGB) or simple pairs (RGB + LiDAR), failing to support heterogeneous multi-modal collaboration and unified cross-sensor retrieval.
Fragmented Paradigms: Current research treats single-modal, cross-modal, and multi-modal recognition as separate tasks, lacking a unified framework to handle arbitrary sensor inputs and combinations.

2. Methodology

The paper proposes a comprehensive solution involving a new dataset, extensive benchmarking, and a novel unified model.

A. The MMGait Dataset

The authors introduce MMGait, a large-scale, comprehensive multi-modal gait benchmark designed to bridge the gap between controlled environments and real-world multi-sensor deployments.

Scale: Contains 725 subjects and 334,060 sequences.
Sensors: Data is captured using five heterogeneous sensors:
1. RGB Camera
2. Infrared (IR) Camera
3. Depth Camera (ToF)
4. LiDAR Scanner
5. 4D Radar System
Modalities: These sensors generate 12 distinct modalities, including RGB, IR, Silhouettes (RGB/IR), 2D/3D Poses, Events, Depth maps, Projected Depth, and Point Clouds.
Conditions: Data covers 10 viewpoints (0°–360°), 3 walking conditions (Normal, Backpack, Changed Clothing), and diverse lighting (day/night, sunny/cloudy).

B. Benchmarking Protocols

The authors conduct systematic evaluations across three paradigms:

Single-Modal Recognition: Evaluating the standalone performance of each modality under a unified architecture.
Cross-Modal Recognition: Testing generalization and transferability where the query and gallery come from different sensors (e.g., RGB query vs. LiDAR gallery).
Multi-Modal Recognition: Analyzing complementarity by fusing features from intra-sensor (e.g., RGB + Event) and inter-sensor (e.g., RGB + LiDAR) sources.

C. Omni Multi-Modal Gait Recognition (OMGR) Task

To address the fragmentation of existing methods, the authors propose a new task: Omni Multi-Modal Gait Recognition.

Goal: Develop a single unified model capable of accepting any modality as a query and retrieving targets from any modality (single, cross, or multi-modal).
Key Capabilities: Universal Modality Recognition, Adaptive Multi-Modal Fusion, and Modality-Agnostic Retrieval.

D. OmniGait Framework

A simple yet powerful baseline model, OmniGait, is proposed to solve the OMGR task.

Architecture:
1. Modal-Specific Encoding: Independent encoders ( $\varepsilon_m$ ) extract features from raw inputs (e.g., silhouettes, point clouds projected to depth maps) to preserve intrinsic physical characteristics.
2. Cross-Modal Fusion: A lightweight module uses gated weighting to adaptively fuse features from two modalities, balancing their contributions.
3. Shared Representation Learning: A shared residual backbone ( $E$ ) processes both unimodal and fused features to learn modality-invariant embeddings.
Training: Uses a joint objective combining Identity Classification Loss ( $L_{CE}$ ) and Triplet Loss ( $L_{triplet}$ ) to enhance discriminability and alignment.
Efficiency: During inference, only the necessary encoder(s) are activated, allowing the model to handle diverse tasks without retraining separate models.

3. Key Contributions

MMGait Dataset: The most comprehensive multi-modal gait dataset to date, covering 5 sensors, 12 modalities, 725 subjects, and 334k sequences.
Holistic Evaluation: Extensive benchmarks analyzing modality robustness, complementarity, and the challenges of cross-modal retrieval under covariate variations (clothing, bag).
Omni Multi-Modal Gait Recognition (OMGR): A new task formulation that unifies single-modal, cross-modal, and multi-modal recognition into a single framework.
OmniGait Baseline: A unified model that achieves performance comparable to or surpassing specialized single-modal models while offering superior flexibility and efficiency.

4. Key Results

Single-Modal Performance:
- RGB and IR modalities perform best in normal conditions.
- LiDAR Point Clouds and Depth show strong robustness under clothing changes (CL), outperforming RGB in this specific condition.
- 4D Radar shows limited performance alone due to sparsity but offers unique motion cues.
Cross-Modal Retrieval:
- Retrieval between similar modalities (e.g., RGB $\leftrightarrow$ IR) is effective.
- Retrieval between geometric and appearance modalities (e.g., RGB $\leftrightarrow$ Depth) is challenging, especially under clothing changes.
- 4D Radar is difficult to use as a primary query source but serves well as an auxiliary modality.
Multi-Modal Fusion:
- Fusion yields significant gains. For example, fusing RGB Silhouette + LiDAR Projected Depth improves Rank-1 accuracy by +19.7% under clothing changes compared to RGB alone.
- OmniGait successfully unifies these tasks. In cross-modal retrieval (e.g., RGB $\to$ Depth), it outperforms dedicated two-stream baselines (87.0% vs 77.5% Rank-1 in Normal conditions).
- Efficiency: OmniGait uses only 9.96M parameters while supporting all 12 modalities and all three recognition paradigms, whereas training separate models would require significantly more resources.

5. Significance and Impact

Real-World Applicability: MMGait and OmniGait move gait recognition from controlled lab settings to realistic, multi-sensor environments, addressing critical issues like occlusion, lighting changes, and privacy (via non-RGB sensors).
Unified Framework: The OMGR task and OmniGait model demonstrate that a single, lightweight architecture can effectively replace multiple specialized models, reducing deployment complexity and computational overhead.
Future Directions: The paper highlights that while current unified models sacrifice some single-modal optimality for flexibility, future work should focus on better handling raw 3D point clouds and improving cross-covariate robustness.
Resource Availability: The dataset, codebase, and pretrained checkpoints are publicly released, fostering further research in multi-modal biometrics.

In summary, this work establishes a new standard for gait recognition by providing the necessary data, evaluation protocols, and a unified model architecture to handle the complexity of real-world, multi-sensor environments.