Purify-then-Align: Towards Robust Human Sensing under Modality Missing with Knowledge Distillation from Noisy Multimodal Teacher

1. Problem Statement

The paper addresses the critical challenge of robust multimodal human sensing when specific sensor modalities are missing or unavailable due to hardware failure, environmental interference, or cost constraints. The authors identify two fundamental, causally linked barriers that hinder existing solutions:

• The Representation Gap: Different sensors (e.g., depth cameras vs. LiDAR vs. Wi-Fi) operate on vastly different physical principles, resulting in heterogeneous data representations (e.g., dense grids vs. sparse point clouds). Directly fusing these features often leads to information loss or misinterpretation.
• The Contamination Effect: In real-world scenarios, modalities vary in quality. When a high-quality modality is fused with a low-quality, noisy one, the noise from the latter "contaminates" the former, degrading the overall fused representation.

The Core Insight: The authors argue that these issues are causally linked. The Contamination Effect fundamentally impedes the ability to reduce the Representation Gap because a "teacher" model (used for knowledge transfer) cannot provide clean guidance if its own fused representation is corrupted by noisy inputs. Existing methods (generative imputation, shared representation learning, or standard knowledge distillation) fail to address this causal dependency simultaneously.

2. Methodology: The PTA Framework

The authors propose PTA (Purify-then-Align), a novel framework that solves the causal dependency through a synergistic two-stage process:

Stage 1: Purify (Meta-Learning Weighting)

The goal is to create a "clean" teacher representation by mitigating the Contamination Effect before any alignment occurs.

• Mechanism: PTA employs a meta-learning-driven weighting mechanism. Instead of fixed fusion rules, the model learns dynamic importance weights ($w$) for each modality.
• Process: It uses a nested optimization loop:
  • Inner Loop: Optimizes the main model parameters ($\Theta$) using the current weights $w$ to minimize task loss and alignment loss.
  • Outer Loop: Optimizes the meta-parameters ($w$) based on the performance of the inner-loop model on a validation set.
• Outcome: The mechanism dynamically identifies and down-weights noisy, low-contributing modalities during the fusion process. This ensures the resulting "Teacher" feature is a high-quality, consensus representation, free from significant contamination.

Stage 2: Align (Diffusion-Based Knowledge Distillation)

Once the teacher is purified, the framework bridges the Representation Gap by aligning single-modality "student" features to the clean teacher.

• Mechanism: A diffusion-based knowledge distillation paradigm.
• Process:
  • The clean teacher feature ($f_T$) is constructed as a weighted sum of all available modalities.
  • Single-modality student features ($f_S$) are projected into a compressed latent space.
  • The student feature is treated as a "noisy" or incomplete version of the teacher. A noise prediction network (trained on the clean teacher) learns the underlying data distribution.
  • Noise Adapter: A lightweight module predicts a fusion coefficient to blend the student's latent feature with Gaussian noise, creating an optimal starting point for the reverse diffusion process.
  • DDIM Sampling: The model uses Denoising Diffusion Implicit Models (DDIM) with a small number of steps (e.g., 5) to deterministically refine the student feature, effectively "denoising" it and imbuing it with cross-modal knowledge from the teacher.

3. Key Contributions

1. Causal Identification: The paper is the first to explicitly identify and solve the causal link between the Contamination Effect and the Representation Gap in multimodal sensing.
2. Purify-then-Align Paradigm: A novel framework that decouples the problem into two synergistic stages: purifying the knowledge source (teacher) and then aligning the students to it.
3. Meta-Learning Weighting: A dynamic mechanism that automatically suppresses noisy modalities without requiring manual tuning of dropout rates or hard-coded fusion strategies.
4. Diffusion-Based Alignment: The introduction of a diffusion-based distillation strategy that allows single-modality encoders to learn rich, cross-modal representations, significantly enhancing their robustness when other sensors fail.

4. Experimental Results

The framework was evaluated on two large-scale, heterogeneous datasets: MM-Fi (Human Pose Estimation) and XRF55 (Human Action Recognition).

• MM-Fi Results:
  • PTA achieved State-of-the-Art (SOTA) performance across various missing-modality scenarios.
  • Single-Modality Gains: It significantly improved the standalone performance of individual modalities compared to baselines (e.g., LiDAR MPJPE improved by 47.5% over X-Fi).
  • Contamination Mitigation: In fusion scenarios involving low-quality Wi-Fi data, PTA successfully down-weighted the noise, preventing the degradation seen in other methods (e.g., LiDAR+Wi-Fi fusion improved by 32.8% over X-Fi).
• XRF55 Results:
  • PTA outperformed baselines in all modality combinations.
  • Single-Modality Gains: Significant accuracy improvements were observed, particularly for Wi-Fi (+26.64%) and RFID (+12.54%).
  • Stability: Unlike X-Fi, which is sensitive to specific dropout rate configurations, PTA provided stable performance using uniform random dropout.
• Ablation Studies:
  • Removing the Diffusion Alignment module caused a performance drop, confirming its role in bridging the representation gap.
  • Removing the Meta-Learning (Purify) module caused catastrophic performance degradation (e.g., LiDAR+Wi-Fi MPJPE skyrocketed from 74.74 to 189.24), proving that without purifying the teacher, the Representation Gap cannot be effectively bridged.

5. Significance

This work provides a robust solution for deploying human sensing systems in real-world environments where sensor reliability is not guaranteed. By treating the "teacher" as a dynamic entity that must be purified before knowledge transfer, PTA overcomes the limitations of traditional fusion and distillation methods. The framework enables single-modality models to operate with near-multimodal robustness, making it highly valuable for applications in healthcare, smart homes, and the metaverse where sensor dropout is a common failure mode. The integration of meta-learning and diffusion models offers a new direction for handling heterogeneous and noisy data in multimodal AI.