Technical Summary: A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset

Problem Statement

While deep learning models for Human Activity Recognition (HAR) using wearable sensors have advanced significantly, uni-modal approaches (relying solely on Inertial Measurement Units or IMUs) remain fragile to real-world confounds such as sensor placement variability, handedness, and activities with similar motion signatures. Although multi-modal deep learning—combining IMUs, audio, and environmental sensors—has shown consistent performance improvements over uni-modal counterparts, the field lacks a systematic, head-to-head comparison of diverse sensor fusion paradigms. Existing literature typically proposes a single new fusion method and compares it against a simple baseline on a specific dataset, making it impossible to determine which fusion architecture is most effective under controlled, identical conditions. This gap leaves researchers without empirical guidance for selecting a fusion strategy for new multi-modal HAR systems.

Methodology

To address this gap, the authors conducted a systematic comparison of seven state-of-the-art sensor fusion methods on the HARMES dataset. This dataset comprises 61 hours of fully labeled data from 20 participants performing 15 Activities of Daily Living (ADLs), including self-care, household, and feeding tasks. The data includes three synchronized modalities:

1. IMU: Dual-wrist accelerometer and gyroscope data (12 channels) at 50 Hz.
2. Audio: Ambient acoustic recordings at 44.1 kHz (no speech).
3. Humidity: Atmospheric humidity at 1 Hz (up-sampled).

Experimental Setup

The study employed a unified pipeline to ensure fair comparison:

• Encoders: Each modality was processed by a dedicated encoder mapping raw inputs to a fixed 128-dimensional embedding.
  • IMU: TinyHAR (lightweight CNN + temporal self-attention).
  • Audio: Audio Spectrogram Transformer (AST) with a frozen backbone and a trainable projection head.
  • Humidity: TSMixer (all-MLP architecture).
• Fusion Strategies: Seven distinct fusion paradigms were applied to the fixed embeddings:
  1. Late Fusion: Simple concatenation of embeddings followed by an MLP.
  2. Gated Multi-modal Fusion (GMF): Uses a sigmoid gate to learn per-sample modality weights.
  3. Low-Rank Multi-modal Fusion (LMF): Decomposes tensor interactions into low-rank factors.
  4. Cross-Modal Attention (CMA): Directional attention between every modality pair.
  5. Multi-modal Bottleneck Transformer (MBT): Routes cross-modal information through shared bottleneck tokens.
  6. CLS-Token Transformer: Aggregates modalities via a learnable [CLS] token using unrestricted self-attention.
  7. Decision Fusion: A learned weighted sum of per-modality class distributions.
• Evaluation Protocols:
  • 3-Fold Group Cross-Validation: Participants split into three disjoint folds.
  • Leave-One-Participant-Out (LOPO): A stricter 20-fold evaluation where one participant is held out for testing, matching the original HARMES benchmark protocol.
• Metrics: Primary metric was Macro F1-score to account for class imbalance, alongside accuracy and per-class confusion analysis.

Key Results

Performance Comparison

• Superiority of Multi-Modal: All seven fusion methods outperformed the strongest uni-modal baseline (Audio/AST at 0.734 Macro F1).
• Best Performing Method: Gated Multi-modal Fusion (GMF) achieved the highest performance with a Macro F1 of 0.827 (3-fold CV) and 0.819 (LOPO).
• Simple vs. Complex: The two simplest mechanisms, GMF and Late Fusion (concatenation), outperformed more complex attention-based (CMA, CLS-Transformer, MBT) and tensor-based (LMF) methods. The attention-based methods clustered around 0.79, while LMF and Decision Fusion trailed at 0.747.
• Benchmark Improvement: The GMF model under LOPO evaluation surpassed the original HARMES multi-modal baseline (0.760) by 5.9 percentage points.

Modality Contributions and Confusions

• Audio Dominance: Audio was the most informative single modality. Humidity alone performed near chance level (0.088 F1) and contributed minimally to the overall fusion performance in 10-second windows.
• Targeted Improvements: Fusion gains were not uniform but concentrated on activities that are acoustically ambiguous but have distinct motion patterns (e.g., "Putting away dishes," "Disinfecting hands," "Drinking"). Conversely, activities with strong acoustic signatures (e.g., "Vacuum cleaning") were already well-classified by the audio model alone.
• Humidity Utility: The study found that for 10-second windows, the humidity sensor could be dropped without significant performance loss, as its slow dynamics did not align well with the short temporal context.

Generalization and Robustness

• Participant Generalization: The GMF model generalized well to unseen participants, with LOPO performance (0.819) nearly matching the 3-fold CV score (0.827).
• Handedness Gap Mitigation: A significant performance gap was observed for the IMU-only model (TinyHAR) when applied to left-handed participants (0.54 vs. 0.72 for right-handers), as the dominant motion appeared on the non-instrumented wrist. Audio was handedness-invariant. Multi-modal fusion, particularly GMF, successfully mitigated this gap, narrowing the performance difference to just 0.027 and becoming the best model for left-handed users.

Significance and Claims

The paper claims three primary contributions:

1. Empirical Ranking: It provides the first systematic, head-to-head comparison of seven prominent fusion paradigms on a uniform HAR benchmark, offering an empirical ranking independent of backbone or protocol variations.
2. Practical Drivers: It identifies that the performance drivers on the HARMES dataset are the resolution of specific confusions between activities with overlapping acoustic or motion signatures, rather than a uniform lift across all classes.
3. Robustness to Bias: It demonstrates that multi-modal fusion partially mitigates the "handedness gap," proving that adding audio cues can make wearable HAR systems more equitable and robust for minority user groups (left-handed individuals) where IMU signals alone are unreliable.

The authors conclude that for the HARMES dataset, simple fusion mechanisms (specifically Gated Multi-modal Fusion) are the most accurate and practical choice, outperforming complex attention and tensor-based architectures. They suggest that for datasets of this size and modality count, the additional parameters in complex models may not be justified, and that a lightweight fusion of IMU and audio captures the majority of achievable performance.