Multilevel neural networks with dual-stage feature fusion for human activity recognition

1. Problem Statement

Human Activity Recognition (HAR) is critical for applications in healthcare, security, and intelligent environments. While deep learning models like Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks have advanced the field, there is a lack of systematic evaluation regarding:

• The optimal structural arrangement of hybrid models (combining CNN, LSTM, and Convolutional LSTM).
• The effectiveness of different feature fusion strategies (specifically comparing late fusion vs. intermediate fusion).
• The performance of Convolutional LSTM (CLSTM) in HAR, a novel application for this specific domain.
• The suitability of 1D vs. 2D architectures for processing inertial measurement unit (IMU) sensor data.

2. Methodology

The authors propose a two-level network architecture that integrates features from multiple sensors (accelerometer and gyroscope) using a dual-stage feature fusion strategy.

A. Data Preprocessing

• Datasets: Two public benchmarks were used:
  • USC-HAD: 14 subjects, 12 activities, hip-mounted sensors (accelerometer, gyroscope, magnetometer).
  • UCI-HAR: 30 subjects, 6 activities, waist-mounted smartphone sensors.
• Processing: Raw sensor data undergoes filtering and normalization (zero mean, unit norm). Accelerometer and gyroscope signals are processed in parallel paths.

B. Proposed Architecture

The model consists of two sequential network levels:

1. First Level (Parallel Processing):
   • Accelerometer data and Gyroscope data are fed into separate but identical neural networks.
   • The networks can be CNN, LSTM, or Convolutional LSTM (CLSTM).
   • Late Fusion (Stage 1): The outputs of these two parallel networks are concatenated to create a unified feature map.
2. Intermediate Fusion (Stage 2):
   • The fused features from the first level are passed to a second neural network for further abstraction.
   • Global Average Pooling (GAP) is applied to the outputs of both the first and second networks to summarize spatial information and reduce dimensionality.
   • Intermediate Fusion: The pooled features from the first and second stages are concatenated. This allows the model to leverage both low-level and high-level representations simultaneously.
3. Classification: The final concatenated vector passes through a dense layer and a Softmax layer for activity classification.

C. Experimental Design

• Ablation Study: The authors evaluated 15 different configurations by varying the first and second network types (1D/2D CNN, LSTM, 1D/2D CLSTM).
• Fusion Comparison: Models were tested with Late Fusion only vs. Dual-Stage Fusion (Late + Intermediate).
• Training: Used ADAM optimizer, categorical cross-entropy loss, and 5-fold cross-validation.

3. Key Contributions

1. Novel Dual-Stage Fusion Framework: The study introduces a specific architecture that combines late fusion (merging sensor streams) with intermediate fusion (merging features across network depths), demonstrating that integrating features at multiple stages improves performance.
2. First Application of CLSTM in HAR: This is the first study to apply Convolutional LSTM (CLSTM) in both individual and hybrid configurations for HAR, exploring its ability to capture spatio-temporal dependencies directly within the recurrent structure.
3. 1D vs. 2D Analysis: The research provides empirical evidence that 1D architectures (1D CNN, 1D CLSTM) are superior to 2D counterparts for IMU data, as the data is fundamentally a 1D temporal signal. 2D models add complexity without proportional performance gains.
4. Comprehensive Benchmarking: A systematic comparison of 15 network combinations against state-of-the-art (SOTA) methods on two distinct datasets.

4. Results

The experiments were conducted on raw sensor data and pre-extracted feature vectors.

• USC-HAD Dataset (Raw Sensors):
  • The optimal configuration was Dual 1D CNNs with Intermediate Fusion, achieving 94.40% accuracy.
  • This outperformed models using only late fusion and models based on 2D architectures.
• UCI-HAR Dataset (Raw Sensors):
  • The best model (LSTM + 2D CNN with intermediate fusion) achieved 88.90% accuracy.
  • Intermediate fusion consistently improved accuracy over late-fusion-only baselines.
• UCI-HAR Dataset (Feature Vectors):
  • The highest overall accuracy was 96.75%, achieved by a 1D CNN followed by another 1D CNN with intermediate fusion.
  • Configurations using 1D CNNs as the second stage generally outperformed those using LSTM or CLSTM.

Comparison with State-of-the-Art (SOTA):
The proposed model outperformed existing methods on both datasets:

• USC-HAD: 94.40% (Proposed) vs. 90.88–92.83% (Previous CNN-LSTM/H-LSTM).
• UCI-HAR: 96.75% (Proposed) vs. 91.65–95.76% (Previous CNN-LSTM/H-LSTM).

5. Significance and Conclusion

• Effectiveness of Intermediate Fusion: The study conclusively demonstrates that fusing features at an intermediate stage (combining shallow and deep features) yields higher accuracy than relying solely on late fusion.
• Architecture Efficiency: For IMU-based HAR, 1D convolutional layers are more efficient and effective than 2D layers, offering a better trade-off between computational cost and accuracy.
• Generalizability: The proposed dual-stage 1D CNN architecture with intermediate fusion achieved the highest reported accuracy on both USC-HAD and UCI-HAR, validating its robustness across different sensor configurations and activity sets.
• Future Directions: The authors suggest exploring adaptive fusion mechanisms (e.g., attention-based) to dynamically weight sensor contributions and optimizing the model via pruning/quantization for deployment on low-power edge devices.

In summary, this paper provides a robust framework for HAR that leverages multi-level feature fusion and 1D convolutional structures to achieve state-of-the-art performance, offering clear guidelines for designing efficient deep learning models for sensor-based activity recognition.