Foundation Models Defining A New Era In Sensor-based Human Activity Recognition: A Survey And Outlook

1. Problem Statement

Sensor-based Human Activity Recognition (HAR) is a cornerstone of ubiquitous and wearable computing, yet it faces significant limitations compared to vision and language domains:

• Data Scarcity & Heterogeneity: Unlike ImageNet or web-scale text, sensor datasets are small, fragmented, and lack standardization. Data comes from diverse modalities (IMU, RF, physiological signals) with varying sampling rates, placements, and noise levels.
• Poor Generalization: Current deep learning models are often task-specific and fail to generalize across different users, devices, or contexts (the "non-IID" problem).
• Annotation Costs: High reliance on labor-intensive manual labeling limits scalability.
• Semantic Gap: Traditional classifiers output discrete labels, failing to capture complex routines, context, or intent, and lacking explainability.

The paper argues that Foundation Models (FMs)—pretrained at scale using self-supervised and multimodal learning—offer a unifying paradigm to overcome these constraints by learning reusable, adaptable representations.

2. Methodology and Scope

The authors conducted a comprehensive survey of 132 representative papers (up to late 2025) to synthesize the emerging field of sensor-based HAR foundation models.

• Scope Definition: The survey focuses strictly on sensor-based HAR (inertial, physiological, RF, acoustic) and explicitly excludes vision/video-based HAR to maintain depth and clarity, treating vision only as an auxiliary modality for cross-modal alignment.
• Taxonomic Framework: The authors propose a lifecycle-oriented taxonomy spanning four phases:
  1. Input Design: Modality scope and data landscape.
  2. Pretraining: Learning objectives and architectures.
  3. Adaptation: Mechanisms for fine-tuning and personalization.
  4. Utilization: Downstream capabilities and deployment settings.
• Analysis Axes: The survey analyzes existing work across nine technical axes, including modality scope, tokenization strategies, base architectures, pretraining paradigms, and adaptation mechanisms.

3. Key Contributions

A. Unified Definition and Criteria

The paper formalizes a definition for Sensor-based HAR Foundation Models: Pretrained, sensor-grounded models and their adapters that can solve diverse activity-understanding tasks across the sensing–temporal–context continuum while generalizing across sensors, users, devices, and environments.
They establish six diagnostic criteria for a model to be considered an FM in this domain:

1. Corpus Coverage & Diversity: Training on heterogeneous, large-scale data.
2. Cross-Domain Generalization: Zero-shot/few-shot transfer across users and devices.
3. Modality Extensibility: Ability to handle multiple sensor types or align with language/vision.
4. Label-Efficient Pretraining: Reliance on self-supervised objectives (masked modeling, contrastive learning).
5. Adaptation Surfaces: Lightweight interfaces (adapters, LoRA) for rapid retargeting.
6. Broad Applicability: Emergent capabilities beyond training tasks (e.g., reasoning, retrieval).

B. Technical Taxonomy (The 9 Axes)

The survey categorizes the field into specific technical patterns:

• Modality Scope: Ranges from Unimodal (e.g., IMU-only) to Multimodal (fusing IMU+PPG) and Cross-modal (aligning sensors with text/vision for zero-shot reasoning).
• Tokenization: Strategies include window-based segmentation, statistical aggregation, spectrogram embeddings, and discrete quantization (mapping continuous signals to codebooks for LLM compatibility).
• Base Architectures: Four primary computation graphs are identified:
  • Encoder-only: For discriminative tasks (recognition).
  • Dual Encoders: For retrieval and alignment (CLIP-style).
  • Encoder-Decoder: For conditional generation (captioning, Q&A).
  • Language-Model Stacks: Treating sensor data as tokens for direct LLM reasoning.
• Pretraining Paradigms:
  • Contrastive: Maximizing agreement between augmented views (good for invariance).
  • Generative: Masked signal modeling or forecasting (good for temporal continuity).
  • Hybrid: Combining both for robustness and semantic grounding.
• Adaptation Strategies: Emphasis on Parameter-Efficient Fine-Tuning (PEFT) (e.g., LoRA, Adapters) to enable on-device personalization without full retraining, alongside instruction tuning for natural language interaction.

C. Three Dominant Development Trajectories

The authors identify three distinct evolutionary paths in the field:

1. HAR-Specific FMs (From Scratch): Models trained entirely on large sensor corpora (e.g., HAR-FM, MASTER) to learn domain-native representations. These offer the best cross-dataset transfer but require massive, curated sensor data.
2. Adaptation of General FMs: Leveraging pre-trained time-series (e.g., Chronos) or multimodal models and adapting them to sensor data via projection heads or adapters. This is a pragmatic approach for efficiency.
3. LLM Integration: Using Large Language Models as reasoning engines to interpret sensor data, enabling semantic grounding, explainability, and human-AI interaction (e.g., SensorLLM, HARGPT).

4. Results and Findings

• Generalization: Foundation models significantly outperform traditional supervised models in cross-user, cross-device, and cross-dataset scenarios, particularly when using self-supervised pretraining.
• Label Efficiency: Self-supervised pretraining (masked reconstruction, contrastive learning) drastically reduces the need for labeled data, enabling effective few-shot and zero-shot learning.
• Semantic Grounding: Cross-modal models (Sensor-Text) successfully enable zero-shot retrieval and natural language captioning of activities, bridging the gap between raw signals and human-understandable concepts.
• Deployment: While cloud-scale pretraining is dominant, on-device adaptation via PEFT is emerging as a critical capability for privacy-preserving personalization.
• Scaling Laws: Preliminary studies (e.g., LSM) suggest that scaling model size and data volume leads to performance plateaus, but current sensor datasets are still too small compared to vision/language to fully realize the potential of massive models.

5. Significance and Future Outlook

• Paradigm Shift: The paper marks a transition from "task-specific classifiers" to "general-purpose activity understanding," moving HAR from perception to cognition.
• Open Challenges:
  • Data Curation: The lack of large-scale, standardized, privacy-compliant sensor datasets remains the biggest bottleneck.
  • Temporal Hierarchy: Current models struggle to encode nested temporal structures (from micro-gestures to daily routines).
  • Responsible Deployment: Issues of privacy, hallucination in LLM-based HAR, and energy efficiency on edge devices need rigorous solutions.
• Future Directions: The field is moving toward hybrid frameworks that combine sensor-native pretraining, modular adaptation, and language-grounded reasoning. The ultimate goal is human-centered foundation models that are interpretable, robust, and capable of reasoning about human behavior in real-world contexts.

In conclusion, this survey provides the first comprehensive roadmap for the field, establishing that foundation models are not just an incremental improvement but a fundamental redefinition of how sensor-based human activity is understood, modeled, and deployed.