Closing the Modality Gap in Zero-Shot HAR: Contrastive Training and Separability-Optimized Prototypes on IMU Data

Technical Summary: Closing the Modality Gap in Zero-Shot HAR

Problem Statement
Zero-Shot Learning (ZSL) for Inertial Measurement Unit (IMU)-based Human Activity Recognition (HAR) faces a fundamental "modality gap." Sensor embeddings generated by temporal encoders and semantic prototypes generated by language models (e.g., Sentence-BERT) occupy geometrically separate regions of the embedding space. Consequently, a sensor window labeled "running" does not naturally align with the text embedding of the word "running." Previous approaches often treated this gap as an inference-time issue, relying on post-hoc corrections, or used insufficient semantic targets (single-word labels) that failed to incentivize the encoder to enter the regions of the space occupied by richer semantic representations.

Methodology
The authors systematically evaluated seven configurations on the PAMAP2 dataset, utilizing 14 seen activity classes and 4 unseen classes (running, descending stairs, vacuum cleaning, folding laundry) with subjects 108 and 109 held out for testing. The experimental design varied two primary factors:

1. Encoder Training Objectives:

   • Baseline: A Temporal Convolutional Network (TCN) trained with standard cross-entropy loss using single-word label names as Sentence-BERT prototypes.
   • Contrastive Training: The same TCN architecture trained with a contrastive loss (CLIP-style) using richer semantic prototypes. Two types of text targets were tested: "rich biomechanical descriptions" and "discriminative descriptions" (using class-exclusive vocabulary).

2. Inference Strategies:

   • Nearest-Prototype Cosine Matching: Assigning test embeddings to the unseen class with the highest cosine similarity.
   • ISF + Cross-Modal Centering: Applying Inverted Softmax (ISF) to penalize "hub" prototypes and centering embeddings to align modalities.
   • Cross-Modal Alignment: Learning a linear map from text prototype space to sensor space using seen-class pairs.

Key Findings and Results

• The Modality Gap is Training-Dependent: The study demonstrates that the modality gap is governed by the encoder objective. Replacing label-name prototypes with discriminative descriptions in a contrastive training setup increased the mean cosine similarity between sensor embeddings and text prototypes from 0.30 to 0.69.
• Encoder Alignment Dominates Inference: The most significant performance gains were driven by the training objective, not the inference method. A contrastively trained encoder paired with the simplest inference method (Nearest-Prototype Cosine) outperformed a baseline encoder paired with sophisticated inference corrections (Alignment + ISF).
  • Best Configuration: Contrastive training with discriminative descriptions + ISF correction achieved 73.2% accuracy and 0.583 macro F1 on unseen classes.
  • Baseline: Label-name baseline with similar inference achieved 58.3% accuracy and 0.34 macro F1.
• The Separability Trade-off: A counterintuitive finding emerged: richer text descriptions (biomechanical sentences) reduced inter-prototype separability in Sentence-BERT space (mean off-diagonal cosine increased from 0.29 to 0.50) due to shared vocabulary compressing the prototype cloud. However, the authors found that high-quality encoder alignment allows the model to navigate this compressed space effectively. The combination of discriminative descriptions (which retained some separability) and contrastive training yielded the best balance.
• Metric Validity: The paper argues that overall accuracy is a misleading metric for ZSL-HAR on imbalanced datasets (where "folding laundry" comprised 51% of test windows). A model predicting only the dominant class could achieve ~50% accuracy while failing on all other classes. The authors advocate for macro-averaged F1 as the standard reporting metric to ensure equal weighting of all unseen classes.

Significance and Claims
The paper claims that the primary barrier to ZSL-HAR performance is not the lack of inference-time corrections, but the lack of training-time geometric alignment between sensor and semantic spaces.

• Training Objective is the Dominant Variable: The study concludes that no amount of inference-time correction can compensate for a misaligned encoder. Conversely, a well-aligned encoder significantly reduces the burden on inference stages, allowing simple cosine matching to outperform complex correction pipelines trained on misaligned data.
• Prototype Quality vs. Alignment: While richer descriptions can compress prototype separability, the quality of the encoder's alignment with the prototype space is the first-order variable determining performance. A well-aligned encoder can successfully navigate a less separable prototype space, whereas a misaligned encoder fails even with well-separated prototypes.
• Limitations and Scope: The authors modestly note that results are based on a single subject split (194 test windows) and that certain activities (e.g., vacuum cleaning) remain difficult due to a lack of biomechanically analogous seen classes. They do not claim parity with video-based ZSL methods but highlight the advantage of their text-only approach in scenarios where video data is unavailable.

In summary, the work establishes that bridging the modality gap in IMU-based ZSL requires explicit contrastive alignment during training using discriminative semantic targets, and that macro-averaged F1 is the necessary metric for evaluating such systems on imbalanced benchmarks.