COMPASS: Complete Multimodal Fusion via Proxy Tokens and Shared Spaces for Ubiquitous Sensing

1. Problem Statement

The Challenge of Missing Modalities:
Multimodal sensing systems (combining WiFi, mmWave radar, LiDAR, cameras, etc.) typically outperform single-modality systems. However, in real-world deployments, sensors often fail, are occluded, or face privacy/power constraints, leading to missing data at inference time.

The Limitation of Existing Approaches:
Current methods handle missing modalities by:

1. Skipping: Dropping absent branches, which changes the input structure for the fusion head.
2. Subset-Adaptive Fusion: Switching fusion modules based on the observed subset.
3. Imputation: Reconstructing missing features from observed ones.

The Core Issue:
The authors argue these methods suffer from a "fusion-interface mismatch." Models are trained on a fixed set of modality slots but tested on varying input patterns. Even feature imputation fails because generated substitutes may be statistically plausible but structurally incompatible with the fusion head's expected interaction patterns. This leads to degraded cross-modal interaction and incomplete fusion.

2. Methodology: COMPASS

COMPASS (Complete Multimodal Fusion via Proxy Tokens and Shared Spaces) addresses this by enforcing a fixed $N$-slot fusion interface. Regardless of which modalities are missing, the fusion head always receives exactly one token per modality slot.

Key Components:

1. Modality-Specific Encoding & Shared Projection:

   • Raw inputs from $N$ modalities are processed by specific encoders (e.g., ResNet, Point Transformer).
   • Features are projected into a shared latent space via modality-specific projection heads, resulting in token sequences of uniform length ($L$) and dimension ($d$).
   • A global token ($\bar{z}$) is derived for each modality via mean pooling.

2. Directed Source-to-Target Proxy Generation:

   • When a target modality $v_j$ is missing, COMPASS synthesizes a proxy token for its slot using all observed source modalities ($v_i$).
   • Mechanism: For every pair $(v_i \to v_j)$, a learnable generator $G_{i \to j}$ (a single-layer Transformer with a learnable proxy query) transforms the source token sequence into a target-specific proxy token $\hat{t}_{i \to j}$.
   • Aggregation: If multiple modalities are observed, their individual proxy tokens for the missing slot are averaged to form the final proxy token $\hat{t}_j$.
   • Rationale: Cross-modal transfer is asymmetric; specific generators for each direction capture unique information flows better than a shared generator.

3. Fixed Fusion:

   • The fusion input consists of $N$ tokens: real global tokens for observed modalities and aggregated proxy tokens for missing ones.
   • Fusion is performed via simple element-wise summation of these $N$ tokens, followed by a task head (Linear + LayerNorm). This ensures the fusion computation remains identical across all missingness patterns.

4. Training Objectives (Three-Level Strategy):
   To ensure proxies are both structurally compatible and task-relevant, COMPASS uses a composite loss function:

   • Task Loss ($L_{task}$): Standard cross-entropy on the final fused prediction.
   • Proxy Alignment Loss ($L_{align}$): Minimizes MSE between generated proxy tokens and the real global tokens of the missing modality (available during training via synthetic masking). This anchors proxies to the true target distribution.
   • Shared-Space Regularization ($L_{ss}$): Uses VICReg on observed global tokens to ensure the latent space has a geometry that prevents collapse and encourages invariance across modalities without negative pairs.
   • Per-Proxy Discriminative Loss ($L_{proxy}$): Applies the task head to each individual proxy token (before aggregation) to ensure every generated token carries discriminative information for the downstream task.

3. Key Contributions

• Formulation: Reframes missing-modality learning as a slot-complete fusion problem, prioritizing a fixed fusion interface over adapting to subsets.
• Architecture: Introduces COMPASS, a framework using pairwise directed generators in a shared latent space to synthesize missing tokens, ensuring the fusion head always sees a complete $N$-token input.
• Training Strategy: Proposes a novel three-level supervision scheme (alignment, shared-space regularization, and per-proxy discriminative loss) to make proxies both representation-aligned and task-informative.
• Efficiency: Achieves robustness without complex attention-based fusion layers, using simple summation for the final fusion step.

4. Experimental Results

The authors evaluated COMPASS on three ubiquitous sensing benchmarks: XRF55 (RF sensing), MM-Fi (Multi-modal Human Activity Recognition), and OctoNet (IoT sensing).

• Performance:
  • XRF55: COMPASS achieved a 7-scenario average of 84.6%, significantly outperforming X-Fi (72.1%) and the baseline (76.9%). Notable gains were seen in difficult scenarios like WiFi+RFID (missing mmWave), jumping from 58.1% to 86.3%.
  • MM-Fi: Achieved a 15-combination average of 70.6% vs. 62.5% (X-Fi). Single-modality LiDAR performance improved drastically from 52.7% to 80.1%.
  • OctoNet: Outperformed X-Fi in 30 out of 31 modality combinations. Single-modality and dual-modality averages improved by 28.0 and 10.7 percentage points respectively over X-Fi.
• Geometry Analysis: Visualization (t-SNE) showed that COMPASS creates a highly aligned shared latent space where modality boundaries are reduced, unlike X-Fi where modalities remain separated. Crucially, this alignment did not sacrifice class discriminability.
• Efficiency:
  • COMPASS is 1.9x to 2.6x faster than X-Fi during inference because it avoids iterative cross-attention layers in the fusion stage.
  • Model size is comparable to X-Fi (~34M parameters), with the extra parameters dedicated to proxy generators rather than fusion mechanisms.

5. Significance and Limitations

Significance:

• Design Principle: The paper establishes that preserving a modality-complete fusion interface is a simple yet powerful principle for robust multimodal sensing.
• Practicality: It offers a unified model that handles arbitrary missingness patterns without retraining or complex conditional logic, making it highly suitable for real-world IoT and ubiquitous computing environments.
• Efficiency: It demonstrates that robustness does not require heavy fusion architectures; simple summation of well-aligned tokens is sufficient.

Limitations:

• Summation Fusion: While efficient, element-wise summation lacks learnable parameters to model complex, non-linear cross-modal interactions.
• Weak Modalities: Performance on inherently weak modalities (e.g., RFID-only at 48.4%) remains low; aggressive alignment can sometimes erode modality-specific discriminability.
• Task Scope: Currently evaluated only on classification (HAR); performance on regression tasks (e.g., pose estimation) involving spatial alignment is unverified.