⚡ electrical engineering

VCR: Learning Valid Contextual Representation for Incomplete Wearable Signals

The paper proposes VCR, a self-supervised framework that leverages orthogonal tokenization to disentangle shared semantics from modality-specific residuals, thereby enabling robust health monitoring from incomplete wearable signals by reconstructing only inferable shared components to prevent hallucinations.

Original authors: Yuxuan Weng, Wenhan Luo, Qijia Shao

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuxuan Weng, Wenhan Luo, Qijia Shao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a person's health by listening to a choir. The choir has four singers: one sings the heartbeat (PPG), one sings the sweat levels (EDA), one sings the movement (ACC), and one sings the body temperature (TEMP).

In a perfect world, all four singers are always present, and you can hear the full song. But in the real world, things go wrong. Maybe the sweat sensor breaks, or the user turns off the heart monitor to save battery, or the movement sensor gets loose. Suddenly, you are trying to understand the song with only three, two, or even just one singer left.

Most current AI models try to "fill in the blanks" by guessing what the missing singers might have sounded like. The problem? They often guess wrong. They might invent details that never existed (like a specific high note the sweat singer never actually sang), which confuses the AI and makes it less reliable. This is called hallucination.

The paper introduces VCR (Valid Contextual Representation), a new way to train AI to listen to these health wearables without getting confused when singers drop out.

Here is how VCR works, using simple analogies:

1. The "Strict Librarian" (Orthogonal Tokenizer)

Imagine the AI's brain is a library. When the choir sings, the AI needs to sort the notes into two separate piles:

Pile A (Shared Secrets): Notes that all singers agree on. For example, if the heart beats fast, the body moves, and sweat increases, that's a shared "stress" signal.
Pile B (Solo Acts): Notes that are unique to just one singer. For example, the specific "static noise" in the sweat sensor or the unique rhythm of the movement sensor.

Old AI models tried to keep these piles mixed together. VCR uses a "Strict Librarian" (called an Orthogonal Tokenizer) to physically separate these piles. It uses a geometric trick (like a rigid ruler) to ensure the "Shared Secrets" pile and the "Solo Acts" pile never touch. They are mathematically guaranteed to be independent.

2. The "Smart Guessing Game" (Avoiding Hallucinations)

Here is the magic trick. When a singer is missing (e.g., the Sweat singer is gone):

Old AI: Tries to guess the entire song, including the unique "Solo Acts" of the missing singer. It invents fake sweat data. This is bad because the AI is guessing things it can't possibly know.
VCR: Knows the rules. It says, "I can't guess the unique sweat noise because that singer is gone. But I can guess the 'Shared Secrets' because the other singers (Heartbeat, Movement, Temp) are still singing those parts."

VCR only tries to reconstruct the Shared Secrets that are inferable from the remaining singers. It refuses to guess the "Solo Acts" of the missing singer. This prevents the AI from making up fake data (hallucinations) and keeps its understanding grounded in reality.

3. The "Flexible Team" (Missing-Aware MoE Backbone)

Once the notes are sorted, they go to a "brain" made of a Mixture of Experts (MoE). Think of this as a team of specialists.

If the Heartbeat and Movement singers are present, a specific specialist takes the lead.
If the Temperature singer is missing, a different specialist steps in who knows how to work without that input.

Instead of forcing one giant brain to try to do everything at once (which gets confused when data is missing), VCR uses a router to send the task to the specific expert best suited for the current combination of available sensors.

Why This Matters (The Results)

The authors tested VCR on real-world health data for tasks like:

Emotion Detection: Knowing if someone is stressed or happy.
Activity Recognition: Knowing if someone is walking, running, or sleeping.
Sleep Staging: Figuring out if someone is in deep sleep or light sleep.
VO2 Prediction: Estimating how much oxygen the body is using.

They found that VCR works better than all previous methods, even when:

All sensors work: It's still the most accurate.
One sensor is missing: It barely loses performance.
Multiple sensors are missing: It stays robust while other models fail completely.

The Bottom Line

VCR is like a smart listener who knows exactly what they can and cannot hear. Instead of making up a story to fill the silence, it focuses only on the facts that are still audible. This makes it a much more reliable tool for wearable health devices, ensuring that even if a sensor breaks or a user turns one off, the health insights remain accurate and trustworthy.

Technical Summary: VCR: Learning Valid Contextual Representation for Incomplete Wearable Signals

1. Problem Statement

Wearable devices offer continuous, non-invasive health monitoring through multimodal signals (e.g., PPG, EDA, ACC, TEMP). However, real-world deployment is hindered by two coupled challenges:

Label Scarcity: Downstream health tasks (emotion recognition, sleep staging, activity recognition) suffer from a lack of labeled data.
Modality Missingness: Sensors frequently drop out due to hardware constraints, user behavior (e.g., disabling PPG to save power), or device limitations. This results in "structural missingness," where an entire modality is absent for a given time window.

Existing self-supervised learning (SSL) approaches often assume full modality availability during pretraining. When adapted to missing scenarios, current methods typically attempt to reconstruct the entire absent signal. The authors argue this is flawed because modality-specific details (e.g., specific noise patterns or unique physiological drifts) are often non-identifiable from the remaining sensors. Forcing a model to reconstruct these unobservable details encourages "hallucination"—generating plausible but ungrounded data that injects noise into the learned representations, thereby degrading robustness.

2. Methodology: The VCR Framework

The authors propose VCR (Valid Contextual Representation), a two-stage, self-supervised framework designed to learn representations robust to modality missingness without hallucinating unobservable details.

A. Orthogonal Tokenizer (Stage 1)

The first stage focuses on disentanglement. The core hypothesis is that any modality's latent information can be separated into:

Shared Semantics ( $s$ ): Information correlated across modalities (e.g., heart rate patterns visible in both PPG and ACC).
Modality-Specific Residuals ( $p$ ): Unique information not inferable from other sensors (e.g., skin impedance drift in EDA).

To enforce strict separation and prevent information leakage, VCR employs a Geometric Orthogonal Tokenizer:

Rectification & Whitening: A lightweight residual adapter rectifies encoder embeddings, followed by a whitening process to approximate an identity covariance matrix.
Orthogonal Projection: A learnable parameter matrix undergoes QR decomposition to form an orthonormal basis. This basis defines a projection operator $P$ that strictly separates the embedding into a shared subspace ($s = Ph$) and a specific residual subspace ( $p = (I-P)h$ ).
Statistical Independence: Under the assumption of Gaussian-distributed latent representations, this geometric orthogonality ensures statistical independence ( $I(s; p) \approx 0$ ).
Hybrid Optimization: The tokenizer is trained using:
- Cross-Modal Contrastive Loss: Applied only to $s$ to align shared semantics across modalities.
- Intra-Modal Reconstruction: Applied only to $p$ to preserve unique details.
- Whitening Regularization: To maintain the independence condition.

B. Missing-Aware MoE Backbone (Stage 2)

The second stage processes the tokens from the tokenizer to learn contextual representations under varying availability patterns.

Architecture: A sparse Mixture-of-Experts (MoE) backbone. Learnable partition embeddings are injected to explicitly identify the modality origin and component type ( $s$ vs. $p$ ) of each token.
Routing: The router dynamically activates different experts based on the observed modality set, allowing the model to specialize in specific modality combinations without interference from missing data.
Training Objectives (Masked Reconstruction):
- Random Masking: Standard patch masking on available modalities to reconstruct both $s$ and $p$ .
- Structural Masking: Entire modalities are dropped. Crucially, the reconstruction objective is validity-aware:
  - If one modality is missing, the model reconstructs only its shared component ( $s$ ), explicitly excluding the non-inferable specific component ( $p$ ).
  - If multiple modalities are missing, reconstruction for those modalities is skipped entirely, as even shared components may be uninferable.
- This design prevents the model from being forced to hallucinate unobservable details.

3. Key Contributions

Validity-Aware Learning: The introduction of an orthogonal tokenizer that structurally disentangles shared and specific components. This enables a reconstruction objective that only targets inferable information, effectively mitigating hallucination-like completions.
Missing-Aware MoE Backbone: A sparse architecture with learnable partition embeddings that adapts computation to diverse modality-availability patterns, avoiding the noise introduced by zero-padding or special tokens.
Two-Stage Information Preservation: A training strategy that first preserves information integrity via disentanglement and contrastive alignment, then learns robust contextual semantics via masked latent reconstruction, avoiding domination by high-variance modalities.

4. Experimental Results

VCR was evaluated on a comprehensive benchmark spanning four datasets (WESAD, DaLiA, AAUWSS, WEEE) and four tasks (emotion detection, activity recognition, sleep staging, VO2 prediction).

Full Modality Performance: VCR outperformed both supervised baselines (ResNet, ViT, FlexMoE) and pretrained self-supervised baselines (SimCLR, LSM, LSM-2, etc.) across all tasks, demonstrating that robustness to missingness does not come at the cost of performance when all sensors are available.
Single Modality Missingness: In scenarios where one sensor was removed, VCR achieved the best or second-best performance in 65 out of 88 reported metrics. Notably, when critical sensors like ACC were missing, VCR maintained significantly higher performance than baselines, showing a graceful degradation compared to the sharp drops seen in other methods.
Multiple Modality Missingness: VCR remained robust across all combinations of 2 or 3 missing modalities, consistently ranking in the Top-1 or Top-2 against baselines.
Hallucination Analysis: A case study comparing VCR with LSM-2 (which reconstructs full raw signals) showed that LSM-2 induced larger representation shifts and semantic inconsistencies when reconstructing missing data. VCR's strict separation of $s$ and $p$ prevented these shifts, confirming that avoiding the reconstruction of non-inferable details preserves representation validity.
Disentanglement Verification: Quantitative analysis using the Hilbert-Schmidt Independence Criterion (HSIC) showed VCR achieved significantly lower dependence between $s$ and $p$ compared to soft-constraint baselines like FOCAL.

5. Significance and Claims

The paper claims that VCR establishes a new standard for pretraining multimodal wearable models by prioritizing representation validity. The authors argue that the common practice of reconstructing entire missing signals is fundamentally flawed because it forces models to guess unobservable details, leading to hallucinations that degrade downstream performance.

By enforcing strict geometric disentanglement and limiting reconstruction targets to inferable shared components, VCR provides a structural foundation for robust learning. The framework is presented as a general-purpose solution that remains effective across diverse health monitoring tasks and sensor configurations, addressing the critical gap between idealized full-sensor training and the reality of incomplete wearable data. The authors note that while the current evaluation focuses on a single device type (Empatica E4), the modality-agnostic nature of the framework suggests potential for broader generalization, which remains a direction for future work.