Technical Summary: RhythmJEPA

1. Problem Statement

Remote Photoplethysmography (rPPG) estimates physiological signals, such as heart rate, from facial videos by analyzing subtle skin color variations induced by blood flow. Despite recent advancements, existing self-supervised rPPG methods face two primary limitations:

1. Representation Bias: Most self-supervised approaches rely on reconstructing masked RGB pixels or low-level visual features. This biases the model toward learning facial appearance and texture rather than latent physiological dynamics.
2. Temporal Scanning Constraints: Recent state-space models (e.g., Mamba-based approaches) typically process facial video tokens strictly in chronological order. This limits their ability to exploit the cyclic, quasi-periodic nature of pulse signals, where temporally distant frames may share similar physiological states while adjacent frames may belong to different cardiac phases.

Consequently, current methods struggle to generalize across datasets with varying cameras, subjects, and environmental conditions, and they often fail to capture inter-period relationships critical for robust rPPG estimation.

2. Methodology: RhythmJEPA

The authors propose RhythmJEPA, a rhythm-structured joint-embedding predictive learning framework. Instead of reconstructing input frames, the model predicts latent teacher representations from masked facial videos, encouraging the learning of physiology-aware embeddings. The architecture consists of four key modules:

A. Spatial Pulse Mixer (SPM)

To avoid the computational complexity of Transformer-based spatial modeling, RhythmJEPA employs a lightweight SPM.

• Function: It captures pulse-sensitive spatial cues by combining RGB appearance with short-term temporal differences.
• Mechanism: It utilizes horizontal, vertical, and local depth-wise convolutional branches followed by attentive spatial pooling. This produces compact, frame-level pulse tokens ($Z$) that balance complexity and performance.

B. Cyclic Rhythm-State Planner (CRSP)

To explicitly model the temporal structure of pulse signals, CRSP assigns latent physiological states to frames.

• State Estimation: A Fourier Analysis Network (FAN) predicts frame-wise rhythm-state logits.
• Path Decoding: The model decodes the most plausible cyclic state path ($s^*$) using dynamic programming (Viterbi-style) constrained by a transition grammar. This grammar enforces that a state can either remain unchanged or transition to the next state in the cycle ($A(i) = \{i, (i+1) \mod K\}$).
• Regularization: Two losses are introduced to ensure stable state assignments:
  • Cyclic Transition Loss ($L_{cyc}$): Encourages adherence to the cyclic transition grammar.
  • State Balance Loss ($L_{bal}$): Prevents state collapse by ensuring an even distribution of state usage across the pulse cycle.

C. Dual-Order Mamba Encoder (DOM)

The DOM addresses the limitation of chronological-only scanning by processing the token sequence in two complementary orders:

1. Chronological Branch: Scans tokens in their original temporal order to capture local continuity.
2. Rhythm-State Branch: Reorders tokens based on the decoded cyclic states ($s^*$) before scanning. This allows the Mamba module to capture long-range dependencies between frames sharing similar physiological phases, regardless of their temporal distance.

• Fusion: The outputs of both branches are fused via a token-wise gating mechanism to produce rhythm-aware temporal features ($Y$).

D. Prediction Head and Training Strategy

• Pre-training: The model is pre-trained using a Joint-Embedding Predictive Architecture (JEPA) objective. A student branch processes masked videos, while an Exponential Moving Average (EMA) teacher processes the original video. The student predicts the teacher's latent representations at masked positions.
• Fine-tuning: The pre-trained model is fine-tuned for waveform regression using a negative Pearson correlation loss to align the predicted rPPG waveform with ground truth.

3. Key Contributions

The paper outlines four main contributions:

1. RhythmJEPA Framework: A joint-embedding predictive learning framework that learns latent physiological representations from masked facial videos, avoiding the bias of pixel reconstruction.
2. Spatial Pulse Mixer (SPM): A lightweight spatial modeling module that effectively captures pulse-sensitive cues without the overhead of Transformer-based spatial attention.
3. Rhythm-Aware Temporal Modeling: The introduction of CRSP and the Dual-Order Mamba Encoder (DOM) to model cyclic physiological recurrence. This moves beyond conventional chronological scanning to capture inter-period relationships, regularized by cycle-consistency and state-balance losses.
4. Empirical Validation: Extensive intra-dataset and cross-dataset experiments demonstrating the framework's effectiveness and robustness.

4. Experimental Results

The authors evaluated RhythmJEPA on three benchmarks: PURE, UBFC-rPPG, and MMPD.

• Intra-dataset Performance: RhythmJEPA achieved state-of-the-art results across all metrics (MAE, MAPE, RMSE, and Pearson's $r$).
  • On PURE, it achieved a perfect correlation ($r=1.00$) and reduced MAE by 20.0% compared to the previous best (LSTS).
  • On UBFC-rPPG, it reduced MAE from 0.51 to 0.21 (a 58.8% improvement).
  • On the challenging MMPD dataset, it improved correlation from 0.74 to 0.76 and reduced MAE by 4.9%.
• Cross-dataset Generalization: The model demonstrated superior transferability when trained on one dataset and tested on another (e.g., PURE $\to$ UBFC-rPPG). It consistently outperformed baselines like PhysNet and RhythmFormer, suggesting it learns domain-invariant physiological representations rather than dataset-specific appearance statistics.
• Efficiency: With 3.20M parameters and 33.50G MACs, RhythmJEPA is significantly lighter than recent Transformer-based methods (e.g., PhysFormer, LSTS) while maintaining superior accuracy. It offers a favorable accuracy-efficiency trade-off, being the second most efficient model in terms of parameters and latency while achieving the highest accuracy.

5. Significance and Claims

The paper claims that RhythmJEPA represents a shift from appearance-based reconstruction to physiology-aware representation learning. By predicting latent teacher embeddings rather than RGB pixels, the model is encouraged to focus on the underlying pulse signal.

The significance of the work lies in its explicit modeling of the cyclic nature of rPPG signals. The authors argue that chronological continuity alone is insufficient for robust estimation because frames with similar physiological states may be temporally distant. The integration of CRSP and DOM allows the model to "reorder" the temporal sequence based on latent rhythm states, thereby capturing inter-period dependencies that standard sequential models miss.

The authors modestly note that while the learned rhythm states are effective for estimation, they remain implicit and may not directly correspond to clinically interpretable cardiac phases. Future work is suggested to investigate more interpretable rhythm modeling and evaluation under more diverse real-world conditions.