BTS-rPPG: Orthogonal Butterfly Temporal Shifting for Remote Photoplethysmography

1. Problem Statement

Remote Photoplethysmography (rPPG) estimates physiological signals (like heart rate) from facial videos by analyzing subtle color changes caused by blood circulation. While deep learning has improved rPPG, a critical bottleneck remains: modeling long-range temporal dynamics.

• Limitation of Existing Methods: Most current deep learning approaches rely on temporal shifting or convolutional operators that aggregate information primarily from neighboring frames. This results in:
  • Predominantly local temporal interactions.
  • Limited temporal receptive fields.
  • Inability to effectively capture the periodic, long-range dependencies inherent in cardiac signals.
• Redundancy Issue: Directly exchanging features between frames often propagates redundant information because adjacent frames are highly correlated, leading to inefficient communication.

2. Methodology: BTS-rPPG

The authors propose BTS-rPPG, a framework integrating two core mechanisms: Orthogonal Butterfly Temporal Shifting (BTS) and Orthogonal Feature Transfer (OFT). The architecture is built upon a Swin Transformer backbone.

A. Input Representation

To emphasize subtle temporal changes, the model constructs a Normalized Difference Frame (NDF) representation. Each RGB frame ($X_t$) is concatenated with its NDF ($D_t = \frac{X_{t+1} - X_t}{X_{t+1} + X_t}$), creating a 6-channel input tensor.

B. Orthogonal Butterfly Temporal Shifting (BTS)

Inspired by the butterfly communication pattern in the Fast Fourier Transform (FFT), BTS establishes structured frame interactions to progressively expand the temporal receptive field.

• Butterfly Pairing Schedule: Instead of shifting features only to adjacent frames, BTS uses an XOR-based pairing schedule. At stage $\ell$, frame $t$ interacts with frame $\pi_\ell(t) = t \oplus 2^{\ell-1}$.
• Progressive Expansion: This creates temporal offsets of $1, 2, 4, \dots, 2^{L-1}$ across stages. This allows the model to transition from local interactions in early stages to global, long-range interactions in deeper stages, mimicking the hierarchical nature of physiological signal formation.

C. Orthogonal Feature Transfer (OFT)

To address the redundancy of transferring highly correlated features between frames, OFT filters the source feature before shifting.

• Mechanism: Before a feature is transferred from a source frame to a target frame, it is projected against the target's context (the retained features of the target frame).
• Orthogonalization: The component of the source feature that is aligned (parallel) with the target context is removed. Only the orthogonal component (the part complementary to the target) is transmitted.
• Formula: The transfer operation is defined as $OFT(a, b) = a - \frac{\langle a, b \rangle}{\|b\|^2 + \epsilon}b$, where $a$ is the source fold and $b$ is the target context.
• Benefit: This ensures that cross-frame communication carries only new or complementary information, reducing redundancy and improving efficiency.

D. Architecture Overview

1. Input: RGB + NDF frames.
2. Backbone: A Swin Transformer with 6 stages.
3. Integration: Before the self-attention mechanism in each stage, the BTS operator applies the XOR-based shifting, and OFT filters the transferred features.
4. Output: A predictor head aggregates spatial features and regresses the final rPPG waveform using a negative Pearson correlation loss.

3. Key Contributions

1. BTS Mechanism: Introduction of a structured temporal interaction mechanism using an XOR-based butterfly schedule. This enables efficient, stage-wise expansion of the temporal receptive field from local to global scales without excessive computational cost.
2. OFT Strategy: A novel feature filtering mechanism that retains only the orthogonal component of source features relative to the target context. This minimizes redundant feature propagation and encourages complementary temporal interaction.
3. State-of-the-Art Performance: The proposed framework consistently outperforms existing methods across multiple datasets, demonstrating superior ability to model long-range physiological dynamics.

4. Experimental Results

The method was evaluated on three benchmark datasets: PURE (controlled motion), UBFC-rPPG (natural variations), and MMPD (large-scale, multi-domain).

• Intra-dataset Performance:
  • PURE: BTS-rPPG achieved an MAE of 0.21 (vs. 0.27 for the previous best, RhythmFormer) and a correlation ($r$) of 1.00.
  • UBFC-rPPG: Achieved an MAE of 0.24 and $r$ of 1.00, significantly outperforming competitors (e.g., RhythmFormer MAE 0.81).
  • MMPD: Achieved an MAE of 4.05 and $r$ of 0.75, showing robustness in challenging, real-world conditions with diverse skin tones and lighting.
• Cross-dataset Generalization:
  • In cross-dataset tests (e.g., training on PURE, testing on UBFC-rPPG), BTS-rPPG maintained the best transfer performance, indicating it learns physiologically meaningful representations rather than dataset-specific artifacts.
• Efficiency:
  • Despite being a Transformer-based model, it achieves 2.05 Kfps throughput on a single NVIDIA T4 GPU, offering a favorable trade-off between accuracy and inference speed compared to other Transformer models like PhysFormer.

5. Ablation Studies & Analysis

• OFT vs. BTS: Both components contribute significantly. OFT improves the baseline even without BTS, while BTS provides the largest gains by enabling hierarchical temporal reasoning. Their combination yields the best results.
• Fold Ratio Sensitivity: The optimal ratio for the transferable feature fold ($C_s/d$) was found to be 1/4. This balances the need for cross-frame transfer with the preservation of target-specific information.
• Temporal Schedule Design: The proposed "Butterfly" schedule (expanding from local to global: 1, 2, 4, 8...) outperformed "Linear" (uniform expansion) and "Reverse Butterfly" (global to local). This suggests that physiological signals are best modeled by first consolidating short-term dynamics before integrating long-range rhythmic structures.

6. Significance

BTS-rPPG addresses a fundamental limitation in rPPG: the inability of standard temporal operators to capture long-range dependencies efficiently. By borrowing the FFT butterfly pattern and combining it with orthogonal feature filtering, the authors created a model that:

1. Captures Long-Range Dependencies: Essential for accurate heart rate estimation in periodic signals.
2. Reduces Redundancy: Makes temporal communication more efficient by filtering out correlated noise.
3. Enhances Robustness: Performs exceptionally well under motion, lighting changes, and domain shifts, making it a strong candidate for real-world, contactless health monitoring applications.