💻 computer science

TempoGFormer: A Gating Mechanism-based Transformer for Physiological Measurement from Facial Videos

This paper introduces TempoGFormer, a novel Transformer-based model for remote photoplethysmography that leverages a gating mechanism, a Temporal-Spectral Fusion Head, and a Spatially-Overlapped Tokenization strategy to effectively mitigate environmental noise and motion artifacts, thereby achieving superior accuracy and efficiency in physiological measurement from facial videos across multiple public datasets.

Original authors: Qilei Zhu, Guanlei Xu, Jie Sun, Xiaogang Xu

Published 2026-08-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qilei Zhu, Guanlei Xu, Jie Sun, Xiaogang Xu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to listen to a whisper in the middle of a bustling city square. The voice you want to hear is there, carried on the air, but it is drowned out by traffic, chatter, and wind. This is the fundamental challenge of measuring human heartbeats without touching the skin. For decades, doctors have relied on contact sensors, like the clip placed on a finger, to track the pulse. These devices work well, but they are intrusive and impractical for many situations, such as monitoring a sleeping infant or a patient with severe burns. In recent years, scientists have turned to a technique called remote photoplethysmography, which attempts to read the heartbeat from a simple video of a person's face. The idea is elegant: as blood pumps through the face, it changes the way light reflects off the skin, creating a tiny, rhythmic color shift. However, this signal is incredibly faint, easily lost in the noise of changing light, head movements, or even the person blinking.

A team of researchers at Zhejiang Gongshang University has developed a new way to solve this problem, creating a system that can hear that whisper clearly. They built a sophisticated computer model designed specifically to find the heartbeat hidden inside a video stream. Instead of just looking at the video frame by frame, their system learns to ignore the distractions of the environment and focus intensely on the subtle, rhythmic changes in skin color that indicate a beating heart. By combining a method that smooths out the video data with a new way of filtering out noise, they have created a tool that is more accurate and efficient than previous attempts, proving that a camera can indeed replace a sensor in many vital health scenarios.

The researchers call their creation TempoGFormer. To understand how it works, one must first understand the difficulty it faces. When a camera records a face, the resulting video is a massive amount of data, most of which is irrelevant to the heartbeat. The skin might look different because of a shadow, a smile, or a shift in the room's lighting. Traditional computer models often get confused by these changes, treating a shadow as a heartbeat or missing the real signal entirely. The new system starts by breaking the video down into small pieces, but it does something different from older methods. Instead of cutting the video into rigid, non-overlapping blocks, it uses a sliding approach that lets the pieces overlap. This preserves the smooth flow of time between frames, ensuring that the model sees the continuous movement of blood rather than a series of disjointed snapshots. This step is crucial because the heartbeat is a continuous rhythm, and breaking it apart too harshly destroys the very pattern the model needs to find.

Once the video is prepared, the system applies a special kind of attention mechanism to decide which parts of the image matter most. In many computer vision systems, the model might get distracted by the most obvious features, like the eyes or the mouth, rather than the subtle color changes on the skin. The TempoGFormer solves this with a gating mechanism, a sort of intelligent filter that sits between the model's observation and its decision-making. This filter acts like a volume knob for the signal. It analyzes the attention the model is paying to different parts of the face and automatically turns down the volume on noisy or irrelevant areas while turning up the volume on the regions where the blood flow is most visible. This allows the system to ignore the distractions of motion and light, focusing its energy strictly on the physiological signal.

After filtering the signal, the system uses a specialized head to reconstruct the heartbeat waveform. This part of the model looks at the data from multiple angles, considering both the timing of the beats and the frequency of the rhythm. It combines these different views to create a precise prediction of the heart rate. The researchers tested this system on three different public datasets, which included videos of people sitting still, playing games, and even walking around. In every test, the new model outperformed existing methods, including those based on older deep learning techniques. It achieved higher accuracy in estimating the heart rate and produced a cleaner signal with less noise. Even when the model was trained on one set of videos and tested on a completely different set with different people and lighting conditions, it maintained its high level of performance, showing that it had learned the true nature of the heartbeat rather than just memorizing specific video clips.

The study also looked at the cost of running this system. While the new model is slightly more complex than some simpler alternatives, it does not require an excessive amount of computing power. It strikes a balance, using a manageable number of parameters to achieve results that are significantly better than the competition. The researchers demonstrated that by refining how the model pays attention to the video and how it processes the time-based data, it is possible to extract vital signs with a level of precision that was previously difficult to achieve without physical contact. This work suggests that in the near future, cameras could become standard tools for health monitoring, capable of tracking heart rates in hospitals, gyms, or even at home, all without a single wire or sensor attached to the body. The findings offer a promising step forward in making health monitoring more accessible and less intrusive for everyone.

Technical Summary: TempoGFormer

Problem Statement

Remote Photoplethysmography (rPPG) offers a non-contact method for estimating vital signs like heart rate by analyzing subtle skin color changes in facial videos. However, the rPPG signal is inherently weak and easily corrupted by environmental noise, motion artifacts, and illumination variations, resulting in a low signal-to-noise ratio (SNR).

While deep learning models, particularly Transformers, have shown promise in capturing long-range dependencies essential for periodic physiological signals, existing approaches face two critical limitations:

Attention Shift and Noise Sensitivity: Traditional self-attention mechanisms often prioritize prominent spatial features (e.g., facial structure) over subtle temporal color changes, leading to "attention shift." Furthermore, they treat all tokens equally, inadvertently amplifying noisy regions.
Inadequate Temporal Modeling: Many Transformer-based methods focus heavily on spatial modeling or convert video inputs into coarse-grained tokens, failing to fully exploit long-term temporal dependencies and the continuity of physiological signals. Additionally, standard patch embedding creates artificial spatial boundaries that disrupt the temporal smoothness required for accurate rPPG extraction.

Methodology

The authors propose TempoGFormer, an end-to-end Transformer-based framework designed to enhance rPPG signal extraction through three core innovations:

1. Spatially-Overlapped Tokenization Strategy

To address the loss of temporal continuity and the introduction of spatial boundaries by standard patch embedding, the authors introduce a specialized tokenization pipeline:

Fusion Stem: Utilizes lightweight 3D convolutions (e.g., $5\times5$ and $3\times3$ kernels) to jointly embed spatial appearance and temporal motion cues, emphasizing subtle blood volume pulse changes.
Spatio-Temporal Shallow Encoder (STSE): Employs overlapping 3D convolutions with a kernel size slightly larger than the stride (e.g., $7\times7$ spatial kernel with stride 4). This design ensures that adjacent receptive fields share pixels, creating smooth spatial transitions and mitigating the discontinuities caused by block boundaries. This preserves the continuity of facial features and reduces noise before the data enters the Transformer.

2. Selective Post-Attention Modulator (SPAM)

To combat attention shift and noise amplification, the core of the Transformer is enhanced with the SPAM module, which integrates:

Sparse Attention: Filters out irrelevant key-value pairs to focus computational resources on the most relevant tokens, reducing overhead.
Attention Intensity Modulation Gate (AIMG): A learnable gating mechanism applied after the Standardized Scaled Dot-Product Attention (SDPA) calculation but before the linear projection. The AIMG dynamically recalibrates attention weights using a sigmoid-activated gating function ( $G = \sigma(W_g \cdot f(Q, K))$ ). This allows the model to adaptively suppress attention to noisy regions while highlighting physiologically relevant areas, effectively breaking the linear dependency of traditional attention and alleviating the "Attention Sink" problem.

3. Temporal-Spectral Fusion Head (TSFH)

Moving beyond simple convolutional prediction heads, the TSFH is designed to model the multi-scale, rhythmic nature of rPPG signals. It processes features through three parallel branches:

Time-domain branch: Extracts temporal continuity features.
Frequency-domain branch: Captures frequency rhythmicity.
Multi-scale sensing branch: Utilizes convolutional kernels of varying sizes ($1, 3, 5, 7$) to capture features at different scales.
These branches are concatenated and optimized via a gating mechanism that adaptively balances the fused features with a residual direct connection path, ensuring high-precision waveform estimation.

Key Contributions

The paper claims the following primary contributions:

Novel Gating Mechanism: The introduction of the AIMG within the SPAM module to modulate SDPA outputs, reducing the impact of noise on the attention mechanism and better capturing long-term periodic changes in both time and frequency domains.
Spatially-Overlapped Tokenization: A strategy that avoids hard spatial partitioning to enhance temporal smoothness and reduce inter-frame discontinuities, improving the model's ability to capture subtle blood flow-related color changes.
Temporal-Spectral Fusion Head (TSFH): An innovative prediction head that integrates temporal, frequency, and multi-scale features to improve cross-domain signal stability and expressiveness.
Comprehensive Evaluation: Extensive experiments demonstrating that TempoGFormer outperforms existing deep learning and non-deep learning methods in both accuracy and efficiency.

Experimental Results

The authors evaluated TempoGFormer on three public datasets: PURE, UBFC-rPPG, and mini-MMPD.

Intra-dataset Performance: On the PURE and UBFC-rPPG datasets, TempoGFormer achieved state-of-the-art (SOTA) results across all metrics (MAE, RMSE, MAPE, Pearson Correlation $\rho$ , and SNR). For instance, on PURE, it achieved an MAE of 0.27 and RMSE of 0.46, outperforming PhysMamba (MAE 0.30) and RhythmMamba (MAE 0.29).
Cross-dataset Generalization: In cross-dataset tests (e.g., training on PURE/UBFC and testing on mini-MMPD), the model demonstrated superior robustness. On mini-MMPD, it achieved an MAE of 9.00 (trained on PURE) and 9.18 (trained on UBFC), significantly outperforming other methods like PhysMamba and RhythmMamba.
Ablation Studies: Experiments confirmed that removing any of the three core components (STSE, TSFH, or AIMG) led to significant performance degradation. Specifically, placing the gating mechanism after SDPA (rather than on the Query vector) yielded the best Signal-to-Noise Ratio, validating the design choice to refine attention weights rather than alter semantic matching.
Computational Cost: While TempoGFormer has a moderate parameter count (3.629M) and MACs (37.464G)—ranking second in parameter count (after PhysMamba's 0.56M) and second in MACs among compared models—it achieves the best performance, indicating a favorable balance between computational cost and feature representation capability.

Significance and Claims

The paper positions TempoGFormer as a significant advancement in non-contact physiological measurement by addressing the specific weaknesses of current Transformer applications in rPPG. The authors claim that by explicitly modeling the conflict between weak physiological signals and strong environmental noise through gating mechanisms and overlapping tokenization, the model can focus more effectively on the rPPG signal itself rather than irrelevant facial features or noise.

The study concludes that TempoGFormer achieves SOTA rPPG estimation performance with acceptable computational overhead, making it a robust solution for applications requiring accurate vital sign monitoring in diverse and challenging environments. The authors note that future work aims to integrate multimodal data (infrared, radio frequency) to further enhance prediction accuracy.