💻 computer science

SIR-PhysNet: a deadline-aware ultra-lightweight pipeline for camera-based heart-rate monitoring on embedded hardware

The paper introduces SIR-PhysNet, an ultra-lightweight, deadline-aware rPPG pipeline that offloads geometric and illumination processing to fixed-cost operations and uses an analytical Fourier mask for heart rate extraction, enabling real-time, high-accuracy monitoring on resource-constrained edge devices like the Raspberry Pi 5 where face detection, rather than network inference, is the primary performance bottleneck.

Original authors: Haochen Chai, Zining Liu, Wentao Zhang, Fangfang Jiang

Published 2026-09-08

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Haochen Chai, Zining Liu, Wentao Zhang, Fangfang Jiang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your heartbeat can be read without a single wire, a single sensor, or a single touch. This is the promise of camera-based health monitoring, a field that turns ordinary video into a window on the body's most vital rhythm. The science behind this relies on a simple, invisible truth: every time your heart beats, it pumps a tiny surge of blood through your face, changing the color of your skin by a fraction of a percent. These changes are too small for the human eye to see, but a camera can capture them. By analyzing these subtle shifts in light and color, it is possible to calculate how fast a heart is beating. This technology is already being considered for use in cars to monitor driver fatigue, in hospitals to track patients, and in smart homes to check on the elderly. However, for this to work in the real world, the computer doing the math must be fast enough to keep up with the video, and it must be small and efficient enough to run on a device that fits on a shelf or in a pocket, not just in a massive data center.

A team of researchers at Northeastern University has built a new system designed specifically to solve this problem of speed and size. They call their creation SIR-PhysNet, a pipeline that acts as a complete workflow for turning raw video into a heart rate number. The core challenge they faced was not just making the math accurate, but ensuring the entire process could finish its job before the next frame of video arrived. In their setup, the system must process a video stream, find a face, clean up the image, and calculate a new heart rate estimate every 2.5 seconds. If the computer takes too long, the system falls behind, the data becomes useless, and the monitoring fails. The researchers discovered that the biggest bottleneck was not the complex brain of the system, but the eyes that looked for the face.

To build a system that could run on a Raspberry Pi 5, a small, affordable computer often used by hobbyists and engineers, the team had to rethink how the work was divided. Most previous attempts tried to teach a single, massive computer program to do everything: find the face, ignore the shadows, handle head movements, and calculate the heart rate all at once. This approach required a lot of memory and processing power, making it too slow for small devices. The new approach, SIR-PhysNet, splits the job into two distinct parts. First, it uses simple, fixed rules to prepare the image. Before the computer ever looks at the video to find a pattern, it uses standard image tools to lock onto the face, correct for the angle of the head, and remove the effects of changing light. This step is like putting a frame around a painting and cleaning the glass before you try to analyze the art; it makes the job much easier for the computer that follows. By doing this heavy lifting with simple tools, the researchers were able to shrink the complex part of the system down to a tiny size, requiring only a fraction of the memory and power of other models.

The second part of the system is the actual heart rate calculator. Because the image was already cleaned and stabilized by the first step, this part of the system does not need to be a giant, heavy network. It is a compact model with just 0.26 million parameters, a number that represents the amount of information it needs to remember. Instead of trying to reconstruct the entire shape of the blood pulse wave, which is a complex and unnecessary task for simply counting beats, this model predicts the dominant frequency of the heartbeat directly. It then uses a mathematical filter to extract the rate from the signal. This design choice means the system is incredibly efficient, using only 0.12 gigaflops of computing power, which is a measure of how many calculations it performs per second.

The researchers tested their system on several different video datasets, including recordings of people exercising on a treadmill, where head movement and sweat make the task very difficult. On a standard, controlled dataset, the system achieved an accuracy that matched or exceeded much larger, more complex models. On the difficult treadmill dataset, it produced an average error of just 3.39 beats per minute, a result that is comparable to systems that are ten times larger. This proved that by cleaning the image first, the system could remain small and fast without sacrificing the ability to handle real-world chaos.

To see if the system could actually run on the small hardware it was designed for, the team installed it on a fan-cooled Raspberry Pi 5. They ran the system for eight hours straight, processing over 800,000 frames of video. The results were clear: the system maintained a steady speed of 34.60 frames per second, comfortably beating the 30 frames per second required to keep up with the video stream. The computer's temperature rose to a peak of 68.85 degrees Celsius, but it never slowed down or shut off due to heat. The most surprising finding came from looking at where the time was spent. The complex heart rate calculation, the part that most people assume is the hardest, took up only about 13 percent of the time allowed for each update. The other 70 percent of the time was spent simply finding the face and adjusting the image.

This discovery changed the understanding of what limits these systems. The researchers found that the speed of the heart rate monitor was not held back by the intelligence of the computer model, but by the effort required to find and track the face in the video. They tested a strategy where the system only looked for the face once every few frames and used a simpler method to guess where it moved in between. This "detect-and-track" approach allowed the system to speed up to nearly 47 frames per second, proving that there is plenty of room to make these monitors even faster if the face-finding step is optimized.

The study concludes that for camera-based heart rate monitoring on small devices, the key to success is not building a bigger, smarter brain, but building a better set of eyes and a cleaner workflow. By handling the messy parts of the real world with simple, predictable tools before the data ever reaches the complex model, the system can remain small enough to run on a device the size of a deck of cards. The researchers are careful to note that while their system works well on the specific videos they tested, it has not yet been proven to work across all types of cameras, lighting conditions, or skin tones. They also point out that their system is designed to give a general heart rate number, not to diagnose heart conditions or detect irregular rhythms. However, the work provides a clear path forward: to make health monitoring truly portable and accessible, the focus must shift from making the models more complex to making the entire pipeline, from the camera lens to the final number, more efficient and balanced.

Technical Summary: SIR-PhysNet

Problem Statement
Camera-based heart rate (HR) monitoring on edge devices is constrained not merely by the computational cost of neural network inference, but by the total frame budget of the entire processing pipeline. A real-time system must decode video, detect and track faces, standardize skin regions, and infer HR within a strict deadline (e.g., every 2.5 seconds for a 150-frame window). Existing approaches often focus on reducing network parameters or FLOPs while neglecting the fact that preprocessing stages—specifically face detection and region-of-interest (ROI) stabilization—can consume the majority of the time budget. Furthermore, many deep learning models regress dense blood volume pulse (BVP) waveforms, committing significant capacity to morphological details that are unnecessary for rate-only monitoring. The challenge is to design a pipeline that is "deadline-aware," ensuring the entire system, not just the network, fits within the hardware constraints of embedded devices like the Raspberry Pi 5.

Methodology
SIR-PhysNet addresses these constraints through two primary design principles: moving nuisance invariance to fixed-cost preprocessing and decoupling rate estimation from waveform generation.

Pipeline Architecture & Preprocessing:
- Fixed-Cost Invariance: Instead of learning to handle geometric variations, skin regions, and illumination drift within the network, SIR-PhysNet imposes these constraints via deterministic image operations before the network.
- ROI Stabilization: Uses MediaPipe Face Mesh to extract 468 landmarks, applying a partial affine transform to map faces to a canonical 200×200 ROI. Frames with excessive rotation (>15°) or displacement (>10 pixels) are discarded or handled via temporal continuity.
- Signal-Informed Standardization: A skin support mask is generated from the landmark convex hull (excluding eyes and lips). Illumination drift is suppressed using a guided filter to estimate a smooth intensity component, creating an illumination quotient. The final input is a weighted fusion of the original RGB stream and the illumination-normalized stream.
- Grid-Based Features: The standardized frames are partitioned into an 8×8 grid, computing mean RGB values to form a compact spatiotemporal descriptor (192 dimensions per frame), reducing input dimensionality before the network.
Network Architecture (SIR-PhysNet):
- The core is a compact spatiotemporal model (0.26 M parameters, 0.12 GFLOPs) that processes the grid descriptors.
- It employs multi-scale 3D encoders (Gaussian pyramid scales) followed by CBAM (Convolutional Block Attention Module) refinement and a temporal Transformer encoder.
- HR-DFF (Heart-Rate Guided Dynamic Fourier Filter): Instead of regressing a dense waveform, the network predicts a dominant cardiac frequency prior ( $f_{hr}$ ). This prior guides an analytical Gaussian mask in the Fourier domain to filter temporal tokens. The surrogate signal is derived via an inverse transform. This readout mechanism introduces zero trainable parameters, as the mask is computed analytically rather than learned.
Scheduling Strategy:
- The system employs a "detect-and-track" scheduling policy. While the accuracy benchmarks use per-frame face detection, the profiling experiments compare this against a hybrid schedule where the detector runs every $N$ -th frame (e.g., every 5th) and optical flow propagates the ROI in between.

Key Contributions

Deadline-Aware Pipeline Design: The authors treat edge rPPG as an image-processing task with an explicit deadline, organizing the pipeline to move invariance to fixed-cost preprocessing, keeping the learned model extremely small (0.26 M parameters).
Parameter-Free Readout (HR-DFF): Introduction of a readout mechanism that separates rate estimation from waveform generation using an analytical Fourier mask conditioned on a learned prior. This eliminates the need for a dense decoder, contributing no trainable parameters to the readout stage.
Comprehensive Embedded Profiling: Unlike studies that report only theoretical FLOPs, this work profiles the complete streaming pipeline on a fan-cooled Raspberry Pi 5. It includes per-stage timing, two scheduling policies, and an 8-hour soak test to measure thermal stability and sustained throughput.
Subject-Disjoint Evaluation: The method is evaluated on UBFC-rPPG, MAHNOB-HCI, and the motion-intensive ECG-Fitness dataset using strict subject-disjoint protocols, separating protocol-matched comparisons from contextual literature values.

Results

Accuracy:
- UBFC-rPPG: Achieved 2.75 BPM RMSE and $r=0.98$ , comparable to larger models like PhysNet (3.51 BPM RMSE) but with significantly fewer parameters.
- ECG-Fitness (Motion-Robust): Achieved 3.39 BPM MAE and 4.80 BPM RMSE. This performance is comparable to much larger architectures (e.g., BigSmall, EfficientPhys) despite using a fraction of the parameters.
- MAHNOB-HCI: Achieved 3.71 BPM MAE and the highest correlation ( $r=0.92$ ) among compared methods.
Embedded Performance (Raspberry Pi 5):
- Throughput: The "every-frame" detection schedule sustains 34.60 fps, exceeding the 30 fps input requirement. The "detect-and-track" (hybrid) schedule reaches 46.97 fps.
- Resource Usage: Face detection and ROI rectification account for approximately 70% of the per-frame preprocessing time, making them the primary bottleneck rather than the network.
- Stability: Over an 8-hour continuous run (814,576 frames), the system maintained >30 fps with a peak CPU temperature of 68.85°C and no throttling flags raised.
- Latency: The network inference (measured via a parameter-matched scaffold) consumes only ~13% of the 2.5-second stride budget, leaving ample headroom for the heavy preprocessing stages.

Significance and Claims
The paper claims that for camera-based HR monitoring on embedded hardware, the face processing stage, not the network inference, is the binding constraint on achievable frame rates. The significance of SIR-PhysNet lies in its demonstration that:

Efficiency is achieved through input standardization: By handling geometric and illumination variations via fixed-cost operations, the network can remain ultra-lightweight without sacrificing robustness.
Parameter economy is possible without accuracy loss: The HR-DFF readout proves that dense waveform regression is unnecessary for rate estimation, allowing for a model that fits comfortably within tight hardware budgets.
Real-time behavior requires system-level profiling: Theoretical operation counts are insufficient; actual performance depends on the interaction between the pipeline stages and the specific hardware (e.g., the Raspberry Pi 5), where preprocessing dominates the cost.

The authors modestly note that the accuracy claims are limited to intra-dataset subject-disjoint protocols and that the "detect-and-track" scheduling, while faster, has not been validated for HR accuracy under motion. The work establishes an operating point where a small model and a heavy preprocessing pipeline can coexist to meet real-time deadlines on consumer-grade embedded hardware.

Technical Summary: SIR-PhysNet

More like this