VideoPulse: Neonatal heart rate and peripheral capillary oxygen saturation (SpO2) estimation from contact free video

The paper introduces VideoPulse, a comprehensive dataset and end-to-end deep learning pipeline that enables accurate, contact-free estimation of neonatal heart rate and SpO2 from facial video, offering a low-cost, non-invasive alternative to traditional adhesive monitoring methods in intensive care settings.

The Gist

Imagine a tiny newborn baby in a hospital nursery. They are fragile, their skin is like wet tissue paper, and they are constantly squirming, kicking, and turning their heads. Right now, doctors have to stick adhesive sensors and probes onto this baby's skin to check their heart rate and oxygen levels. It's uncomfortable for the baby, can cause skin irritation, and increases the risk of infection.

VideoPulse is a new technology that says, "Let's just use a camera."

Think of it like a super-powered digital stethoscope that doesn't need to touch the baby at all. Instead of listening to the heart, it "sees" the heartbeat.

How Does It Work? (The Magic of "Invisible Colors")

You know how your face turns slightly pink when you get excited or exercise? That's because blood is rushing to your skin. Even when you are sleeping, your heart pumps blood in a rhythmic wave. This causes tiny, almost invisible changes in the color of your skin with every beat.

• The Old Way: Doctors used to try to find these color changes using complex math formulas (like trying to hear a whisper in a hurricane). It often failed if the baby moved or if the lights flickered.
• The VideoPulse Way: This system uses Deep Learning, which is like teaching a computer to be a master detective. It watches a video of the baby's face and learns to spot those microscopic color shifts, ignoring the baby's wiggles and the changing room lights.

The Three Big Problems They Solved

The researchers faced three huge hurdles, and they built clever tools to overcome them:

1. The "Wobbly Baby" Problem (Face Alignment)

• The Issue: Babies don't sit still like adults. They roll, tilt, and turn their heads. If you try to take a photo of a baby who is upside down or sideways, a normal camera app might get confused.
• The Solution: The team built a digital "head-turner." Before the computer tries to read the heartbeat, it automatically rotates the video in its mind (like turning a photo album) until the baby's face is straight. It's like having a nurse gently hold the baby's head steady, but done entirely by software.

2. The "Noisy Signal" Problem (Cleaning the Data)

• The Issue: To teach the computer, they needed to compare the video to a real heart monitor. But because babies move so much, the real monitor's data was often "noisy" (full of static, like a bad radio station).
• The Solution: They used a digital "noise-canceling" filter (based on a technology called GANs). Imagine you have a recording of a baby crying, but there's a loud fan in the background. This tool learns to subtract the fan noise and leave only the baby's cry. They cleaned up the "real" data so the computer could learn the right patterns.

3. The "Rare Numbers" Problem (Oxygen Levels)

• The Issue: Most healthy babies have oxygen levels near 100%. Very few have low levels. If you teach a student only with examples of "100," they will be terrible at guessing "85" when it actually happens.
• The Solution: They used a technique called Label Distribution Smoothing. Think of it like a teacher who doesn't just grade the student on the most common answers, but gives extra credit for paying attention to the rare, difficult questions. This ensures the system is ready to detect dangerous low-oxygen levels, not just the safe ones.

The New "Training Manual" (VideoPulse Dataset)

To make this work, the researchers couldn't just use old data from adults or babies from other countries. Babies in Sri Lanka (where this study happened) have different skin tones and facial features than babies in the datasets used before.

So, they created a new library of videos called VideoPulse.

• They filmed 52 real newborns in a Sri Lankan hospital.
• They recorded 2.6 hours of video, capturing babies in all sorts of positions.
• This is like creating a new, diverse textbook for the computer to study, ensuring it works for all babies, not just a specific type.

The Results: Fast and Accurate

The results are impressive:

• Speed: The system can predict the heart rate in just 2 seconds. That's almost instant! (Older methods needed 6 seconds or more).
• Accuracy: It's accurate enough to meet strict medical standards. The error rate is so small it's almost like guessing the time within a few seconds.
• Oxygen: This is the big breakthrough. While other systems could guess heart rates, VideoPulse is one of the first to accurately guess oxygen levels (SpO2) just by looking at a video.

Why Does This Matter?

Imagine a future where a baby in the NICU (Neonatal Intensive Care Unit) is monitored by a simple webcam on the ceiling.

• No more sticky tapes peeling off fragile skin.
• No more wires tangling the baby.
• 24/7 monitoring that doesn't disturb the baby's sleep.

VideoPulse turns a standard camera into a life-saving medical tool, making hospital care gentler, safer, and more comfortable for the tiniest patients. It's a perfect example of how technology can be used to give a baby a little more peace and a lot more comfort.

Technical Summary

1. Problem Statement

Remote Photoplethysmography (rPPG) allows for contact-free monitoring of vital signs by tracking subtle color variations in facial video. While valuable for neonates (to avoid skin irritation from adhesive probes and reduce infection risks), existing solutions face significant challenges:

• Data Scarcity: Most rPPG models are trained on adult data. There is a severe lack of neonatal datasets, particularly those with synchronized SpO2 ground truth.
• Environmental Noise: Neonatal ward recordings suffer from motion artifacts, variable illumination, occlusion, and unpredictable face orientations (rotated faces).
• Signal Quality: Ground-truth PPG signals from neonates are often noisy due to movement, making supervised training difficult.
• Label Imbalance: Clinical SpO2 data is heavily skewed toward high saturation values (near 100%), causing standard regression models to underperform on rare but critical low-saturation cases.
• Latency: Many existing deep learning models require long video windows (e.g., 6–12 seconds), which is suboptimal for real-time monitoring.

2. Methodology

The authors propose VideoPulse, an end-to-end deep learning pipeline designed to estimate Heart Rate (HR) and SpO2 from short, unaligned facial video segments.

A. Dataset Creation: VideoPulse

• Source: A new dataset collected in Sri Lanka involving 52 neonates (0–6 days old) in a postnatal ward.
• Content: 157 video recordings totaling 2.6 hours, synchronized with ground-truth PPG and SpO2 from a pulse oximeter.
• Diversity: Includes diverse skin tones, face orientations (rotated poses), and realistic ward lighting conditions.
• Privacy: Due to privacy concerns, the dataset is not public but available to researchers upon request.

B. Preprocessing Pipeline

• Face Alignment: Uses a YOLOv5 detector (pretrained on adults) to localize faces. To handle rotated neonatal faces, the system attempts detection on the first frame; if failed, it rotates the video in 90° increments and retries.
• Temporal Normalization: Applies temporal difference normalization to emphasize pulse-related intensity changes while suppressing static lighting and appearance biases.
• Ground-Truth Denoising: A critical step where noisy PPG signals (caused by neonatal movement) are reconstructed using a GAN-based denoising pipeline. A one-class SVM identifies noisy segments, which are then cleaned by a pretrained generator. Abnormal HRV windows (>15 bpm fluctuation) are discarded to ensure consistency between video and labels.

C. Model Architecture

The system utilizes PhysNet, a 3D CNN backbone, adapted for both HR and SpO2:

• Heart Rate (HR): Uses the standard PhysNet architecture trained with Negative Pearson Correlation Loss to maximize waveform similarity between predicted and ground-truth rPPG signals.
• SpO2 Estimation:
  • Architecture: PhysNet backbone + a custom 3-layer fully connected head (60 $\to$ 32 $\to$ 1 neurons).
  • Loss Function: Uses Weighted RMSE combined with Label Distribution Smoothing (LDS). This addresses the class imbalance (skew toward 100% saturation) by assigning higher weights to rare, low-saturation labels.
  • Augmentation: Employs Time Reversal augmentation (reversing video sequences) to improve robustness against motion trends and temporal artifacts.
• Training Strategy: Models are pre-trained on the public NBHR dataset and then fine-tuned on the VideoPulse dataset. For SpO2, the top two 3D convolutional layers are frozen during fine-tuning to retain general spatiotemporal features.

3. Key Contributions

1. VideoPulse Dataset: The creation of a novel, diverse neonatal rPPG dataset with synchronized HR, SpO2, and PPG data, addressing the lack of demographic diversity in existing public datasets.
2. First Deep Learning SpO2 Estimation for Neonates: To the authors' knowledge, this is the first end-to-end deep learning approach to estimate neonatal SpO2 directly from standard RGB facial video.
3. Robust Pipeline: Integration of GAN-based signal denoising, face alignment for rotated poses, and label-distribution smoothing to handle real-world clinical noise and data imbalance.
4. Low Latency: The system achieves accurate predictions using very short 2-second video windows, enabling near real-time monitoring.

4. Results

The system was evaluated on the NBHR dataset, the new VideoPulse dataset, and the adult PURE dataset.

• Neonatal Heart Rate (HR):
  • NBHR Dataset: Achieved a Mean Absolute Error (MAE) of 2.97 bpm (2.80 bpm at 6s window) using 2-second windows. This outperforms the previous state-of-the-art (NBHRnet) which had an MAE of 3.76 bpm.
  • VideoPulse (Cross-Dataset): When an NBHR-trained model was tested on VideoPulse, it achieved an MAE of 5.34 bpm, demonstrating reasonable cross-dataset transferability.
• Neonatal SpO2:
  • NBHR Dataset: Achieved an RMSE of 2.20% and MAE of 1.69%.
  • VideoPulse Dataset: Fine-tuning yielded an RMSE of 2.18% and MAE of 1.68%.
  • Ablation Study: Showed that combining Label Distribution Smoothing (LDS) and Time Reversal reduced RMSE from 2.74% (baseline) to 2.20%.
• Adult SpO2 (PURE Dataset): The modified PhysNet achieved an RMSE of 0.96%, outperforming other state-of-the-art models (e.g., RhythmMamba, ST Maps) which had errors >26%.

5. Significance

• Clinical Impact: The work demonstrates that accurate, contact-free monitoring of both HR and SpO2 is feasible for neonates using standard webcams and short video segments. This reduces the need for adhesive probes, lowering infection risks and skin irritation.
• Real-Time Feasibility: By operating on 2-second windows with low latency, the system is suitable for real-time integration into Neonatal Intensive Care Units (NICUs).
• Generalization: The successful cross-dataset evaluation (NBHR $\to$ VideoPulse) and performance on diverse skin tones and orientations suggest the model is robust to domain shifts, a critical requirement for clinical deployment.
• Foundation for Future Work: The introduction of the VideoPulse dataset and the specific techniques for handling neonatal noise (GAN denoising, LDS) provide a strong foundation for future research in non-invasive pediatric monitoring.