Benchmarking Video Foundation Models for Remote Parkinson's Disease Screening

This paper presents a large-scale systematic benchmark of seven video foundation models on a novel dataset of 32,847 videos from 1,888 participants, revealing that model performance for remote Parkinson's disease screening is highly task-dependent and establishing a rigorous baseline with AUCs up to 85.3% while highlighting the need for task-aware calibration to improve sensitivity.

The Gist

Imagine you have a very smart, super-observant robot that has watched millions of hours of videos on the internet. It knows how people walk, talk, smile, and move their hands better than almost anyone else. Now, imagine we want to use this robot to help doctors spot Parkinson's disease early, even if the patient is sitting in their living room instead of a hospital.

That is exactly what this paper is about. The researchers took a bunch of these "super-robots" (called Video Foundation Models) and put them through a rigorous test to see which one is best at spotting the subtle signs of Parkinson's just by watching a video.

Here is the breakdown in simple terms:

1. The Problem: The "Specialist" Bottleneck

Parkinson's disease usually starts with small movement issues: a hand shaking, a face that doesn't smile enough, or a voice that sounds flat. To diagnose it, you usually need a specialist doctor to watch you do specific tasks (like tapping your fingers or saying a tongue twister).

• The Issue: Not everyone can see a specialist. Some people live far away, can't afford it, or don't have time.
• The Goal: Can we use a regular smartphone camera and a computer program to do a "pre-check" for Parkinson's?

2. The Solution: The "Video Foundation Models" (VFMs)

Instead of teaching a computer from scratch what Parkinson's looks like (which takes a lot of data), the researchers used pre-trained AI models.

• The Analogy: Think of these models as culinary students who have already graduated. They have already learned how to cook (understand movement and video) by tasting millions of dishes (watching millions of videos). The researchers didn't teach them to cook; they just asked, "Can you taste this specific soup and tell me if it's spoiled?"
• They tested 7 different "graduates" (different AI architectures like VideoPrism, V-JEPA, TimeSformer, etc.) to see which one had the best palate for Parkinson's symptoms.

3. The Test: The "Gym Class" for AI

They gathered a massive dataset: 32,847 videos from 1,888 people (about 727 of whom have Parkinson's).
The participants were asked to do 16 different tasks, which the researchers grouped into four "gym stations":

1. The Gymnast Station (Upper Limbs): Tapping fingers, flipping palms, extending arms. (Testing for slowness or stiffness).
2. The Actor Station (Face): Smiling, looking surprised, looking disgusted. (Testing for "masked face," a common Parkinson's symptom).
3. The Orator Station (Speech): Reading a sentence with all the letters of the alphabet, saying vowels, doing tongue twisters. (Testing for slurred or quiet speech).
4. The Pilot Station (Eyes & Head): Following a dot with your eyes, tilting your head, counting backward. (Testing for coordination and focus).

4. The Results: Who Won the Trophy?

The researchers found that no single robot was perfect at everything. It depended entirely on what the robot was watching.

• The "Motion Detective" (V-JEPA): This model was the champion for arm and hand movements. If you wanted to see if someone's hands were stiff or slow, this was the best AI to use. It was like a coach who specializes in gymnastics.
• The "Expression Reader" (VideoPrism): This model was the champion for faces and speech. It was incredibly good at spotting a lack of facial expression or subtle changes in how someone moves their mouth while talking. It was like a drama coach who notices the tiniest twitch in an actor's face.
• The "Rhythm Keeper" (TimeSformer): This one did surprisingly well at finger tapping, which is a very fast, rhythmic task.

The Scorecard:

• The models were pretty good at ruling out healthy people (high "Specificity"). If the AI says "You look healthy," you probably are.
• However, they were not great at catching every single case of Parkinson's (lower "Sensitivity"). They missed about half the people who actually had the disease.
• Why? The AI is like a security guard who is very good at spotting obvious intruders but might miss someone hiding in the shadows. The symptoms of Parkinson's can be very subtle, and the AI needs more training or a combination of different "guards" to catch them all.

5. The Big Takeaway

This study is a roadmap. It tells future developers:

• "If you are building an app to check arm movements, use V-JEPA."
• "If you are building an app to check facial expressions, use VideoPrism."
• "Don't just pick one random AI; pick the right tool for the specific job."

6. The Catch (Limitations)

• Privacy First: They ran the AI on local computers, not the cloud, so no patient videos were sent to big tech companies.
• Demographics: The group tested was mostly White. The AI might not work as well for people of other races, so more diverse data is needed.
• Not a Doctor Yet: The AI is currently a "screening tool," not a diagnostic tool. It can say, "Hey, you should probably see a doctor," but it can't give the final diagnosis.

Summary

Think of this paper as a review of different security cameras. The researchers tested 7 different cameras to see which one is best at spotting a specific type of thief (Parkinson's symptoms). They found that some cameras are great at spotting slow walkers (arm stiffness), while others are great at spotting people with blank faces. By knowing which camera to use for which job, we can build better, more accessible tools to help people get diagnosed with Parkinson's faster, right from their own homes.

Technical Summary

1. Problem Statement

Parkinson's Disease (PD) diagnosis traditionally relies on the MDS-UPDRS scale, requiring in-person clinical evaluations by specialists to assess motor signs like bradykinesia, tremor, and rigidity. This creates significant barriers for underserved populations due to geographic and financial constraints.

• The Gap: While video-based remote screening offers a scalable solution, existing computer vision approaches often rely on handcrafted features (e.g., specific finger-tapping speeds) or task-specific deep learning models (e.g., 3D ConvNets) that lack generalizability and require large labeled datasets.
• The Opportunity: Video Foundation Models (VFMs), pre-trained on massive datasets using self-supervised learning, can learn versatile spatiotemporal representations without task-specific customization. However, it remains unclear which VFM architectures are most effective for the diverse clinical tasks required in PD screening (e.g., rapid limb movements vs. subtle facial expressions).

2. Methodology

A. Dataset Construction

The authors curated a large-scale, heterogeneous dataset comprising:

• Participants: 1,888 individuals (727 with PD, 1,161 without).
• Volume: 32,847 videos.
• Tasks: 16 standardized clinical tasks inspired by MDS-UPDRS, categorized into four domains:
  1. Upper-Limb Motor Kinematics: Finger tapping, flip palm, open fist, extending arms, nose touching.
  2. Visual Speech Kinematics: Uttering pangrams, tongue twisters, and sustained vowel phonation (video-only modality).
  3. Facial Expressivity: Mimicking emotions (smile, disgust, surprise) to assess hypomimia.
  4. Oculomotor, Cervical, & Cognitive Control: Eye gaze tracking, head pose, and reverse counting.
• Demographics: Predominantly White (1,366), with representation from Asian, Black, and other groups.

B. Models Evaluated

Seven state-of-the-art VFMs were benchmarked, representing diverse architectural paradigms and pre-training objectives:

1. VideoPrism: Hybrid approach (video-text contrastive learning + masked video modeling with global-local distillation).
2. V-JEPA2 & V-JEPA2-SSv2: Joint Embedding Predictive Architecture (latent prediction vs. pixel reconstruction). The SSv2 variant is fine-tuned on the Something-Something v2 dataset (human-object interactions).
3. TimeSformer: Divided space-time attention mechanism.
4. ViViT: Factorized spatial and temporal encoders.
5. VideoMAE & VideoMAEv2: Masked autoencoders (pixel reconstruction) with dual masking in v2.

C. Experimental Protocol

• Frozen-Backbone Evaluation: To strictly benchmark the inherent representational power of the models, VFM weights were kept frozen after pretraining. They served solely as feature extractors.
• Classification Head: A simple neural network (one hidden layer with ReLU + linear classification head) was trained on the extracted embeddings to classify PD vs. Non-PD.
• Data Splitting: Stratified by unique participants (60% train, 20% validation, 20% test) to prevent data leakage.
• Privacy: All experiments were conducted locally on a high-performance workstation to ensure patient data never left the premises.

3. Key Contributions

1. First Large-Scale VFM Benchmark for PD: Established a rigorous baseline using 32k+ videos and 16 clinical tasks, moving beyond handcrafted features to foundation model embeddings.
2. Task-Model Saliency Discovery: Demonstrated that model performance is highly dependent on the specific clinical domain, revealing that no single architecture dominates all tasks.
3. Privacy-Preserving Framework: Validated a "frozen-backbone" approach suitable for local deployment, ensuring sensitive medical video data is not exposed to cloud APIs.
4. Public Release: Provided code and anonymized structured data to facilitate future research in remote neurological monitoring.

4. Key Results

A. Overall Performance

• Metrics: The best models achieved AUCs of 76.4% – 85.3% and Accuracies of 71.5% – 80.6%.
• Specificity vs. Sensitivity: Models showed high Specificity (up to 90.3%), making them excellent for ruling out healthy individuals. However, Sensitivity was lower (43.2% – 57.3%), indicating a need for better calibration to detect PD cases reliably.

B. Architecture-Specific Strengths (Task-Model Saliency)

• Upper-Limb Motor Kinematics: V-JEPA2-SSv2 was the clear winner.
  • Top Task: Flip Palm (AUC: 85.3%, Specificity: 90.3%).
  • Reasoning: The SSv2 fine-tuning (focused on directional motion and object interaction) enhanced the model's ability to capture complex temporal dynamics and limb coordination.
• Visual Speech & Facial Expressivity: VideoPrism dominated.
  • Top Tasks: Open Fist, Pangram Utterance, Tongue Twister, Eye Gaze.
  • Reasoning: VideoPrism's semantic-visual distillation is superior at capturing subtle, low-amplitude spatio-temporal features associated with hypomimia (reduced facial expression) and dysarthria.
• Rhythmic/Fine-Motor Tasks: TimeSformer performed best on Finger Tapping (AUC: 76.4%).
  • Reasoning: Its divided space-time attention mechanism is better suited for high-frequency temporal reasoning required for rhythmic tasks.

C. Ablation Studies

• Multi-View Training: Aggregating likelihoods from multiple video views did not significantly improve performance over single-view processing, suggesting diagnostic features are often captured in short segments.
• Oversampling: Artificially balancing classes via oversampling slightly degraded performance, indicating that frozen VFM embeddings are already robust to the existing class imbalance.

5. Significance and Future Directions

• Clinical Utility: The high specificity suggests these models are viable for remote triage/screening to rule out healthy individuals, reducing the burden on clinical specialists.
• Architectural Guidance: The study provides a roadmap for system designers:
  • Use V-JEPA2-SSv2 for gross motor coordination and limb tasks.
  • Use VideoPrism for orofacial, speech, and facial expression analysis.
  • Use TimeSformer for high-frequency rhythmic tasks.
• Limitations & Next Steps:
  • Current "frozen" evaluation does not explore the potential of task-specific fine-tuning (e.g., LoRA).
  • Sensitivity needs improvement; future work should focus on multi-modal integration (combining video with audio or sensor data) and task-aware calibration.
  • The dataset is predominantly White; future studies must address demographic diversity to ensure generalizability.

Conclusion: This work establishes that Video Foundation Models can capture clinically meaningful signs of Parkinson's Disease without task-specific training. By matching the right VFM architecture to the specific physiological domain of the clinical task, remote screening systems can achieve robust performance, paving the way for scalable, accessible neurological monitoring.