OmniFall: From Staged Through Synthetic to Wild, A Unified Multi-Domain Dataset for Robust Fall Detection

The paper introduces OmniFall, a unified multi-domain benchmark comprising 80 hours of harmonized staged, synthetic, and in-the-wild videos with dense frame-level annotations, designed to enable robust and privacy-preserving fall detection models that generalize effectively to uncontrolled real-world environments.

The Gist

Imagine you are trying to teach a robot how to spot when an elderly person falls down. This is a life-or-death skill; if the robot misses a fall, the person might lie on the floor for hours without help, which can be fatal.

The problem is that most robots today are trained like actors in a movie studio. They practice on healthy people pretending to fall in perfect, well-lit rooms with fixed cameras. When you take that robot out into the real world—where lighting is bad, cameras shake, and people are actually hurt and confused—the robot often gets confused and fails.

This paper introduces OmniFall, a massive new "training gym" designed to fix this. Think of it as a three-part boot camp that prepares the robot for any situation.

The Three Parts of the OmniFall Boot Camp

1. The "Scripted Rehearsal" (OF-Staged)
This is the old way of training. The researchers gathered eight different existing datasets where healthy actors pretended to fall in controlled environments.

• The Analogy: Imagine a flight simulator where the weather is always sunny and the pilot is a calm instructor. It's good for learning the basics, but it doesn't prepare you for a real storm.
• What they did: They took these eight different "simulators," cleaned them up, and gave them all the same rulebook so the robot can learn from all of them at once.

2. The "Digital Twin" (OF-Synthetic)
This is the paper's big innovation. Instead of asking real elderly people to fall (which is dangerous and unethical), they used advanced AI video generators to create 12,000 fake videos.

• The Analogy: Imagine a video game where you can spawn a character who is 90 years old, 6 feet tall, wearing a red coat, and falls in a rainy kitchen. Then you spawn another who is 80, short, wearing pajamas, and falls on a sunny porch. You can do this thousands of times in seconds.
• The Magic: The researchers found something surprising: The fake videos actually trained the robot better than the real "rehearsal" videos. Because the fake videos could cover every possible body type, age, and environment without privacy risks, the robot learned to recognize the concept of a fall, not just the specific look of a rehearsal.

3. The "Real-World Stress Test" (OF-In-the-Wild)
This is the final exam. The researchers collected 2.6 hours of genuine accident videos from the internet (where people actually fell).

• The Analogy: This is the "road test." The robot has to prove it can spot a fall in a messy living room, with a shaky camera, bad lighting, and a person who is genuinely in distress.
• The Rule: The robot is never allowed to train on these videos. They are strictly for testing to see if the training actually worked.

The Big Discovery

The researchers ran a massive experiment to see which training method worked best. Here is what they found:

• The Old Way: If you only train on the "Scripted Rehearsals," the robot is great at the studio but fails miserably in the real world.
• The New Way: If you train the robot on the "Digital Twins" (Synthetic data), it performs better on real accidents than the robot trained on real actors.
• The Ultimate Combo: The best robot was trained on a mix of the "Digital Twins" and the "Scripted Rehearsals." This combination gave the robot the best of both worlds: the motion patterns from real actors and the diverse, realistic variety from the AI-generated videos.

Why Does This Matter?

1. Privacy: We don't need to film vulnerable, elderly people falling to teach robots. We can use AI to generate the data safely.
2. Diversity: Real datasets usually only have young, fit actors. The AI can generate data for toddlers, the elderly, different body types, and different skin tones, ensuring the robot works for everyone.
3. Safety: By using this new benchmark, we can build fall-detection systems that actually work in the messy, unpredictable real world, potentially saving thousands of lives.

In short: OmniFall is a new, super-charged training manual for robots. It proves that sometimes, fake data (generated by AI) is actually better than real data for teaching machines how to handle the messy reality of human life, as long as you test them on the real thing at the end.

Technical Summary

1. Problem Statement

Visual fall detection systems face critical limitations in real-world deployment due to:

• Data Scarcity and Bias: Existing datasets consist of small, staged recordings of healthy actors in controlled environments, lacking the diversity of real-world accidents (lighting, occlusion, camera motion).
• Annotation Heterogeneity: Different datasets use incompatible label taxonomies (e.g., binary fall/no-fall vs. coarse ADL categories), preventing cross-dataset comparison and unified training.
• Evaluation Gaps: There is no standardized protocol to evaluate how well models trained on staged or synthetic data generalize to genuine "in-the-wild" accidents.
• Ethical Barriers: Collecting diverse fall data from vulnerable populations (elderly, children) raises significant privacy and ethical concerns.
• State Limitation: Current systems often detect the brief event of a fall but fail to recognize the sustained fallen state (lying on the ground), which is critical for medical intervention ("long-lie" episodes).

2. Methodology: The OmniFall Benchmark

The authors introduce OmniFall, a unified benchmark comprising three complementary domains totaling ~80 hours of video (33 hours of unique single-view footage) with dense, frame-level annotations.

A. Three Core Components

1. OF-Staged (OmniFall-Staged):

   • Source: Aggregates 8 public datasets (CMDFall, UP Fall, Le2i, GMDCSA, EDF, OCCU, MCFD, CAUCA).
   • Scale: 14 hours of unique recordings (42 hours multi-view) across 101 subjects and 29 camera views.
   • Standardization: All videos are re-annotated to a unified taxonomy. Standardized Cross-Subject (CS) and Cross-View (CV) splits are provided to enable rigorous generalization testing within the staged domain.

2. OF-In-the-Wild (OF-ItW):

   • Source: Curated from the OOPS dataset (accident videos).
   • Scale: 818 videos (~2.6 hours) of genuine accidents.
   • Characteristics: Uncontrolled environments, natural camera motion, varied lighting, occlusions, and clutter.
   • Role: Serves as a test-only Out-of-Distribution (OOD) benchmark to measure the generalization gap from staged/synthetic data to real accidents.

3. OF-Synthetic (OF-Synthetic):

   • Source: Generated using the Wan 2.2 video diffusion model.
   • Scale: 12,000 videos (~17 hours).
   • Controlled Diversity: Systematically varies age (toddlers to elderly), ethnicity (7 OMB categories), skin tone (Monk Scale), body type (BMI), gender, environment, and camera placement.
   • Privacy: Addresses ethical concerns by avoiding the collection of real vulnerable subjects while providing demographic diversity often missing in staged data.

B. Unified Taxonomy

The paper proposes a 16-class frame-level taxonomy to support both video classification and timeline segmentation:

• Transient Actions: Fall, sit_down, lie_down, stand_up.
• Locomotion/Transition: Walk, kneel_down, squat_down, crawl, jump.
• Static States: Fallen, sitting, lying, standing, kneeling, squatting.
• Other: Miscellaneous activities.
• Significance: This allows the detection of medically critical "fallen" states even if the initial "fall" event was missed, and disambiguates similar actions (e.g., lying down vs. falling).

C. Experimental Protocol

• Domain Shift Analysis: Uses Fréchet Video Distance (FVD) and t-SNE/h-NNE visualizations on frozen features (I3D, VideoMAE-K400, SigLIP2) to quantify the gap between domains.
• Classification: Fine-tunes VideoMAE-K400 on different training combinations (Staged only, Synthetic only, Combined) and evaluates on OF-ItW.
• Segmentation: Uses ASQuery with I3D (motion-centric) and SigLIP2 (appearance-centric) features for timeline segmentation tasks.

3. Key Contributions

1. Unified Benchmark: The first dataset to unify staged, synthetic, and wild data under a single, dense, 16-class taxonomy with standardized evaluation protocols.
2. Synthetic Data Superiority: Demonstrates that carefully designed synthetic data can bridge the domain gap to real-world accidents more effectively than traditional staged data.
3. Privacy-Preserving Diversity: Provides a scalable method to generate diverse demographic data (age, body type, ethnicity) without ethical risks associated with filming vulnerable populations.
4. Granular Annotation: Releases dense, frame-level timelines enabling research into both fall detection and the critical "fallen state" segmentation.

4. Key Results

• Generalization Performance:
  • Training on OF-Staged (aggregated 8 datasets) improved OF-ItW performance (Fall F1: ~60.5–61.2%) compared to training on a single dataset like CMDFall (Fall F1: ~52.3%).
  • Synthetic Data Outperforms Staged: Training exclusively on OF-Synthetic achieved the highest single-source performance on OF-ItW (Fall F1: 64.2%), surpassing the aggregated staged data.
  • Combined Training: Combining Staged and Synthetic data yielded the best overall result (Fall F1: 65.6%), though the marginal gain over synthetic-only suggests synthetic data captures most necessary variation.
• Feature Sensitivity:
  • Appearance-centric features (SigLIP2) benefit significantly from synthetic diversity, outperforming motion-centric features (I3D) on the wild test set.
  • Motion-centric features (I3D) rely more heavily on the dynamics found in staged data.
  • This indicates a complementary relationship: synthetic data provides appearance diversity, while staged data provides realistic motion dynamics.
• Timeline Segmentation:
  • Models trained on synthetic data achieved F1@50 of 27.60% on OF-ItW, significantly outperforming staged-only training (20.26%) when using appearance-based features.
  • There is a significant performance drop (OOD gap) for all models when moving from controlled to wild environments, highlighting the difficulty of the task.

5. Significance and Impact

• Paradigm Shift: The paper challenges the assumption that real staged data is always superior to synthetic data for real-world generalization. It suggests that controlled diversity in synthetic data can be more effective for OOD generalization than uncontrolled but homogeneous staged data.
• Ethical Advancement: By validating synthetic data as a viable (and sometimes superior) alternative, the benchmark offers a path forward for developing robust fall detection systems without compromising the privacy or safety of the elderly.
• Clinical Relevance: The focus on "fallen state" detection and timeline segmentation addresses the critical need to identify patients who are unable to summon help, potentially reducing mortality rates associated with "long-lie" episodes.
• Standardization: OmniFall establishes a rigorous, reproducible protocol for evaluating cross-domain generalization, setting a new standard for future research in human activity recognition and safety monitoring.

Project Page: http://simplexsigil.github.io/omnifall/