🤖 AI

Hierarchical and Multimodal Data for Daily Activity Understanding

This paper introduces DARai, a comprehensive multimodal dataset featuring over 200 hours of continuous recordings from 50 participants across 10 environments with 20 diverse sensors, uniquely structured with a three-level hierarchical annotation system to advance human activity understanding through recognition, localization, and anticipation tasks.

Original authors: Ghazal Kaviani, Yavuz Yarici, Seulgi Kim, Mohit Prabhushankar, Ghassan AlRegib, Mashhour Solh, Ameya Patil

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ghazal Kaviani, Yavuz Yarici, Seulgi Kim, Mohit Prabhushankar, Ghassan AlRegib, Mashhour Solh, Ameya Patil

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to live a normal human life. You can't just show it a video of someone making coffee; you need to understand how they move, what they feel, and why they do things the way they do.

This paper introduces DARai (Daily Activity Recordings for AI), which is essentially a massive, ultra-detailed "training manual" for artificial intelligence, built to help computers understand human behavior in the real world.

Here is the breakdown of what makes DARai special, using some everyday analogies:

1. The "Swiss Army Knife" of Sensors

Most old datasets for AI are like a single-lens camera: they only see what happens visually. If the person is behind a wall or the light is bad, the AI gets confused.

DARai is different. Imagine a participant wearing a high-tech suit of armor that includes:

Eyes: Multiple cameras (RGB and Depth) to see the scene.
Muscles: Sensors on the arms to feel how hard they are gripping a cup.
Feet: Pressure sensors in their shoes to feel how they shift their weight.
Heart: Monitors to track their heartbeat and breathing.
Eyes (again): A gaze tracker to see exactly where they are looking.
Radar: To detect movement even if the person is hidden.

The Analogy: If teaching an AI with just a camera is like trying to learn to cook by watching a silent movie, DARai is like letting the AI taste the food, feel the heat of the stove, smell the spices, and hear the sizzle all at once.

2. The "Russian Nesting Doll" of Actions

Human activities aren't just one big block; they are layered. The paper organizes data like a set of Russian nesting dolls:

Level 1 (The Big Picture): The main goal. Example: "Making a Sandwich."
Level 2 (The Steps): The specific actions. Example: "Get bread," "Spread butter," "Put on ham."
Level 3 (The Tiny Details): The exact mechanics. Example: "Twist the knife," "Press down gently," "Lift the slice."

The Analogy: Think of a movie.

Level 1 is the Genre (Comedy).
Level 2 is the Scene (The car chase).
Level 3 is the specific frame (The actor's foot hitting the gas pedal).
DARai teaches the AI to understand the whole movie, the specific scenes, and the tiny details simultaneously.

3. The "What If?" Scenarios (Counterfactuals)

This is the coolest part. In many datasets, everyone does the exact same thing. In DARai, the researchers asked participants to do the same task but with a twist.

Scenario A: Carry a light box.
Scenario B: Carry a heavy box.

Visually, these might look almost identical. But the pressure sensors in the shoes and the muscle sensors in the arms will look totally different. The heavy box makes the feet press harder and the muscles tense up more.

The Analogy: Imagine two people walking down a hallway. One is carrying a feather; the other is carrying a bowling ball. To a camera, they look the same. To DARai's sensors, they are walking on completely different planets. This helps AI learn the physics of human movement, not just the visual appearance.

4. Why This Matters (The "Privacy" Superpower)

Because DARai uses so many non-visual sensors (like heart rate and muscle movement), we can teach AI to recognize activities without needing to see the person's face.

The Analogy: Usually, to recognize someone, you need a photo. With DARai, you can recognize "Someone is sleeping" just by their slow breathing and lack of movement, even if their face is blurred or they are in a dark room. This is huge for privacy in smart homes and hospitals.

5. The "Real World" Test

The researchers didn't just film people in a sterile lab. They filmed them in real kitchens, living rooms, and offices with real lighting, real noise, and real mess. They found that AI models that work perfectly in a lab often fail when the camera angle changes or the lighting gets weird. DARai exposes these weaknesses so engineers can build stronger, more reliable robots.

Summary

DARai is a massive, open-source library of human movement data that combines vision, touch, sound, and biology. It teaches AI to understand not just what we are doing, but how we are doing it, how hard we are working, and why we might do it differently depending on the situation. It's the ultimate "real-world" simulator for the next generation of helpful robots and smart assistants.

1. Problem Statement

Current research in human activity recognition (HAR) faces significant limitations when transitioning from controlled laboratory settings to real-world applications:

Modality Limitations: Visual data (RGB/Depth) is dominant but suffers from occlusions, privacy concerns, viewpoint sensitivity, and environmental noise. Non-visual sensors (wearables) are often underutilized or lack rich multimodal integration.
Data Structure: Existing datasets are often scripted (participants follow rigid scripts) or lack hierarchical annotations. They rarely capture action counterfactuals (alternative executions of the same high-level task under different conditions, e.g., lifting a heavy vs. light object).
Granularity: Most datasets focus on coarse-grained activities, failing to model the nested, hierarchical nature of human behavior (Goal $\to$ Action $\to$ Procedure) and the temporal dependencies required for fine-grained segmentation and anticipation.
Robustness: Models trained on specific camera views or body placements often fail to generalize to cross-view or cross-body scenarios.

2. Methodology: The DARai Dataset

The authors introduce DARai, a comprehensive, open-source, multimodal, and hierarchically annotated dataset designed to address these gaps.

Data Collection

Participants & Environments: 50 participants recorded in 10 unique indoor environments (5 kitchens, 5 living spaces/offices) across 5 locations.
Recording Nature: Continuous, unscripted recordings totaling over 200 hours of data.
Sensor Suite (20 Modalities from 12 Devices):
- Visual: 3rd-person RGB and Depth cameras (Kinect, LiDAR) at multiple angles; 1st-person Eye Tracker.
- Wearable: Inertial Measurement Units (IMUs) on wrists and head; Electromyography (EMG) on forearms; Insole pressure sensors; Biomonitoring (ECG, respiration, heart rate).
- Environmental: Radar, microphones, CO2, humidity, temperature, and light sensors.
Synchronization: A multi-threaded system with a global timestamp signal aligns all streams, achieving a synchronization drift of only 1 ms over 24 hours.
Action Counterfactuals: The dataset explicitly includes "counterfactual" scenarios where the high-level goal (L1) and fine-grained steps (L3) remain constant, but the intermediate actions (L2) or physical conditions (e.g., object weight) vary.

Hierarchical Annotation Structure

The dataset is annotated at three distinct levels, providing a nested taxonomy:

Level 1 (L1) - Activities: High-level, independent tasks (e.g., "Moving an Object," "Sleeping"). (18 unique classes).
Level 2 (L2) - Actions: Intermediate steps shared across activities (e.g., "Pick up," "Carry," "Put down"). (44 unique classes).
Level 3 (L3) - Procedures: Fine-grained execution details (e.g., "Bend," "Lift," "Use blanket"). (98 unique classes).

Language Descriptions: Natural language descriptions are provided for L2 actions to support text-based queries.

3. Key Contributions

Scale and Diversity: The largest available dataset in terms of sensor count (12 devices, 20 modalities) and participant diversity (50 subjects) for daily activity understanding.
Hierarchical & Counterfactual Design: A unique structure that supports the study of action counterfactuals and long-range temporal dependencies, allowing researchers to analyze how specific sensors react to physical variations (e.g., weight changes) that are visually imperceptible.
Comprehensive Benchmarks: The authors provide standardized benchmarks for:
- Activity, Action, and Procedure Recognition.
- Temporal Segmentation (Localization).
- Short- and Long-Horizon Action Anticipation.
- Cross-view and Cross-body robustness.
Open Source: The dataset, code, and documentation are publicly available on IEEE Dataport and GitHub.

4. Experimental Results & Key Findings

The authors conducted extensive experiments using Transformer-based, CNN, and RNN architectures to evaluate the dataset's utility.

A. Visual Robustness & Limitations

Fine-Grained Drop: Visual models (ResNet, MViT, Swin) perform well on L1 (Activity) but suffer significant accuracy drops at L2 and L3.
Cross-View Failure: Visual models degrade severely when tested on camera views different from training (e.g., accuracy dropping from 89% to 15%).
Depth vs. RGB: In kitchen/living room subsets, depth models trained from scratch outperformed RGB models at fine-grained levels, likely due to stable spatial representations. However, fine-tuned RGB models generally outperformed depth models when pre-trained weights were used.

B. Multimodal Fusion & Sensor Strengths

Complementarity: No single modality is sufficient.
- Insole Pressure: Excellent for recognizing weight-bearing changes (e.g., carrying heavy objects) and symmetric movements.
- EMG: Effective for distinct muscle activation patterns (e.g., whisking vs. grasping) but struggles with low-movement tasks.
- Gaze: Crucial for distinguishing activities with similar physical movements but different visual focus (e.g., reading vs. watching TV).
- Biomonitoring: Good for physiological state changes (e.g., heart rate during gaming) but lacks resolution for fine-grained action differentiation.
Fusion Performance: Fusing all wearable sensors (Physiological + Biomechanical + Behavioral) yielded the most robust performance, maintaining over 42% accuracy at L3, significantly outperforming single modalities.

C. Temporal Dependencies & Anticipation

Granularity Impact: Performance decline from L1 to L3 is non-linear. Visual data degrades sharply, while Insole and EMG data show more resilience between L1 and L2.
Anticipation Horizons:
- L2 (Action): Short-horizon anticipation benefits from moderate observation windows (~50%), but long-horizon prediction remains difficult due to the independence of individual actions.
- L3 (Procedure): Long-horizon anticipation improves consistently with longer observation windows, peaking around 70% observation. This indicates fine-grained procedures have stronger temporal dependencies.
Counterfactual Generalization: Models trained on "factual" scenarios (e.g., carrying a light box) struggled to generalize to "counterfactual" scenarios (carrying a heavy box) when relying on contextual biases (e.g., background setting) rather than intrinsic action features.

5. Significance and Impact

Privacy-Preserving AI: DARai demonstrates that fusing non-visual modalities can achieve robust activity recognition, offering a pathway for privacy-focused applications where cameras are undesirable.
Real-World Robustness: By exposing the fragility of vision-only models to viewpoint and placement changes, the dataset drives the development of more robust, multi-sensor fusion architectures.
Scientific Utility: The inclusion of action counterfactuals allows for the study of causal relationships and execution variability, which is critical for robotics, assistive technologies, and health monitoring.
Benchmark Standard: DARai sets a new standard for evaluating temporal modeling and hierarchical learning in unstructured, real-world environments, moving beyond the limitations of scripted, single-modality datasets.

In conclusion, DARai is a foundational resource that bridges the gap between controlled AI research and the complexities of real-world human activity, emphasizing the necessity of multimodal, hierarchical, and counterfactual-aware data for the next generation of human-centered AI.