IMPACT: A Dataset for Multi-Granularity Human Procedural Action Understanding in Industrial Assembly

1. Problem Statement

Current datasets for training intelligent industrial assistants suffer from significant gaps when applied to real-world deployment:

• Lack of Realism: Existing benchmarks often use toy models, simplified artifacts, or near-deterministic linear sequences, failing to capture the complexity of professional industrial workflows.
• Isolated Evaluation: Tasks such as action segmentation, error detection, and state tracking are typically evaluated in isolation. Real deployment requires a system to simultaneously handle occlusion, non-linear execution paths, and anomaly recovery.
• Missing Supervision: There is a lack of datasets that jointly provide synchronized ego-exo views, explicit anomaly-recovery labeling, and compliance-aware state tracking within a single real-world workflow.
• The Gap: Systems struggle to verify engineering constraints, track product states under heavy occlusion (hands/tools), and recover from errors without external intervention.

2. Methodology: The IMPACT Dataset

The authors introduce IMPACT, a synchronized five-view RGB-D dataset designed specifically for deployment-oriented industrial procedural understanding.

Data Collection & Setup

• Task: Real-world assembly and disassembly of a commercial angle grinder (12 components, 4 tool types) using professional-grade tools.
• Participants: 13 participants (4 experts, 9 novices) performing 112 trials totaling 39.5 hours of footage.
• Sensors:
  • 4 Exocentric Views: Intel RealSense D455 cameras (Top, Front, Left, Right) capturing RGB-D at 30fps.
  • 1 Egocentric View: Tobii Pro Glasses 3 (RGB + Gaze + Audio) at 25fps.
• Variability: The task follows a partial-order prerequisite graph rather than a fixed sequence, allowing for multiple valid execution paths. It includes two different grinder models to test cross-configuration generalization.
• Cognitive Metadata: Each trial includes NASA-TLX surveys to measure operator cognitive load.

Multi-Granularity Annotation Hierarchy

The dataset features a unique hierarchical annotation system linking different levels of reasoning:

1. Atomic Actions: Hand-specific (Left/Right) fine-grained labels (137 classes, e.g., "pick up," "loosen").
2. Procedural Steps: Coarse workflow segments (26 categories) and completion events (51 categories).
3. Assembly States: Ternary state tracking for 17 components: {misassembled, unassembled, correctly assembled}.
4. Compliance Phases: Explicit labeling of Normal, Anomaly, and Recovery phases. Anomalies are further categorized into 6 types (temporal, spatial, handling, wrong part, wrong tool, procedural).
5. Null Spans: Synchronized "null" spans across views to decouple perceptual limitations (occlusion) from algorithmic failures.

3. Benchmark Suite

IMPACT defines a unified benchmark suite covering four families of tasks, all sharing the same recordings to ensure cross-task consistency:

1. Temporal Understanding:
   • Step-level Segmentation (TAS-S).
   • Bimanual Atomic Action Segmentation (TAS-BL/BR).
2. Cross-View Understanding:
   • Temporal Alignment (CV-TA) and Semantic Matching (CV-SM) across ego-exo views.
3. Action Forecasting:
   • Short-term Anticipation (AF-S) and Long-horizon Forecasting (AF-L) over multiple valid paths.
4. State & Reasoning:
   • Procedure Step Recognition (PSR) via prerequisite graphs.
   • Assembly State Recognition (ASR).
   • Procedural Phase Recognition (PPR) and Anomaly Type Recognition (ATR).

4. Key Results & Baselines

The authors evaluated various state-of-the-art models (e.g., I3D, VideoMAE v2, MViTv2, LLMs like Gemini and Qwen) across the benchmark splits. Key findings include:

• Granularity Gap: Performance drops significantly when moving from coarse step-level segmentation (F1@50 ≈ 71.5) to fine-grained bimanual atomic actions (F1@50 ≈ 28.0).
• The Egocentric Observability Gap: Cross-view alignment is severely limited by occlusion. Models trained on exocentric views fail to align with egocentric views (Global R@1 drops from ~7.4 to ~1.4), indicating that view-invariant pretraining cannot resolve missing visual evidence caused by hands and tools.
• Forecasting Ceiling: Long-horizon forecasting performance is capped by the graph-structural ambiguity of the task. Even advanced LLMs and symbolic reasoning models fail to predict steps beyond 2–3 steps ahead because the prerequisite graph allows too many valid paths, a limitation not solved by increased model capacity.
• Knowledge-Execution Gap: Vision-Language Models (VLMs) show strong procedural ordering priors (knowing what to do) but fail to ground these actions in time (knowing when it happens), resulting in high detection delays.
• Anomaly & Recovery: Current models struggle immensely with recovery phases (F1 scores near zero). The extreme class imbalance (recovery is only 2.22% of frames) and the complexity of corrective behavior remain unsolved challenges.
• Bimanual Asymmetry: The "stabilizing" hand (Left) is easier to predict and track than the "tool-driving" hand (Right), which exhibits sharper, decision-dependent transitions.

5. Significance and Contributions

• First Real Industrial Benchmark: IMPACT is the first dataset to combine synchronized ego-exo RGB-D capture, decoupled bimanual annotation, state tracking, and explicit anomaly-recovery supervision in a real commercial tool workflow.
• Unified Evaluation: It shifts the paradigm from isolated task evaluation to structured reasoning over the full conjunction of conditions (occlusion, non-linearity, errors) required for real deployment.
• Revealing Open Challenges: The benchmark exposes three fundamental limitations invisible to single-task datasets:
  1. The Egocentric Observability Gap (occlusion limits).
  2. The Graph-Structural Forecasting Ceiling (ambiguity in multi-path execution).
  3. The Knowledge-Execution Gap in VLMs (failure to translate recognition into temporal reasoning).
• Resource Availability: The dataset, annotations, and evaluation code are open-source, providing a rigorous testbed for developing next-generation industrial AI assistants capable of handling real-world complexity.

In conclusion, IMPACT demonstrates that achieving deployment-grade industrial understanding requires moving beyond simple action recognition toward joint reasoning over actions, physical states, and corrective behaviors under realistic, imperfect conditions.