OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments

OpenMarcie is a comprehensive multimodal dataset comprising over 37 hours of data from 36 participants performing industrial assembly tasks, designed to advance human activity recognition in smart factories through diverse sensing modalities and benchmarked across classification, captioning, and alignment tasks.

The Gist

Imagine you are trying to teach a robot how to fix a bicycle or build a 3D printer. If you only show the robot a video, it's like trying to learn how to ride a bike by watching a movie of someone else doing it—you see the movement, but you don't feel the balance, the wind, or the effort.

OpenMarcie is a massive, super-detailed "training manual" for robots and AI, designed specifically for the messy, real world of factories and workshops. It's not just a video library; it's a multisensory experience that captures human work from every angle possible.

Here is a breakdown of what makes this dataset special, using some everyday analogies:

1. The "Full-Body Suit" vs. Just a Camera

Most old datasets are like security cameras in a store: they only see what's happening from the outside (the "exocentric" view). They miss the details of what the worker's hands are actually feeling.

OpenMarcie is different. It's like giving every worker a super-powered "Iron Man" suit that records everything:

• Eyes: Cameras on their heads and chests (what they see).
• Muscles: Sensors on their wrists and heads that feel every tiny shake, turn, and movement (like an accelerometer in your phone).
• Ears: Microphones on their chests listening to the click of a screwdriver or the whir of a motor.
• Skin: Thermal cameras and temperature sensors feeling the heat of a tool.
• Voice: A narrator describing what is happening in real-time.

This creates a "360-degree" understanding of the task, not just a flat picture.

2. Two Different "Gym Classes" for the Data

The researchers didn't just record people doing one thing. They set up two very different "workout scenarios" to test how well AI can learn:

• Scenario A: The "Freestyle" Bike Build (Ad-hoc)
  Imagine a group of friends trying to assemble a bicycle without a manual. They have to figure it out as they go. Some might put the seat on first; others might start with the wheels. They might make mistakes, fix them, and talk about it.

  • Why this matters: This teaches AI how to handle chaos, creativity, and problem-solving. It's like teaching a robot to navigate a busy kitchen where everyone is improvising.

• Scenario B: The "Strict" 3D Printer Build (Procedural)
  Now, imagine a group building a complex 3D printer, but they have to follow a strict, step-by-step instruction manual. They can't skip steps. If one person stops, the next person has to pick up exactly where they left off.

  • Why this matters: This teaches AI how to handle precision, rules, and teamwork. It's like teaching a robot to work on an assembly line where timing and sequence are everything.

3. The "Secret Sauce": Why It's Better Than Others

Think of previous industrial datasets as a black-and-white photo album. They have pictures, maybe some notes, but they lack depth.

OpenMarcie is like a VR simulation with haptic feedback.

• It captures "Concurrent" actions: In real life, you can walk while carrying a box. Old datasets often force you to choose one action or the other. OpenMarcie understands you can do both at once.
• It's "Multimodal": It combines sight, sound, and touch. If the robot can't see a tool because your hand is blocking the camera, it can still "hear" the screwdriver tightening or "feel" the vibration in the wrist sensor.
• It's Realistic: The people building these things aren't actors following a script. They are real engineers and students making real decisions, getting tired, and correcting mistakes.

4. What Can We Do With This?

Once AI is trained on OpenMarcie, it can become a super-smart factory assistant:

• Safety Guardian: It can spot if a worker is about to drop a heavy part or use a tool incorrectly before an accident happens.
• The Perfect Coach: It can watch a new employee and say, "Hey, you're holding that wrench at a weird angle; try this instead," just like a human mentor.
• Privacy Protector: Because the system uses sensors and audio (which can be anonymized) rather than just facial recognition, it can monitor safety without invading people's privacy or leaking trade secrets.

The Bottom Line

OpenMarcie is the ultimate "training ground" for the next generation of industrial AI. It moves beyond simple "watch and copy" learning to "feel, hear, and understand" learning. By giving AI a rich, multi-sensory view of human work, it helps us build smarter, safer, and more helpful robots for the factories of the future.

Technical Summary

1. Problem Statement

The integration of human intelligence into automated smart factories requires robust Human Activity Recognition (HAR) systems. However, existing industrial datasets suffer from three critical limitations:

1. Lack of True Multimodality: Most datasets rely heavily on vision alone or lack synchronized integration of wearable sensors, audio, and vision.
2. Over-Constrained Tasks: Existing benchmarks often use scripted, short-duration actions that fail to capture the open-ended, procedural, and multi-step nature of real-world industrial workflows.
3. Limited Diversity: There is a lack of demographic diversity and complex task scenarios (e.g., collaborative assembly, error correction) necessary for training generalizable AI models.

Furthermore, vision-only methods in factories raise privacy concerns and risk technology leakage. The paper addresses the need for a large-scale, multimodal dataset that captures realistic, unscripted industrial assembly tasks with synchronized data from multiple modalities.

2. Methodology and Dataset Design

The authors present OpenMarcie, a dataset comprising over 37 hours of data collected from 36 participants across two distinct experimental scenarios.

A. Experimental Scenarios

1. Ad-hoc Scenario (Bicycle Assembly/Disassembly):
   • Nature: Goal-oriented but open-ended. Participants assemble/disassemble a bicycle without fixed instructions, allowing for divergent problem-solving and natural error correction.
   • Complexity: Involves ~170 components, requiring tool usage, alignment, and fastening.
   • Collaboration: Includes sequential collaborative assembly where participants assess and correct each other's work.
2. Procedural Scenario (3D Printer Assembly):
   • Nature: Highly procedural. Participants follow detailed manufacturer instructions (Prusa i3 MK3S+) to assemble a 3D printer.
   • Complexity: Requires interpreting schematics, handling small components (screws) and large frames, and sequential decision-making.
   • Collaboration: Participants continue the assembly where the previous volunteer left off, necessitating status assessment and adaptation.

B. Multimodal Sensing Setup

The dataset features 8 distinct modalities and 282 independent information channels, collected via a synchronized setup:

• Wearables (Egocentric):
  • IMUs & Magnetometers: Mounted on wrists, forehead, and head (glasses frame) for motion and orientation.
  • Barometers & Temperature: Mounted on wrists/head/chest for vertical displacement and environmental context.
  • Spectrometers & Thermal Cameras: Mounted on chest/shoulder for material differentiation and contact heat.
  • Egocentric RGB-D: Chest-mounted cameras (including LiDAR) for first-person depth and visual data.
• Exocentric (Third-Person):
  • Multi-view RGB-D: Three ZED X AI stereo cameras distributed around the room to capture global scene context.
• Audio:
  • Stereo Microphones: Captured instrumental sounds (tool interactions) while anonymizing human speech (voices removed, replaced with instrumental textures) to protect privacy.
• Annotations:
  • Ground Truth: Manual annotation using an intent-aware Verb–Object–Tool scheme.
  • LLM Integration: Large Language Models (GPT-4o, DeepSeek-r1) were used as "structured translators" to convert human-written soft labels into standardized hard labels and vice-versa, ensuring consistency without replacing human judgment.

3. Key Contributions

1. First Truly Multimodal Industrial Dataset: OpenMarcie is the only dataset to unify wearables, egocentric + exocentric multiview video, multi-action labels (concurrent actions), and full industrial coverage.
2. Realistic Workflow Simulation: Unlike scripted datasets, it captures unscripted, goal-oriented, and procedural tasks, including error correction and collaborative handovers.
3. Rich Metadata & Privacy: Includes detailed participant demographics (age, height, handedness, expertise, nationality) and employs rigorous privacy measures (face blurring, voice removal) while retaining instrumental audio for tool-contact analysis.
4. Comprehensive Benchmarking: The dataset is evaluated across three core tasks:
   • Human Activity Recognition (HAR).
   • Open Vocabulary Captioning.
   • Cross-Modal Alignment.

4. Results and Benchmarks

The authors established baselines using state-of-the-art models (ViT, DeepConvLSTM, EnCodec, and Late-Fusion Transformers).

• Human Activity Recognition (HAR):
  • Best Performance: Inertial + Vision (I + V) consistently outperformed other combinations (Macro F1: ~0.88 for Bicycle, ~0.77 for 3D Printer).
  • Multimodal Gain: Fusion of all modalities (I + A + V) generally improved performance over unimodal inputs, though audio alone performed poorly.
• Open Vocabulary Captioning:
  • Best Performance: Inertial + Vision achieved the highest cosine similarity scores.
  • Audio Limitation: Audio alone contributed minimally to captioning, likely due to the anonymization process removing semantic speech content.
• Cross-Modal Alignment:
  • Best Performance: Combining Inertial + Vision + Text yielded the strongest alignment (Recall@1 ~0.48).
  • Key Insight: While audio alone is weak, it provides complementary cues when fused with other modalities.
• Tool-Contact Detection (Supplementary):
  • A critical finding: While audio is poor for general activity recognition, it is highly effective for detecting tool-contact events (e.g., hammering, screwing). Adding audio to Vision+IMU improved F1 scores significantly (e.g., from 0.905 to 0.923 in the bicycle scenario), proving its value for specific physical interaction detection.

5. Significance and Impact

• Industrial AI Advancement: OpenMarcie provides a necessary testbed for developing Embodied AI and Human-Robot Collaboration (HRC) systems that can understand complex, multi-step industrial workflows.
• Privacy-Preserving Sensing: By demonstrating that anonymized audio (instrumental only) still holds value for tool-contact detection, the dataset offers a pathway for deploying HAR in factories without violating privacy or trade secrets.
• Methodological Rigor: The use of LLMs as a translation layer for annotation validation (bidirectional consistency checks) sets a new standard for scalable, high-quality dataset creation.
• Future Research: The dataset enables research into procedural planning, skill assessment, intent prediction, and cross-modal generation, moving beyond simple action classification to complex workflow understanding.

In summary, OpenMarcie bridges the gap between theoretical HAR research and practical industrial application by providing a diverse, privacy-compliant, and richly annotated multimodal dataset that reflects the complexity of real-world manufacturing environments.