From Instructions to Assistance: a Dataset Aligning Instruction Manuals with Assembly Videos for Evaluating Multimodal LLMs

This paper introduces the Manual to Action Dataset (M2AD), which aligns furniture assembly videos with instruction manuals, to evaluate the capabilities and limitations of current Multimodal Large Language Models in providing real-time, multimodal assistance for procedural tasks.

The Gist

Imagine you are trying to build a complex piece of IKEA furniture. You have the instruction manual (a bunch of paper diagrams) and you are holding the parts in your hands. Now, imagine you have a super-smart robot assistant watching you.

The Big Question: Can this robot look at your hands, look at the paper manual, and actually understand what you are doing? Can it tell you, "Hey, you just finished step 3, so now you need to grab the screw from step 4"?

This paper is about building a test track to see how good current AI robots are at being that helpful assistant.

1. The Problem: Old Tests vs. Real Life

Previously, researchers tested AI with simple tasks, like "Is there a cat in this picture?" or "What is the person doing right now?" (like chopping an onion).

But building furniture isn't just one action; it's a story. It's a sequence of events. You have to remember what you did five minutes ago to know what to do next. Old tests didn't check if the AI could follow the whole story. They were like testing a driver by asking them to stop at a red light, but not seeing if they could actually drive from New York to London.

2. The Solution: The "M2AD" Dataset

The authors created a new training ground called M2AD (Manual-to-Action Dataset).

• The Ingredients: They took 53 different IKEA furniture videos from YouTube (real people building real things) and matched them perfectly with the official PDF instruction manuals.
• The Labeling: They didn't just label "Step 1." They marked exactly when Step 1 started and ended in the video, and which page of the manual it was on.
• The Twist: They made it "real." Real people make mistakes. They skip steps, go back to fix things, or take a coffee break. The dataset captures all that messy, human reality, not just a perfect, robotic version of assembly.

Think of M2AD as a gym for AI. Instead of lifting a single weight, the AI has to run a marathon while reading a map.

3. The Test: Putting AI to Work

The researchers took several "open-source" AI models (the kind you can run on a powerful home computer, not just a giant supercomputer) and put them through three specific challenges:

• Challenge A: The "Did I Finish?" Check.

  • Scenario: The AI sees a video clip of you screwing a leg onto a table and a picture of the manual page.
  • Task: Can the AI say, "Yes, you are done with this step"?
  • Result: Some AIs were okay (about 56% accurate), but many were barely better than guessing.

• Challenge B: The "Where Am I?" Check.

  • Scenario: The AI sees you working and is shown two pages from the manual (one is the right page, one is a trick page).
  • Task: Can the AI pick the correct page that matches what you are doing?
  • Result: Again, only the smartest models could do this reliably. Most got confused.

• Challenge C: The "What Number Step?" Check.

  • Scenario: The AI sees you working and is shown two pages.
  • Task: Can the AI say, "You are currently on Step 12"?
  • Result: This was the hardest. Most AIs failed miserably. One model, MolMo, did surprisingly well, but it cheated a little by just looking at the layout of the image (like saying "The text on the left says 12") rather than truly understanding the logic.

4. The Verdict: Promising, But Not Ready for Prime Time

The paper concludes that while AI is getting smarter, it still struggles with long, complex stories.

• The Good News: AI is getting better at understanding that a video and a manual are connected. We don't need to teach the AI every single tiny detail anymore; it can learn some things on its own.
• The Bad News: Current AI models have "short attention spans" and "weak eyes." They can't look at a whole video sequence and a whole manual at the same time without getting overwhelmed. They often get lost in the middle of the story.
• The Hardware Hurdle: To make this work in a real factory or home, the AI needs to run on a regular computer, not a massive server farm. The current models are too heavy and slow for that.

The Bottom Line

This paper is like a report card for the next generation of robot helpers. It says: "You are getting better at reading the map and watching the road, but you still get lost if the trip is too long."

The authors hope that by using this new dataset (M2AD), developers will build better, more patient, and more observant AI assistants that can finally help us build our furniture without getting frustrated!

Technical Summary

1. Problem Statement

The paper addresses the gap between current Multimodal Large Language Models (MLLMs) and their application in real-world technical assistance, specifically in procedural tasks like furniture assembly. While MLLMs have advanced in text and image processing, existing benchmarks fail to evaluate them in complex, multi-step scenarios requiring:

• Sequential Reasoning: Understanding the progression of assembly steps over time.
• Cross-Modal Alignment: Correlating visual video data (real-world execution) with static textual/visual data (instruction manuals).
• Real-Time Assistance: The ability to track progress, detect errors, and identify current steps without relying on heavy, frame-level annotations that are costly to produce.
• Hardware Constraints: The need for models to run on consumer-grade hardware (e.g., single GPUs) to ensure data privacy and low latency in industrial or home settings.

Current datasets (e.g., IKEA Manuals at Work, ENIGMA-51) often rely on atomic, frame-level annotations or focus on isolated tasks (action recognition) rather than holistic procedural understanding.

2. Methodology

A. The M2AD Dataset

The authors introduce M2AD (Manual-to-Action Dataset), a novel benchmark designed to evaluate MLLMs as technical assistants.

• Source: Derived from publicly available YouTube videos of IKEA furniture assembly, paired with official instruction manuals scraped from the IKEA website.
• Scale: Contains 53 unique furniture items and 1,228 annotated steps.
• Annotation Strategy:
  • Minimalist Approach: Instead of dense frame-level labels, the dataset uses step-level alignment. Annotations link video segments to specific steps and page numbers in the manual.
  • Temporal Granularity: Each annotation includes start/end timestamps, step number, and manual page number.
  • Diversity: Includes both egocentric (first-person/VR-like) and exocentric (third-person) viewpoints to simulate different assistance scenarios.
  • Realism: Captures natural user behaviors, including non-consecutive transitions (skipping or revisiting steps), variable execution speeds, and interruptions.
• Quality Control: Inter-Annotator Agreement (IAA) was measured using Intersection-over-Union (IoU) on time intervals, achieving ~70% consistency.

B. Experimental Setup

The authors evaluated 8 open-source MLLMs on consumer-level hardware (NVIDIA RTX 4090, 64GB RAM, Intel i9) to ensure reproducibility and simulate real-world deployment constraints.

• Models Tested: LLaVa-Video, LLaVa-OneVision, Mantis-Idefics2, MFuyu, MolMo, Ovis, Qwen2-VL, and Pixtral.
• Prompting: Zero-shot learning without quantization.
• Tasks: Three specific evaluation tasks were designed:
  1. Step Completion Detection: Given video frames and a manual page, determine if the step shown is completed.
  2. Step Detection (Correspondence): Given video frames and two manual pages (one correct, one distractor), identify if the video matches the instructions on a specific page.
  3. Step Identification: Given video frames and two manual pages, identify the exact step number currently being executed.

3. Key Contributions

1. M2AD Dataset: A new benchmark specifically tailored for procedural reasoning in technical assistance, bridging the gap between static manuals and dynamic video execution.
2. Annotation Paradigm Shift: Demonstrates that effective evaluation of MLLMs in procedural tasks does not require expensive, atomic frame-level annotations; step-level alignment is sufficient to test reasoning capabilities.
3. Hardware-Constrained Evaluation: Provides a baseline for how state-of-the-art open models perform on consumer-grade hardware, addressing privacy and latency concerns in industrial applications.
4. Insight into Model Limitations: Highlights that while models can understand sequences, they struggle with interleaved text-image reasoning and comparing fundamentally different visual modalities (diagrams vs. real-world photos).

4. Results

• Step Completion Detection:
  • Top performers: LLaVa-Video (56.79% Accuracy), Qwen2-VL (56.71%), and Ovis (55.65%).
  • Most models performed near random chance, indicating difficulty in verifying completion states without explicit fine-tuning.
• Step Detection (Correspondence):
  • LLaVa-Video (59.16%) and Qwen2-VL (55.90%) significantly outperformed others.
  • Most models failed to distinguish between relevant and distractor manual pages effectively.
• Step Identification (Multi-class Classification):
  • MolMo was the standout performer with 79.56% Accuracy, significantly outperforming all others.
  • Analysis: MolMo's success is attributed to its unique "pointing" capability and spatial reasoning (using "left/right" references on concatenated images of frames and manuals), despite lacking native interleaved text-image support.
  • Other models struggled, with many scoring close to the random baseline (1/4 or 1/59).

Key Findings:

• Multi-image and Interleaved Reasoning: Models that can process multiple images and interleaved text (like LLaVa-Video) generally perform better, but MolMo's success suggests that strong spatial reasoning and the ability to "point" to entities are critical for this specific task.
• Hardware/Architecture Limits: Performance is heavily constrained by context window limits and memory, forcing models to process truncated data, which hinders long-term sequential reasoning.

5. Significance and Future Directions

• Practical Impact: M2AD provides a realistic testbed for developing AI assistants that can guide users through complex tasks in real-time, potentially replacing human experts in industrial or DIY settings.
• Cost-Efficiency: The study suggests that future datasets can move away from costly, granular annotations, relying instead on the MLLM's pre-trained ability to infer context from raw inputs.
• Future Work:
  • Improving pre-training/fine-tuning to better align visual and textual modalities.
  • Investigating cross-domain generalization (e.g., applying these models to vehicle repair, healthcare, or coding).
  • Developing models capable of long-horizon reasoning without truncation to handle complex, multi-hour assembly tasks.

In conclusion, the paper establishes that while current open-source MLLMs show promise in technical assistance, significant architectural improvements are needed to handle the nuances of comparing static instructions with dynamic, real-world execution, particularly under hardware constraints.