Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment

1. Problem Definition

The paper addresses the challenge of Cross-Depiction Assembly Instruction Alignment. In mixed-reality assembly guidance systems, an AI assistant must align abstract, schematic 2D assembly diagrams (e.g., IKEA manuals) with real-time, photorealistic video feeds of a user assembling furniture.

The core difficulty is the "Depiction Gap":

• Visual Disparity: Diagrams use silhouettes, exploded views, arrows, and minimalistic styles, while video feeds show cluttered, photorealistic scenes with hands and tools.
• Lack of Shared Features: Despite depicting the same procedural step, the two modalities share almost no low-level visual features.
• Temporal Complexity: Unlike static cross-depiction recognition (e.g., sketch-to-photo), assembly requires understanding temporal progression (e.g., distinguishing a leg placed next to a seat vs. a leg screwed into place).

Current Vision-Language Models (VLMs) struggle with this gap, leading to failures in step recognition, error detection, and progress tracking.

2. Methodology

A. IKEA-Bench: A New Benchmark

The authors introduce IKEA-Bench, the first benchmark specifically designed to evaluate cross-depiction alignment for assembly guidance.

• Data Source: Built upon the IKEA Manuals at Work dataset, covering 29 furniture products with 97 assembly videos and wordless manual step diagrams.
• Scale: 1,623 questions across 6 task types:
  • Core Alignment Tasks (Cross-Depiction):
    • T1 (Step Recognition): Match a video segment to one of 4 diagram options.
    • T2 (Step Verification): Binary judgment if a video matches a specific diagram.
    • T3 (Progress Tracking): Locate the current step among all diagrams for a product.
    • T4 (Next-Step Prediction): Predict the next diagram based on the current video.
  • Diagnostic Tasks (Unimodal Bottlenecks):
    • D1 (Video Discrimination): Distinguish between two video segments of different steps (tests video understanding).
    • D2 (Instruction Comprehension): Order three consecutive diagrams correctly (tests diagram understanding).
• Alignment Strategies: The benchmark evaluates models under three input configurations to isolate the impact of text:
  1. Visual: Diagram images + Video (Full depiction gap).
  2. Visual+Text (V+T): Diagram images + Text descriptions + Video.
  3. Text Only: Text descriptions replacing diagrams + Video (Eliminates depiction gap).

B. Evaluation Protocol

• Models Evaluated: 19 VLMs, including 17 open-source models (ranging from 2B to 38B parameters across 8 architecture families like Qwen, InternVL, Gemma, LLaVA) and 2 proprietary models (Gemini 3.1 Pro, Gemini 3 Flash).
• Metrics: Accuracy (4-way MC and Binary), with random chance baselines of 25% and 50% respectively.
• Mechanistic Analysis: A three-layer analysis on 4 representative models (~8B scale) to explain why models fail:
  1. Layer 1 (ViT Representations): Analyzes if diagrams and video share a feature space using Centered Kernel Alignment (CKA), linear probes, and cross-modal retrieval.
  2. Layer 2 (LLM Hidden States): Measures the influence of diagram tokens on the final prediction vector when text is added.
  3. Layer 3 (Attention Routing): Analyzes attention weights to see if text diverts focus away from visual modalities.

3. Key Results

A. Benchmark Performance

• Low Alignment Accuracy: Even the best models struggle. The top open-source model (Qwen3.5-27B) achieves ~59.4% on Step Recognition (T1), while proprietary Gemini models reach ~65.3%. This is far below the reliability required for real-world deployment.
• Procedural Reasoning is Harder: Performance drops significantly for Next-Step Prediction (T4), with an average drop of ~12 percentage points compared to Step Recognition.
• Architecture > Parameters: Model family matters more than parameter count. Newer dense architectures (e.g., Qwen3.5) outperform older families and larger MoE variants at similar scales. Scaling within a single family often yields diminishing returns or even regressions (e.g., Gemma3-12B performs worse than Gemma3-4B).
• The "Hard Ceiling": Video understanding (D1) remains the primary bottleneck. Even the best models score only ~71% on distinguishing video steps, and this performance is invariant across all alignment strategies.

B. The Text Paradox

• Instruction Recovery: Adding text descriptions significantly improves the understanding of the diagrams themselves (D2 scores improve by +23.6% on average when switching from Visual to Text Only).
• Alignment Degradation: Counter-intuitively, adding text degrades cross-depiction alignment (T1 scores drop by ~3.1% on average). Text does not help the model bridge the gap between the diagram and the video; instead, it distracts the model.

C. Mechanistic Insights

• Disjoint Subspaces (Layer 1): Diagrams and video frames occupy disjoint representational subspaces in the Vision Transformer (ViT). CKA scores are near zero (~0.001–0.101), meaning the visual encoder treats them as completely unrelated geometric structures.
• Text Distraction (Layer 2 & 3): When text is added, the model shifts from visual reasoning to text-based reasoning.
  • Hidden States: The influence of diagram tokens on the final prediction vector drops significantly (e.g., -59% in Qwen3-VL-8B).
  • Attention: Text tokens absorb attention that would otherwise be allocated to diagram and video tokens. For example, diagram attention drops from 8.1% to 3.9% when text is introduced.

4. Key Contributions

1. IKEA-Bench: The first benchmark for cross-depiction assembly alignment, featuring 1,623 questions, 6 task types, and 3 alignment strategies across 29 products.
2. Mechanistic Analysis: A novel three-layer analysis that localizes the failure to the visual encoder's inability to map diagrams and video to a shared space, and reveals that text acts as a distraction rather than a bridge.
3. Empirical Guidelines:
   • Strategy Selection: Use "Visual" inputs for alignment tasks (T1, T3) and "Text Only" for instruction comprehension (D2).
   • Model Selection: Prioritize newer architecture families over simply scaling parameters.
   • Future Direction: Improving cross-depiction alignment requires better visual encoding (e.g., contrastive pre-training on schematic-to-real pairs), not just better text integration.

5. Significance

This paper fundamentally challenges the assumption that adding text descriptions to VLMs will solve complex visual alignment problems. It demonstrates that for tasks involving fundamentally different visual styles (schematic vs. photorealistic), the bottleneck is visual representation learning, not language grounding.

The findings suggest that future research in assembly guidance and cross-depiction recognition must focus on training visual encoders to recognize structural and procedural similarities across disparate visual domains, rather than relying on multimodal text bridges which may actively hinder performance. This has broad implications for robotics, AR/VR assistants, and any system requiring the alignment of technical schematics with real-world execution.