🤖 AI

Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Models

Manual2Skill++ is a vision-language framework that enhances robotic assembly by treating connections as primary entities, automatically extracting structured connection data from instruction manuals to build hierarchical task graphs for reliable execution across diverse scenarios.

Original authors: Chenrui Tie, Shengxiang Sun, Yudi Lin, Yanbo Wang, Zhongrui Li, Zhouhan Zhong, Jinxuan Zhu, Yiman Pang, Haonan Chen, Junting Chen, Ruihai Wu, Lin Shao

Published 2026-03-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chenrui Tie, Shengxiang Sun, Yudi Lin, Yanbo Wang, Zhongrui Li, Zhouhan Zhong, Jinxuan Zhu, Yiman Pang, Haonan Chen, Junting Chen, Ruihai Wu, Lin Shao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a complex piece of furniture, like an IKEA bookshelf, but you've never seen the instructions before. You have a pile of wood, some screws, and a box of parts. If you just guess where the pieces go, you'll likely end up with a wobbly mess or a pile of wood that doesn't fit together at all.

For a long time, robots have been terrible at this. They are great at moving things around, but they often treat the "glue" or "screws" that hold things together as an afterthought. They focus on the shape of the wood but forget that a screw needs to go into a specific hole at a specific angle to actually work.

Enter "Manual2Skill++": The Robot's New Instruction Manual.

This paper introduces a new system that teaches robots how to build things by reading the same instruction manuals humans use. Here is how it works, broken down into simple concepts:

1. The "Connectors" are the Stars of the Show

Most robot builders look at a puzzle and see shapes. This new system looks at a puzzle and sees how the pieces lock together.

Think of it like a game of Lego.

Old way: The robot sees a red block and a blue block and tries to stack them.
Manual2Skill++ way: The robot sees the red block and says, "Ah, this has a 'stud' (a connector). The blue block has a 'hole.' I need to find the exact spot where the stud fits the hole, and I need to know which stud fits which hole."

The system treats these connections (screws, dowels, pegs) as the most important part of the plan. It doesn't just say "put the leg on the table"; it says "put the leg on the table using two wooden dowels at these specific coordinates."

2. The Robot's "Super-Reader" (The Vision-Language Model)

How does the robot learn this? It uses a special AI brain (called a Vision-Language Model) that is like a super-observant librarian.

The Input: You give the robot the instruction manual (the pictures and diagrams).
The Magic: The robot doesn't just look at the picture; it "reads" the diagram. It understands that a little circle in the drawing means "screw here," and a little line means "slide this peg in."
The Output: It turns the messy pictures into a structured map (a hierarchical graph). Imagine turning a messy recipe into a perfectly organized checklist that tells you exactly which ingredient goes with which step.

3. The "Snap-Fit" Math

Once the robot has the map, it needs to figure out exactly where to put the pieces.

In the past, robots had to guess and check, moving a part back and forth until it fit. This was slow and often failed.
Manual2Skill++ is different. Because it knows exactly where the "holes" and "pegs" are supposed to meet, it can do the math instantly. It's like having a magnet that pulls two pieces together perfectly the first time. It calculates the exact position so that when the robot moves the piece, it snaps right into place with millimeter precision.

4. The "Real-World" Test

The researchers didn't just test this on a computer screen. They built a simulation where robots had to build:

A chair (using wooden dowels).
A shoe shelf (using screws).
A toy airplane.
A LEGO figure.

They found that without paying attention to the connectors, the robots failed. But with this new system, the robots could build these complex items with high accuracy, even when the parts were tricky to align.

The Big Picture: Why Does This Matter?

Think of assembly as a dance.

Old Robots: They knew the steps (move arm left, move arm right) but didn't know the rhythm. They often stepped on each other's toes or missed the beat.
Manual2Skill++: It reads the sheet music (the manual) and understands the rhythm (the connectors). It knows exactly when to step, where to hold hands, and how to spin so the dance is perfect.

In summary: This paper teaches robots to stop guessing and start reading. By focusing on the tiny details of how things connect (screws, pegs, joints), robots can finally build complex things just like humans do, turning a pile of parts into a finished product without needing a human to hold their hand every step of the way.

1. Problem Statement

Robotic assembly has traditionally focused on geometric alignment and motion planning, often treating the physical connections between parts (e.g., screws, dowels, mortise-tenon joints) as secondary or predefined constraints. This approach creates a critical gap:

Lack of Connection Awareness: Existing methods often ignore the specific type, quantity, and placement of connectors required for structural integrity.
Complexity of Real-World Assembly: Practical tasks (furniture, toys, industrial parts) involve diverse connector types with distinct physical constraints (e.g., torque for screws vs. precise insertion for dowels).
Long-Horizon Dependencies: Assembly involves long sequences of steps where errors in early connection planning propagate, making robust execution difficult without explicit modeling of connector relationships.
Data Scarcity: There is a lack of datasets and benchmarks that explicitly model connector mechanics and provide ground-truth connection information derived from human instruction manuals.

2. Methodology

The authors propose Manual2Skill++, a vision-language framework that bridges high-level task understanding from instruction manuals to low-level robotic execution. The methodology consists of three core components:

A. Connection-Enriched Hierarchical Graph Representation

Instead of treating assembly as a simple sequence of part placements, the authors represent tasks as a hierarchical graph $G = \{V, E_{comp}, E_{eqv}, E_{conn}\}$ :

Nodes ( $V$ ): Represent atomic parts and sub-assemblies.
Composition Edges ( $E_{comp}$ ): Link parent sub-assemblies to child components.
Equivalence Edges ( $E_{eqv}$ ): Link identical parts (e.g., four identical table legs).
Connection Edges ( $E_{conn}$ ): The core innovation. These edges explicitly encode physical connections between sibling nodes. Each edge contains attributes for:
- Connection Type: (e.g., screw, dowel, mortise-tenon).
- Connector Identifier: Specific ID of the connector used.
- Attachment Features: Precise 3D position and normal vectors of attachment points on both parts involved.

B. VLM-Guided Graph Generation

The framework uses a Reasoning Vision-Language Model (VLM) to parse assembly manuals and instantiate the hierarchical graph:

Input: Manual images (sketches/annotations) and high-fidelity 2D renderings of 3D part models with candidate attachment points.
Two-Stage Prompting Strategy:
1. Stage 1 (High-Level Inference): The VLM identifies the number and type of connectors needed for a specific step and which components are involved. This reduces the search space for attachment points.
2. Stage 2 (Precise Pairing): The VLM predicts the exact binary pairs of attachment points to be joined by each connector, resolving the combinatorial problem of matching abstract sketches to detailed 3D models.
Output: A structured graph encoding the full assembly sequence and connection constraints.

C. Connection-Constrained Pose Alignment

Once the graph is generated, the system computes precise relative poses for parts using geometric constraint optimization, eliminating the need for separate, error-prone pose estimation modules.

Optimization Problem: The system minimizes the distance between attachment point pairs while enforcing that their normal vectors are collinear and opposite.
$\min_{R,t} \sum_{i=1}^{k} (|R x^a_i + t - x^b_i|^2 + \alpha |R n^a_i + n^b_i|^2)$
Result: This yields millimeter-level accuracy in aligning parts, which is critical for successful insertion or fastening.

3. Key Contributions

Connector-Aware Representation: A novel hierarchical graph structure that explicitly models connection relationships (types, quantities, and spatial constraints) as primary entities, rather than afterthoughts.
Manual2Skill++ Framework: A VLM-based pipeline that automatically extracts structured connection information from unstructured instruction manuals, bridging the gap between human-readable instructions and robotic execution.
Comprehensive Dataset & Benchmark:
- Dataset: Curated 20+ diverse assembly tasks (IKEA furniture, toys, industrial parts) with precise 3D models and annotated ground-truth connection sites.
- Simulation Benchmark: Implemented in Isaac Lab with four complex, long-horizon tasks featuring realistic connection mechanics (mortise-tenon, dowels, screws) and multi-stage joint behaviors.

4. Experimental Results

The authors evaluated the system across three dimensions:

Graph Extraction Accuracy:
- Manual2Skill++ significantly outperformed baselines (Random Sampling, Geometric Matching, and non-reasoning VLMs).
- Using a reasoning VLM (Gemini-2.5-Pro), the system achieved a 69.41% success rate in correctly identifying connection pairs (Pair Matching), compared to 28.24% for non-reasoning models.
- The system demonstrated robustness, successfully inferring missing steps even when manual pages were omitted.
Pose Alignment Precision:
- The connection-matching approach achieved millimeter-level accuracy (RMSE: 0.0013, Chamfer Distance: 0.005).
- This drastically outperformed state-of-the-art baselines (SingleImage, Manual2Skill), which suffered from centimeter-level errors, insufficient for precise assembly.
Robotic Execution (Connection Policy):
- In simulation, the system guided a robot to perform assembly tasks.
- Using a Force-Position Hybrid strategy (combining grid search with force feedback), the system achieved connection success rates between 73.7% and 81.6% across diverse tasks (Chair, Shoe Shelf, Plane, LEGO Figure).
- This confirmed that the high-precision pose alignment derived from the graph representation enables robust execution even with simple local search strategies.

5. Significance and Impact

Paradigm Shift: Moves robotic assembly from purely geometric reasoning to connection-aware reasoning, acknowledging that the "how" of joining parts is as critical as the "where."
Generalization: Demonstrates that VLMs can effectively interpret abstract human instructions to solve complex, long-horizon physical tasks without task-specific training.
Real-World Applicability: By validating on realistic connector types (screws, dowels) and complex dependencies, the work lays the groundwork for deploying robots in unstructured environments like furniture assembly and manufacturing.
Resource Availability: The release of a diverse dataset and a realistic simulation benchmark (Isaac Lab) provides a standardized testbed for future research in contact-rich manipulation and multimodal task understanding.

In summary, Manual2Skill++ successfully decouples the complexity of assembly by treating connections as explicit, primary constraints, leveraging VLMs to extract this knowledge from manuals, and utilizing geometric optimization to achieve the precision required for reliable robotic assembly.