🤖 AI

Context-Aware Mapping of 2D Drawing Annotations to 3D CAD Features Using LLM-Assisted Reasoning for Manufacturing Automation

This paper presents a hybrid framework that combines deterministic, context-aware scoring with LLM-assisted reasoning and human-in-the-loop review to accurately map 2D drawing annotations to 3D CAD features, achieving high precision and recall for manufacturing automation in industries where 2D drawings remain the primary source of design intent.

Original authors: Muhammad Tayyab Khan, Lequn Chen, Wenhe Feng, Seung Ki Moon

Published 2026-02-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammad Tayyab Khan, Lequn Chen, Wenhe Feng, Seung Ki Moon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef trying to recreate a complex dish. You have two sources of information:

The 3D Model: A perfect, edible sculpture of the dish sitting on the counter. You can see the shape, the layers, and the ingredients.
The 2D Recipe Card: A flat piece of paper with handwritten notes like "add 10g of salt," "make the crust crispy," or "don't burn the garlic."

The Problem:
In the world of manufacturing (building cars, planes, or machines), engineers have a similar problem. They have a 3D CAD model (the digital sculpture) and a 2D engineering drawing (the recipe card). The 3D model shows what the part looks like, but the 2D drawing holds the secret instructions on how to make it (tolerances, specific measurements, surface finishes).

The trouble is, the 2D drawing is often vague. If the drawing says "10," does it mean a hole that is 10mm wide? A slot that is 10mm deep? Or a distance between two parts? If a robot tries to guess, it might build a part that fits together poorly or breaks immediately.

The Solution: The "Smart Translator" Framework
This paper presents a new system that acts like a super-smart, cautious translator to link the notes on the recipe card to the specific parts of the 3D sculpture.

Here is how it works, step-by-step, using simple analogies:

1. The "Detective" Phase (Deterministic Scoring)

First, the system acts like a strict detective. It doesn't guess; it checks the facts.

Type Check: It asks, "Does this note talk about a hole? Is there a hole in the 3D model nearby?" If the note says "radius" but the model part is a flat square, the detective says, "No match."
The Ruler Check: It measures the numbers. If the drawing says "10mm" and the 3D hole is "10.05mm," it's a match (within a tiny margin of error). If the hole is "20mm," it's a mismatch.
The Context Clue: It looks at where the note is placed. Is the arrow pointing to the top or the side?

The Result: For clear, obvious cases (like a big, unique hole), the detective solves it instantly with 100% certainty.

2. The "Consultant" Phase (VLM Enrichment)

Sometimes, the notes are messy. The handwriting is bad, or the note just says "10" without saying what it measures.

Here, the system calls in a Visual Language Model (VLM)—think of this as a highly educated intern who is great at reading messy handwriting and understanding context.
The intern looks at the cropped image of the note and says, "Ah, this isn't just a number; it's a 'diameter' for a 'threaded hole'."
This turns a confusing scribble into a clear, structured instruction.

3. The "Tie-Breaker" Phase (LLM Reasoning)

What if the detective and the intern still can't decide? Maybe there are five identical holes in a row, and the note points vaguely at the group.

The system escalates to a Large Language Model (LLM)—think of this as a senior engineer with 20 years of experience.
The senior engineer looks at the whole picture, the patterns, and the logic of the design. They might say, "Based on how these parts usually work, this note must refer to the second hole, not the first."
Crucially, this AI doesn't just "guess." It is forced to write down its reasoning (e.g., "I chose this because...").

4. The "Human Safety Net" (Human-in-the-Loop)

This is the most important part. If the AI is still unsure (or if the senior engineer isn't 100% confident), the system stops.

It flags the confusing note and puts it in a "Pending Review" basket.
A real human expert looks at it, makes the final call, and approves it.
This ensures that no bad instructions ever reach the factory floor.

Why is this a big deal?

It's Transparent: Unlike some "black box" AI that just gives an answer, this system explains why it made a choice. You can see the math, the logic, and the human review.
It's Safe: It prioritizes being right over being fast. It would rather ask a human for help than make a mistake.
It Works in the Real World: The authors tested this on 20 real industrial parts. It successfully linked the notes to the parts about 86% of the time without human help, and when it did ask for help, it fixed the remaining errors perfectly.

In Summary:
This paper builds a bridge between the "what" (the 3D model) and the "how" (the 2D drawing). It uses a mix of strict math, smart AI reading, and human oversight to ensure that when a factory robot builds a part, it knows exactly what the engineer intended, preventing costly mistakes and keeping manufacturing automation safe and reliable.

1. Problem Statement

Modern manufacturing automation relies on unifying 2D engineering drawings (which contain critical manufacturing intent like GD&T, tolerances, and surface requirements) with 3D CAD models (which contain precise geometric features).

The Challenge: While Model-Based Definition (MBD) aims to embed this data in 3D models, 2D drawings remain the authoritative source in industries like automotive and aerospace.
The Gap: Automatically linking 2D annotations to 3D features is difficult due to:
- Contextual Ambiguity: A number (e.g., "10") could represent a hole diameter, slot width, or radius depending on drafting conventions.
- Feature Repetition: Identical features (e.g., a pattern of holes) make it hard to know which specific 3D feature a dimension refers to.
- Auditability: Purely data-driven (black-box) AI approaches lack the transparency required for industrial safety and certification. Existing methods either fail to link 2D to 3D or lack auditable decision-making.

2. Methodology

The authors propose a deterministic-first, context-aware framework that prioritizes rule-based logic but escalates to AI only when necessary. The pipeline consists of four main stages:

A. Input & Semantic Enrichment

Inputs: Two pre-extracted sets:
1. 3D Features ( $F_{3D}$ ): Geometric features (holes, slots, etc.) with parameters and confidence scores.
2. 2D Entities ( $E_{2D}$ ): Dimensions, GD&T symbols, and notes extracted via OCR.
Semantic Enrichment (VLM): A Vision-Language Model (VLM) processes cropped images of annotations to convert raw OCR text into structured semantic descriptors (e.g., converting "10" to "diameter: 10mm" with a target feature category). This creates a grounded semantic representation for scoring.

B. Deterministic Correspondence Scoring

A composite score ( $S_{ij}$ ) is calculated for every pair of 3D feature and 2D entity using a weighted sum:
$S_{ij} = w_t S_{type} + w_d S_{dim} + w_c S_{ctx} + h_{ij}$

Type Compatibility ( $S_{type}$ ): Checks if the entity type matches the feature type (e.g., "hole" vs. "hole").
Dimensional Agreement ( $S_{dim}$ ): Compares numeric values with a tolerance threshold ( $\epsilon = 0.1$ mm). It is context-aware (e.g., a diameter dimension is only compared to a 3D diameter, not depth).
Context Consistency ( $S_{ctx}$ ): Uses VLM confidence scores and spatial cues to resolve ambiguity.
Engineering Heuristics ( $h_{ij}$ ): Domain-specific rules (e.g., preferring dimensions with the $\emptyset$ symbol for holes, handling thread callouts like "M8", and managing pattern multiplicities).

C. Assignment & Near-Tie Filtering

Candidates are filtered based on a threshold.
A greedy near-tie strategy allows one-to-many mappings (e.g., one dimension constraining multiple instances of a pattern) if their scores are within a fixed ratio ( $\rho = 0.9$ ) of the best score.

D. Controlled Escalation & Human-in-the-Loop (HITL)

Escalation: If deterministic scoring cannot resolve ambiguity (e.g., multiple candidates have similar scores), the system escalates to a multimodal LLM (GPT-4o).
LLM Role: The LLM analyzes the specific drawing region and candidate list to select the most likely match or explicitly reject candidates if evidence is insufficient.
HITL Review: Unresolved or low-confidence cases are flagged for a single human review step. The final output is a Unified Manufacturing Specification that binds 2D constraints to 3D features with full provenance (mapping method, confidence, and human edits).

3. Key Contributions

Formal Problem Definition: Established a precise mathematical formulation for the 2D–3D manufacturing intent mapping problem with explicit input/output representations.
Deterministic-First Framework: Proposed a novel architecture that combines interpretable scoring with engineering heuristics and controlled LLM escalation. This ensures decisions are transparent, auditable, and manufacturing-grade.
Comprehensive Evaluation: Validated the system on 20 real industrial CAD-drawing pairs, including ablation studies to quantify the contribution of each component (heuristics, LLM, context scoring).

4. Experimental Results

The system was evaluated on a dataset of 20 mechanical parts (101 total mappings) with varying complexity.

Performance Metrics:
- Precision: 83.67%
- Recall: 90.46%
- F1 Score: 86.29%
- Exact Match Rate: 79.19% (perfect correspondence for a feature).
Resolution Statistics:
- 4.0% resolved by deterministic scoring alone.
- 84.1% resolved via the combined deterministic-VLM pathway (LLM selected among near-ties).
- 11.9% required constrained LLM reasoning (LLM made the final decision).
Ablation Study Findings:
- Removing Engineering Heuristics caused precision to collapse from 83.67% to 44.23%, proving the necessity of domain rules.
- Removing LLM Escalation caused recall to drop from 90.46% to 54.17%, proving AI is essential for ambiguous cases.
- The full pipeline outperformed all reduced variants by a significant margin (12.0 F1 points above the next best).

5. Significance and Conclusion

Bridging the Gap: This is the first study to automatically map 2D drawing semantics to 3D CAD features using AI while maintaining the determinism and auditability required for industrial safety.
Practical Deployment: By prioritizing deterministic rules and reserving AI for ambiguous cases, the system minimizes "hallucinations" and provides a clear audit trail, making it suitable for real-world manufacturing automation (process planning, inspection, digital thread integration).
Future Work: The authors plan to scale to larger assemblies, incorporate tolerance stack-up reasoning, and validate on datasets rich in GD&T annotations (which were sparse in the current test set).

In summary, the paper presents a robust, hybrid AI-engineering solution that effectively solves the "2D-to-3D" mapping problem, enabling more reliable and automated manufacturing workflows without sacrificing the transparency needed for industrial certification.