💻 computer science

Structured Extraction and Standardized Reconstruction of Machining Process Documents Based on Vision Language Model under Low Resource Constraints

This paper proposes a vision language model-based framework with constraint example verification feedback and confidence evaluation to achieve high-accuracy structured extraction and standardized reconstruction of heterogeneous machining process documents under low-resource constraints, effectively reducing hallucinations and enabling knowledge digitization in intelligent manufacturing.

Original authors: Li-Xu Mou, Liang Guo, Zhen Yuan, Yi-Qin Xiong

Published 2026-06-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Li-Xu Mou, Liang Guo, Zhen Yuan, Yi-Qin Xiong

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Messy Attic" of Factory Knowledge

Imagine a factory where every time a machine part is made, the instructions are written down on a piece of paper. Over the years, these papers pile up. Some are crisp digital files, some are blurry photos taken with a phone, and some are scanned copies that look like they've been through a washing machine.

The problem is that every engineer writes these instructions differently. One calls a part a "Drawing Number," another calls it a "Product Code," and a third just writes "ID." The layouts are chaotic, the tables are messy, and the handwriting is often hard to read.

Because these documents contain secret company secrets, the factory can't just upload them to a public AI cloud to be read. They need a way to organize this "messy attic" of knowledge locally, without needing thousands of pre-labeled examples (which they don't have because the data is secret).

The Solution: A Smart, Self-Checking Robot Assistant

The authors propose a new method using a Vision Language Model (VLM). Think of this VLM not as a simple scanner, but as a highly intelligent robot assistant that can "see" the blurry photos and "read" the messy text simultaneously.

However, AI assistants are known to "hallucinate"—they might confidently make up facts that aren't there. To fix this, the authors built a three-step system with built-in safety checks.

Step 1: The "Strict Interviewer" (Structured Extraction)

Instead of just asking the AI to "read this," the system uses a special prompting method called CEVF (Constraint Example Verification Feedback).

The Analogy: Imagine you are interviewing a candidate. Instead of just asking, "Tell me about your experience," you give them a strict form to fill out (Constraint), show them one perfect example of how to fill it out (Example), and then immediately check their answers against a rulebook (Verification).
How it works:
1. Constraints: The AI is forced to only output specific fields (like "Step Number" or "Tool Used"). It can't wander off-topic.
2. One-Shot Example: The AI sees one perfect example of a filled-out form to copy the style.
3. Feedback Loop: If the AI makes a mistake (like writing a tool name that doesn't exist), the system catches it, says "Try again," and the AI corrects itself. This happens up to three times.
4. Confidence Score: The system gives every piece of extracted data a "confidence score." If the AI is unsure (low score), it highlights that specific spot in yellow for a human to double-check.

The Result: The AI stopped making up facts (hallucinations) and became very accurate, even with blurry or messy documents.

Step 2: The "Translator" (Semantic Alignment)

Once the AI extracts the data, it still faces the problem of different names. The AI might have pulled out "Drawing No." and "Part ID" as two different things, when they are actually the same.

The Analogy: Imagine a translator who knows that "Soda," "Pop," and "Coke" all mean the same drink in different regions.
How it works: The system uses a "semantic brain" (Sentence-BERT) to understand that these different words mean the same thing. It also checks a small, internal dictionary of factory terms (a domain knowledge base). If the AI is still unsure, it asks a human expert to make the final call, and then it remembers that rule for next time.

The Result: All the messy, different names get converted into one standard, clean list of data.

Step 3: The "Architect" (Standardized Reconstruction)

Now that the data is clean, the system needs to put it back into a perfect, professional-looking document.

The Analogy: Imagine you have a pile of loose bricks (the data). You want to build a perfect house (the standard document). The system acts as an architect that knows exactly where every brick goes, even if the pile of bricks is huge or the house needs to be taller than usual.
How it works: The system takes the clean data and fits it into a pre-approved template. If the list of steps is longer than the template, the system automatically adds more rows to the table, just like a smart spreadsheet. It even knows where to paste the pictures of the parts, resizing them so they fit perfectly.

The Result: The factory gets a brand new, perfectly formatted, professional document that looks like it was made by a top designer, ready to be used on the factory floor.

Why This Matters (The "Low Resource" Magic)

Usually, to teach an AI to do this, you need thousands of examples where humans have already highlighted every single word. But because these factory documents are secret, the authors couldn't do that.

The Innovation: This method works with very few examples (low resource). It doesn't need to be "re-trained" on massive datasets. It uses smart prompting and verification rules to learn on the fly.
The Safety Net: Because the data is secret, the whole process happens locally. The "Human-in-the-Loop" feature ensures that if the AI is confused, a human steps in, but only for the tricky parts, keeping the process fast.

Summary of Results

The authors tested this on real factory documents, including blurry photos and multi-page tables.

Accuracy: They got the right information about 89% of the time.
Mistakes: They reduced the "making things up" (hallucination) rate to just 6.5%.
Speed: It processed complex documents in about a minute, which is much faster than doing it manually.

In short, this paper presents a smart, self-checking robot that can organize a factory's messy, secret instruction manuals into clean, standard digital files without needing a massive library of pre-labeled data.

Technical Summary: Structured Extraction and Standardized Reconstruction of Machining Process Documents Based on Vision Language Model under Low Resource Constraints

1. Problem Statement

Machining process documents are critical assets for knowledge accumulation in intelligent manufacturing, containing essential data on processing sequences, parameters, equipment, and tooling. However, their management faces three primary challenges:

High Heterogeneity: Documents exist in diverse formats (scanned copies, PDFs, Word files, on-site photos) with complex, inconsistent layouts and varying terminologies.
Semantic Inconsistency: Continuous product iteration and specification revisions lead to inconsistent field naming and semantic ambiguity, hindering standardized management.
Resource and Confidentiality Constraints: Strict enterprise data confidentiality prevents the acquisition of sufficient desensitized labeled data, rendering traditional data-driven methods (which require large-scale annotated datasets) ineffective. Furthermore, existing multimodal discriminative models often lack robustness against layout shifts and lack human-machine collaborative verification mechanisms.

2. Methodology

The authors propose an integrated processing framework based on Vision Language Models (VLMs) that operates under low-resource constraints without requiring model fine-tuning or large-scale labeled data. The framework consists of three sequentially coupled modules:

2.1 Structured Extraction via CEVF Prompting

To address cross-modal hallucination and extraction reliability, the authors introduce a Constraint Example Verification Feedback (CEVF) prompting mechanism. This iterative workflow includes:

Constraint Layer: Predefined JSON Schema constraints limit the model's output space to core fields (e.g., basic_info, process_steps, approval_info) with strict type and format definitions.
Example Layer: A "one-shot" prompting strategy provides a single typical document and its structured output to guide the model, balancing context efficiency with clear extraction paradigms.
Verification and Iterative Feedback: Outputs undergo multi-dimensional rule-based verification (completeness, range, format, logic). If verification fails, structured feedback instructions trigger re-extraction. This loop runs up to three times.
Confidence Evaluation: A dual-dimensional strategy calculates a comprehensive confidence score ( $S_f$ ) combining the VLM's internal token probability and external rule-based verification scores. An iterative decay coefficient ( $\gamma = 0.95$ ) is applied to lower the confidence of fields requiring multiple correction rounds, flagging them for manual review if they fall below adaptive thresholds.

2.2 Semantic Alignment and Field Standardization

To resolve inconsistent field naming (e.g., "part number" vs. "drawing number"), an adaptive alignment strategy is employed:

Semantic Embedding: Original extracted field names and enterprise standard fields are encoded using Sentence-BERT to generate 768-dimensional vectors. Cosine similarity quantifies semantic proximity.
Domain Knowledge Base Collaboration: For similarities below a threshold (0.85), a lightweight machining domain knowledge base is consulted to resolve ambiguities.
Incremental Learning: Manually verified mappings are incrementally updated to the knowledge base, reducing future manual intervention without offline retraining.

2.3 Standardized Document Reconstruction

The final module generates compliant process documents from structured data, formalized as a Constraint Satisfaction Problem (CSP):

Placeholder Coordinate Alignment: The method uses semantic placeholders and a Document Object Model (DOM)/Layout Constraint Graph (LCG) to map structured data to a standard template.
Dynamic Layout Adaptation: For variable-length process tables, a dynamic row expansion strategy automatically appends rows based on the total process quantity, inheriting template styles to ensure formatting consistency.
Image Restoration: While the VLM extracts text, process illustrations are restored by converting original files to images and embedding them into designated placeholder positions in the new document to maintain integrity.

3. Key Contributions

Integrated Framework: A closed-loop pipeline for structured extraction, semantic alignment, and standardized reconstruction that operates without large-scale labeled data or model fine-tuning, suitable for high-confidentiality scenarios.
CEVF Prompting Mechanism: A novel method incorporating constraints, one-shot examples, and iterative feedback to effectively suppress VLM hallucinations and improve extraction reliability.
Dual-Modal Alignment Reconstruction: A document generation method that fuses semantic embedding with domain knowledge and uses placeholder coordinate alignment to handle heterogeneous field naming and generate compliant layouts for complex scenarios (e.g., multi-page tables, blurred scans).

4. Experimental Results

Experiments were conducted on 20 desensitized real-world process cards (covering shaft and gear parts) with varying quality levels (clear to moderately degraded). The proposed method was compared against a baseline (PaddleOCR-VL 1.5) and ablation studies of the CEVF components.

Extraction Performance: The full CEVF framework achieved a Field Accuracy (FA) of 88.90% and reduced the Hallucination Rate (HR) to 6.50%, significantly outperforming the baseline (FA: 45.50%, HR: 38.06%) and the constraint-only variant.
Format Compliance: The Format Compliance Rate (FCR) reached 99.40%, demonstrating high adherence to industrial layout standards.
Semantic Alignment: The collaborative strategy reduced semantic shift rates from 10.6% (pure semantic model) to 0.9% with the dynamically updated knowledge base.
Efficiency: Processing time averaged 62.73 seconds for complex cases but dropped to ~25 seconds for high-definition, low-complexity documents, nearly three times faster than traditional methods.
Robustness: The system successfully handled skewed pages, smudges, and multi-page tables. A human-machine collaboration boundary was defined, where low-confidence fields are highlighted for guided review, while severely degraded documents trigger manual intervention.

5. Significance and Claims

The paper claims to provide a practical and feasible technical path for the digitization and standardization of machining process knowledge under resource-constrained and high-confidentiality environments. By leveraging the generalization capabilities of VLMs and reinforcing them with rule-based verification and domain knowledge, the proposed method:

Breaks the unidirectional limitation of traditional methods that rely solely on converting documents to structured data.
Reduces dependence on large-scale annotated data, making it viable for enterprises with strict data privacy requirements.
Enables the automatic generation of standardized process documents from heterogeneous sources, facilitating knowledge inheritance, process optimization, and quality control in intelligent manufacturing.

The authors conclude that while the method effectively addresses current bottlenecks, future work will focus on migrating the framework to other industrial document types and introducing incremental learning for dynamic knowledge base evolution.