💻 computer science

Multi-Modal LLM based Image Captioning in ICT: Bridging the Gap Between General and Industry Domain

This paper proposes a multi-stage progressive training strategy to develop a 7B-parameter domain-specific image captioning model for the ICT industry, which leverages synthesized and expert-annotated data to significantly outperform larger state-of-the-art models in extracting logical text from technical images.

Original authors: Lianying Chao, Kai Zhang, Haoran Cai, Sijie Wu, Xubin Li, Xin Chen

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lianying Chao, Kai Zhang, Haoran Cai, Sijie Wu, Xubin Li, Xin Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but inexperienced student (an Artificial Intelligence) how to read complex technical blueprints used by engineers in the telecommunications industry. These blueprints aren't just pictures; they are flowcharts and signal diagrams full of specific logic that general AI models often misunderstand.

This paper describes a special training program designed to turn a standard AI into an expert "blueprint reader" for the tech industry. Here is how they did it, using simple analogies:

The Problem: The "Generalist" vs. The "Specialist"

Think of current powerful AI models (like the ones you might chat with online) as general librarians. They have read millions of books and can describe a picture of a cat or a sunset perfectly. However, if you hand them a complex engineering flowchart, they might guess the connections wrong or miss the specific meaning of a symbol. They lack the "industry vocabulary."

The authors wanted to build a specialist librarian who could look at these technical diagrams and explain them with perfect accuracy, even though this specialist has a smaller memory (fewer "parameters") than the giant general librarians.

The Solution: A Three-Stage Training Camp

Instead of just feeding the AI a massive pile of random data, the authors built a three-step "training camp" to teach the AI how to read these specific diagrams.

Stage 1: The "Practice Sheets" (Synthetic Data)

The Analogy: Imagine giving the student thousands of practice worksheets where the answers are already written out perfectly.
What they did: They used a computer tool called "Mermaid" to automatically draw thousands of fake flowcharts and signal diagrams. Then, they used another AI to write the "correct answers" (text descriptions) for these fake diagrams.
The Goal: This gave the AI a massive amount of practice to learn the structure of these diagrams without needing humans to draw every single one.

Stage 2: The "Apprenticeship" (Expert Annotations)

The Analogy: Now, the student is put in a real workshop with a master craftsman. The master points to a real diagram and says, "This is how we describe this part. Don't just say 'arrow,' say 'signal flow from Node A to Node B'."
What they did: Real human experts from the tech industry manually wrote descriptions for about 2,000 real diagrams. They taught the AI the specific "dialect" and logic used by professionals.
The Goal: To teach the AI to sound like an expert, not just a general observer.

Stage 3: The "Oral Exam" (Visual Question Answering)

The Analogy: The student is now tested. Instead of just describing the picture, the teacher asks specific questions: "If the signal stops at this node, where does it go next?" or "How many steps are in this process?"
What they did: They created a set of questions and answers based on the diagrams. The AI had to look at the image and answer the question correctly.
The Goal: To ensure the AI truly understands the logic and relationships inside the diagram, not just the words on the page.

The Results: Small but Mighty

The paper claims that their new model, called DICModel, is surprisingly effective:

Size: It is relatively small (7 billion "brain cells" or parameters).
Performance: It outperformed much larger, famous models (some with 32 billion parameters) and even closed-source giants like Google's Gemini and Claude.
The Score: In tests, it was about 57% better at describing the text in the images than the next best 7-billion model, and it was more accurate at answering technical questions than the massive 32-billion models.

Why This Matters (According to the Paper)

The authors explain that this model acts like a translator. It can take a complex image (a flowchart) and turn it into clear, logical text. This is crucial because:

Search Engines: It helps build databases where you can search for images using text, or find text using images.
AI Assistants: It allows future AI agents to "read" technical manuals and diagrams to help engineers or customers solve problems.

The Limitations

The authors are honest about what their model can't do yet:

If a flowchart is extremely messy or has a very complex "jump" to a different part of the page, the model might get confused about where the line goes.
It currently only handles images (diagrams), not audio or video files.

In short, this paper presents a clever "training recipe" that allows a small, efficient AI to become a master of reading technical engineering diagrams, beating much larger competitors in the process.

Technical Summary: Multi-Modal LLM based Image Captioning in ICT

Problem Statement
In the Information and Communications Technology (ICT) industry, training domain-specific Large Language Models (LLMs) or constructing Retrieval-Augmented Generation (RAG) systems requires vast amounts of high-value domain knowledge. While this knowledge exists in both textual and image modalities, traditional document parsing methods (e.g., Miner-U, GOT) are limited to extracting text, thereby discarding critical visual information such as flowcharts and signaling diagrams. Conversely, while general Multi-Modal Large Language Models (MLLMs) possess image understanding capabilities, they lack the specific domain knowledge required for the ICT sector. When applied directly to ICT documents, general MLLMs often produce errors in logical judgment, fail to identify specific information clearly, or misinterpret signal flow directions. The core challenge is the absence of a model capable of accurately converting complex ICT visual information into precise, logical textual representations.

Methodology
To address these limitations, the authors propose a Domain-specific Image Captioning Model (DICModel) based on a three-stage progressive training strategy. The model utilizes Qwen2.5-VL 7B as its base architecture.

Stage 1: Pre-SFT with Synthetic Data
- Objective: To teach the model the standard paradigms of image parsing and structural information.
- Data Generation: Approximately 7,233 image-text pairs are synthesized. The Mermaid tool generates flowcharts and signaling diagrams from structured code. An LLM (Grok-3) then converts the Mermaid code into standardized, expert-approved annotation text.
- Outcome: The model learns to parse images into structured text formats.
Stage 2: Post-SFT with Expert-Annotated Data
- Objective: To inject specific ICT domain knowledge and refine parsing precision.
- Data Generation: ICT domain experts manually annotate approximately 2,274 high-quality image-text pairs from real-world professional documents (including 1,100 flowcharts and 1,174 signal diagrams). Experts use fixed templates to ensure the text aligns with industry standards.
- Outcome: The model adapts to the specific reasoning patterns and terminology of the ICT domain.
Stage 3: Instruction-SFT with VQA Data
- Objective: To enhance the model's ability to follow user instructions and understand fine-grained details, local components, and global layouts.
- Data Generation: Approximately 1,573 Visual Question Answering (VQA) pairs are created. Questions are generated by both MLLMs (Gemini 2.0) and domain experts based on ICT images.
- Outcome: The model improves its instruction compliance and deep comprehension capabilities.

Key Contributions

Progressive Training Strategy: The paper introduces a three-stage training pipeline (Synthetic $\rightarrow$ Expert-Annotated $\rightarrow$ Instruction-based VQA) that effectively bridges the gap between general MLLM capabilities and specific ICT domain requirements.
Data Synthesis Pipeline: A novel method is developed to synthesize high-quality training data by combining the Mermaid diagramming tool with LLM-based text rewriting, addressing the scarcity of annotated domain data.
Comprehensive Evaluation System: The authors constructed a rigorous evaluation system involving ICT domain experts. This includes manual annotation of 100 critical images for parsing quality and the creation of 300 objective questions (200 single-choice, 100 multiple-choice) to test image understanding.

Experimental Results
The DICModel (7B parameters) was evaluated against State-of-the-Art (SOTA) models, including closed-source APIs (Gemini 2.0, Claude-3.5) and open-source models of varying sizes (7B to 32B).

Visual Question Answering (VQA): On expert-constructed objective questions, DICModel achieved an average accuracy ( $Prec_a$ ) of 72.3%. This outperforms the 32B parameter models (Qwen2.5-VL 32B at 69.0%, Intern2.5-VL 32B at 71.3%) and closed-source models (Gemini 2.0 at 64.3%). Compared to the second-best 7B model (Qwen2.5-VL 7B), DICModel improved accuracy by 10.3%.
Image Parsing (Captioning): In terms of text similarity metrics (BLEU, CIDEr, METEOR), DICModel significantly outperformed other 7B models. Specifically, it improved the BLEU score by approximately 56.8% compared to Qwen2.5-VL 7B and by 20.8% compared to Qwen2.5-VL 32B.
Ablation Study: The study confirmed that all three training stages are necessary. The first two stages were critical for maximizing parsing accuracy (BLEU), while the third stage was essential for maximizing question-answering accuracy.

Significance and Claims
The paper claims that DICModel demonstrates that a smaller, domain-specific model (7B parameters) can outperform significantly larger general-purpose models (32B parameters) in specialized ICT tasks. By efficiently extracting logical text from complex images like flowcharts and signaling diagrams, the model enables the creation of high-value multimodal vector databases and supports the development of domain-specific LLMs. The authors position this work as a foundational step toward more advanced AI applications in the ICT sector, such as intelligent customer service, multi-modal RAG systems, and automated data asset management.

Limitations
The authors acknowledge that the model still struggles with highly complex diagrams, specifically in identifying target steps when merge lines are involved in flowcharts or determining signal endpoints in diagrams where signals span multiple nodes. Additionally, the current scope is limited to static images, with audio and video data presenting future challenges for unified cross-modal representation.