Procedural Knowledge Extraction from Industrial Troubleshooting Guides Using Vision Language Models
This paper evaluates the effectiveness of Vision Language Models in automating the extraction of structured diagnostic knowledge from industrial troubleshooting guides, comparing standard prompting against layout-aware strategies to identify trade-offs between spatial sensitivity and semantic robustness.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the noisy, grease-stained world of industrial maintenance, keeping complex machinery running is a race against time. When a massive pump fails or a conveyor belt jams, a technician cannot afford to wait for a slow search through a library of manuals. They need immediate answers. For decades, this knowledge has been locked inside thick binders filled with troubleshooting guides. These documents are not just lists of text; they are intricate maps drawn with arrows, boxes, and diamonds that guide a worker's eye from a problem to a solution. To a human, these visual clues make sense instantly. To a computer, however, they are a chaotic jumble. The challenge for modern engineers is to teach machines to read these visual maps, turning static diagrams into digital instructions that can power smart tools to help workers on the factory floor. This is the frontier where artificial intelligence meets the physical reality of broken machines.
A team of researchers at the University of Groningen set out to test whether the newest generation of artificial intelligence, known as vision-language models, could perform this difficult task. These models are advanced computer programs trained to understand both images and words simultaneously, much like a person who can look at a picture and read the caption at the same time. The researchers wanted to see if these models could look at a page from an industrial troubleshooting guide and automatically pull out the logical steps hidden inside. They took twelve real-world guides from a Dutch manufacturer, which contained roughly thirty to one hundred distinct steps and connections per document, and fed them into two different AI systems. The goal was to see if the machines could identify the conditions to check, the actions to take, and the decision points that branch the path forward, all while reconstructing the correct order of events.
The results were a sobering reminder of how far the technology still has to go. While the AI models could occasionally spot individual words or simple shapes, they largely failed to understand the story the diagram was telling. When asked to extract the specific steps and the connections between them, the models stumbled badly. They managed to identify only a small fraction of the correct steps, and when it came to linking those steps together in the right order, they performed almost no better than random guessing. In many cases, the machines produced outputs that were so broken or incomplete that they could not be used to rebuild the original logic of the guide. The researchers found that the models struggled most with the spatial relationships—the way arrows connect a diamond-shaped decision to a rectangular action box. Without understanding these visual connections, the AI could not reconstruct the procedural flow, rendering the extracted information useless for building a reliable digital assistant.
The study also revealed that the two different AI models behaved in surprisingly different ways, suggesting that there is no single "best" solution yet. One model, while occasionally capable of finding a high number of correct steps, had a dangerous tendency to get stuck in a loop. Once it started making mistakes, it would repeat the same errors over and over, generating hundreds of nearly identical, incorrect steps until it ran out of space to write. It was as if the machine had lost its place and was frantically rewriting the same sentence with different numbers. The other model was more stable and did not get stuck in these loops, but it was far more conservative, often missing most of the steps entirely and failing to find any connections between them. This difference showed that the internal design of the AI matters greatly; one architecture was prone to wild hallucinations, while the other was simply too cautious to be useful.
The researchers tested whether giving the AI more detailed instructions about how to read the diagrams would help. They provided extra clues explaining that a diamond shape meant a yes-or-no question and that arrows showed the direction of the process. For one of the models, this extra guidance helped it find more connections between steps, but it also made it less accurate at identifying the steps themselves. For the other model, the extra instructions made things worse, causing it to perform even more poorly on both counts. This suggests that simply telling the AI more about the rules of the game is not enough to fix its fundamental inability to see the whole picture. The technology is currently not ready to work alone in a safety-critical environment where a mistake could lead to equipment damage or injury.
Despite these limitations, the study does not suggest that the technology is useless, only that it is not yet ready for full automation. The researchers propose that these tools could still play a valuable role if they are used to assist human experts rather than replace them. In a human-in-the-loop system, the AI could act as a first draft, pre-filling a digital form with its best guesses about the steps and connections. A human technician would then review and correct the work, which would be much faster than starting from scratch. The study concludes that while the dream of a machine that can instantly read and understand any industrial diagram is still far off, the path forward lies in collaboration. By combining the speed of the machine with the judgment of the human, factories can begin to unlock the knowledge trapped in their paper guides, paving the way for smarter, safer maintenance in the future.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.