From P&ID Drawings to Process Graphs: A Multimodal Language Model Approach

Imagine you are walking through a massive, ancient library where the books aren't made of paper, but of giant, tangled webs of glowing wires and pipes. These aren't just decorations; they are the blueprints for how a factory breathes, moves, and thinks. In the world of engineering, these blueprints are called P&IDs (Piping and Instrumentation Diagrams). They are the secret language that tells a computer how to run a chemical plant, a power station, or a water treatment facility. For decades, these blueprints have been stuck on old paper or as blurry digital pictures. They are like a locked treasure chest: full of gold (valuable data), but the key to opening them has been missing. Engineers have to stare at these messy drawings and manually type out every single connection, a slow and error-prone task that feels like trying to copy a novel by hand while someone keeps changing the font.

Recently, a new kind of "super-reader" has arrived: the Multimodal Large Language Model (MLLM). Think of these as AI detectives that can look at a picture and read the text inside it at the same time, understanding not just what the symbols look like, but what they mean. The big question scientists have been asking is: Can we just hand these AI detectives a messy blueprint and ask them to instantly turn it into a perfect, digital map? Or is the job too messy for a single, quick glance? This is the puzzle a team of researchers from Imperial College London decided to solve. They wanted to know if we could teach these AI detectives to stop trying to do everything at once and instead break the job into smaller, smarter steps, using their knowledge of how factories actually work to guide them.

The researchers set out to test two different ways of asking an AI to digitize these complex factory blueprints. Their first idea was the "Speed Run" approach: they fed the entire blueprint image (cut into slices so the AI wouldn't get overwhelmed) and a brief description of the process into the AI, asking it to spit out a perfect digital map in one go. They called this the "End-to-End" method. It's like asking a student to read a whole textbook, understand the plot, and then write a summary of the characters and their relationships in a single sitting without taking any notes.

The second idea was the "Team Huddle" approach. Here, they split the job into two distinct steps. First, one AI agent acted as a pure "Scanner," looking at the blueprint slices and simply listing every single piece of equipment and instrument it saw, ignoring how they were connected. It was like a librarian just listing every book title on a shelf without worrying about the story. Then, a second AI agent took that list of names and, crucially, a set of "rules of the factory" (like "pumps need a pipe coming in and a pipe going out"). This second agent used its brain to figure out how those pieces should connect based on engineering logic, rather than trying to trace the messy lines in the original picture. This was the "Decomposed" method.

When they tested these methods on two real-world factory diagrams—one a Nitrogen Pressure Swing Adsorption plant and the other a Wet Flue Gas Desulfurization plant—the results were clear. The "Speed Run" approach struggled. Even with a little help from a process description, it got the connections wrong about 40% of the time. It was like the student guessing the plot twists; it could name the characters (the equipment) correctly, but it kept mixing up who was talking to whom. The "Team Huddle" approach, however, was a game-changer. By separating the task of "seeing" from the task of "thinking," and by giving the second AI a cheat sheet of engineering rules, the accuracy skyrocketed.

For the Nitrogen plant, the "Team Huddle" method achieved a total accuracy of 88.78%, compared to just 68.56% for the single-step method. It got every single piece of equipment right (100% node accuracy) and correctly figured out the connections 80.58% of the time. For the more complex Slime and Off-gas plant, the improvement was even more dramatic. The single-step method got the connections right only about 59% of the time, while the "Team Huddle" method jumped to 74.12% accuracy for connections and 85.21% overall. The researchers measured this using a "Simple Distance" score, which counts how many mistakes (missing pieces, extra pieces, wrong connections) the AI made. The "Team Huddle" method had far fewer mistakes, with a score of 24.0 for the Nitrogen plant compared to 45.6 for the single-step method.

The paper suggests that the secret sauce isn't just having a smarter AI, but having a smarter workflow. It turns out that asking an AI to "see" and "reason" at the exact same moment is too much cognitive load, especially when the drawings are messy and the lines are broken. By letting one part of the system focus purely on reading the text and symbols, and letting another part focus purely on applying engineering logic to figure out the connections, the system becomes much more reliable. The study shows that while a single, massive prompt can work, it performs significantly worse than breaking the task down, particularly for complex diagrams where the AI tends to get lost in the details without the structured guidance of a decomposed workflow.

While the results are promising, the authors are careful to note that this is a measured improvement, not a magic wand that solves everything instantly. They found that simply adding a process description helped a little bit, but it was the combination of splitting the task and adding specific chemical engineering rules that made the real difference. They also point out a practical hurdle: the AI models they used are proprietary (owned by big companies), which raises privacy questions for factories that don't want to share their secret blueprints with external servers. However, they suggest that the same "Team Huddle" workflow could work with open-source models in the future.

In the end, this research suggests that the future of digitizing our industrial world isn't about building a single super-brain that does everything, but about building a team of specialized experts who talk to each other. By teaching AI to separate the act of looking at a drawing from the act of understanding the engineering logic behind it, we can turn those dusty, tangled blueprints into clean, digital maps that computers can actually use to build smarter, safer factories. It's a reminder that sometimes, the best way to solve a giant, messy problem is to break it into smaller, manageable pieces and let the experts do what they do best.