Sketch2Simulation: Automating Flowsheet Generation via Multi Agent Large Language Models

This paper presents "Sketch2Simulation," an end-to-end multi-agent large language model framework that automates the conversion of process sketches into executable Aspen HYSYS simulation models by integrating diagram parsing, model synthesis, and multi-level validation, achieving high structural and stream consistency across diverse chemical engineering case studies.

The Gist

Imagine you are an architect who has just finished a beautiful, hand-drawn sketch of a new factory. You have drawn the pipes, the tanks, the pumps, and the arrows showing where the oil and water flow. It's a great picture, but a computer simulation program (like Aspen HYSYS) can't "read" a drawing. It needs a strict, digital blueprint with exact instructions: "Tank A connects to Pipe B," "Pump C needs 500 gallons per minute," and "The temperature must be 200 degrees."

Usually, a human engineer has to sit down and manually translate that sketch into the computer's language. This is slow, boring, and prone to mistakes.

"Sketch2Simulation" is a new system that acts like a team of super-smart AI robots to do this translation instantly. Instead of one giant robot trying to do everything (which often gets confused), the authors built a Multi-Agent System—a team of specialized robots, each with a specific job, working together to turn your sketch into a working computer model.

Here is how the team works, using a simple analogy:

The Team of AI Robots

Imagine a construction site where a blueprint needs to be turned into a real building. You wouldn't ask one person to draw, calculate, build, and inspect all at once. You'd hire a crew. This system does the same:

1. The Detective (Diagram Parsing & Interpretation):

   • Job: This robot looks at your messy hand-drawn sketch. It uses "eyes" (computer vision) to find the pumps, tanks, and lines.
   • The Trick: It doesn't just see shapes; it understands context. If it sees three pipes merging into one, it knows, "Ah, this is a mixer," even if the label is missing. It creates a clean, digital list of "What is here?" and "How are they connected?"
   • Analogy: Think of this as a translator who looks at a foreign map and writes down a clear list of landmarks and roads, ignoring the scribbles and artistic flourishes.

2. The Architect (Simulation Model Synthesis):

   • Job: Now that we have the list, this robot writes the actual code for the computer program.
   • The Trick: It breaks this down further:
     • The Librarian: Checks the names of chemicals to make sure the computer recognizes them (e.g., ensuring "Benzene" is spelled exactly right in the database).
     • The Builder: Places the virtual tanks and pipes into the software.
     • The Plumber: Connects the pipes to the tanks, making sure the flow direction is correct.
   • Analogy: This is like a contractor who takes the list of materials and starts assembling the actual house, ensuring the walls go in the right place and the plumbing connects to the pipes.

3. The Inspector (Multi-Level Validation):

   • Job: This robot checks the work at every step.
   • The Trick: It asks, "Does this description match the drawing?" and "Does this code actually run without crashing?" If the computer program throws an error, this robot tries to fix the code automatically before trying again.
   • Analogy: This is the building inspector who walks through the house to make sure the roof doesn't leak and the electricity works before you move in.

Why is this a big deal?

• It's a Team, Not a Solo Act: Previous AI attempts tried to use one giant brain to do everything. If that brain got tired or confused, the whole project failed. By splitting the work into a team of specialists, if one robot makes a mistake, the others can catch it.
• It Handles the "Messy" Stuff: Real engineering sketches are often messy. They have missing labels, crossed lines, or implied connections. This system is smart enough to guess what was meant (like realizing two pipes must meet even if the line is faint) and fix it for the computer.
• It Works on Real Industrial Problems: The authors tested this on four different scenarios, from a simple water cleaning process to a massive, complex oil refinery with many recycling loops.
  • The Result: For the simple cases, it was 100% perfect. For the complex industrial cases, it was still over 95% accurate, and the computer models actually ran without crashing.

The Bottom Line

Think of Sketch2Simulation as a magic bridge. On one side, you have the human engineer's creative, hand-drawn ideas. On the other side, you have the rigid, mathematical world of computer simulations.

In the past, you needed a human translator to cross that bridge. Now, this AI team can cross it for you in seconds, turning a sketch into a working, testable factory model. It doesn't replace the engineer; it frees them from the boring typing and allows them to focus on designing better processes.

The Catch? It's not perfect yet. If the drawing is too messy or the factory design is incredibly complex with many loops, the AI might get a little confused on the connections. But it's a massive leap forward in making engineering faster and more automated.

Technical Summary

1. Problem Statement

The conversion of process engineering sketches (diagrams) into executable simulation models remains a significant bottleneck in Process Systems Engineering (PSE).

• The Gap: While recent advances in Computer Vision (CV) have improved the extraction of symbols and topology from diagrams, and Large Language Models (LLMs) have advanced text-to-simulation workflows, these two fields remain disconnected. CV methods typically stop at generating graphs or semantic representations, while LLM-based simulation tools assume structured text inputs rather than raw visual artifacts.
• The Challenge: Converting a diagram to a simulator-ready model (e.g., Aspen HYSYS) requires more than just recognition; it demands engineering interpretation to resolve implicit connections, infer missing parameters, and adhere to strict simulator-specific constraints (object hierarchies, initialization rules, and thermodynamic consistency).
• Current Limitations: Existing approaches often fail to bridge the "symbol-to-parameter" gap, leading to models that are topologically correct but not executable, or requiring substantial manual refinement.

2. Methodology: The Multi-Agent System

The authors propose an end-to-end, multi-agent Large Language Model (LLM) system that transforms raw process diagrams directly into executable Aspen HYSYS flowsheets. The system is decomposed into three coordinated layers to ensure modularity, error localization, and robustness.

A. System Architecture

The workflow is orchestrated by a central controller (LangGraph) managing a deterministic sequence of state transformations across specialized agents.

Layer 1: Diagram Parsing and Interpretation

• Goal: Convert unstructured visual input into a formalized Intermediate Representation (IR) as a directed graph $G=(V, E)$.
• Agents:
  • Descriptor Agent (A1): Uses a multimodal LLM (Gemini 3 Flash) to generate a detailed textual description of the process, enforcing a left-to-right traversal and self-evaluation for visual evidence.
  • Extractor Agent (A2): A multimodal LLM that constructs the JSON-based IR. It performs a two-pass extraction: first identifying unit operations (nodes), then extracting material streams (edges) and classifying them (feed, product, recycle).
  • Normalization Agent (A3): A rule-based agent (not LLM) that enforces simulator constraints. It resolves implicit junctions (e.g., inserting mixers where multiple streams converge on a single inlet) and consolidates complex units (e.g., merging a column, condenser, and reboiler into a single simulator template).

Layer 2: Simulation Model Synthesis

• Goal: Translate the normalized IR into executable Python code interacting with the Aspen HYSYS COM (Component Object Model) interface.
• Agents:
  • Basis Agent (B1): Defines the simulation case (fluid package, components). It uses Retrieval-Augmented Generation (RAG) to map extracted material names to exact HYSYS database entries.
  • Instantiation Agent (B2): Generates code to create unit operation objects based on the IR nodes.
  • Configuration Agent (B3): Generates code to connect streams to unit ports based on the IR edges.
  • Execution Agent (B4): A hybrid agent that runs the generated script. If execution fails, an LLM component analyzes logs and applies targeted code fixes before re-running.

Layer 3: Multi-Level Validation

• Goal: Ensure structural and semantic consistency at every stage.
• Mechanisms:
  • Description Validation (A1.1): An independent LLM verifies the Descriptor Agent's output against the original image.
  • Schema Constraints: Strict JSON schemas and prompt engineering prevent hallucinations during extraction.
  • Execution Validation: Runtime diagnostics confirm the model runs successfully in HYSYS.

B. Model Selection & Deployment

• Interpretation (Cloud): Uses Gemini 3 Flash for its superior multimodal reasoning and OCR capabilities on complex diagrams.
• Synthesis (Local): Uses Qwen2.5-Coder-7B and Qwen3-Coder-30B for code generation to ensure data privacy and security (proprietary process data remains local).
• Hybrid Approach: Combines the visual reasoning power of cloud models with the security and control of local code-synthesis models.

3. Key Contributions

1. End-to-End Automation: The first framework to directly convert raw visual engineering diagrams into executable Aspen HYSYS simulation models, bridging the gap between diagram understanding and simulator instantiation.
2. Structured Intermediate Representation (IR): Introduction of a graph-based IR that acts as a translator between visual semantics and simulator object models, allowing for explicit error localization.
3. Multi-Agent Architecture: A novel decomposition of the task into specialized agents (visual, structural, coding, validation) that reduces hallucination risks and improves robustness compared to monolithic LLM approaches.
4. Industrial Relevance: Validation against a commercial-grade simulator (Aspen HYSYS) rather than just academic or open-source prototypes, addressing real-world constraints like object hierarchies and initialization rules.

4. Results and Evaluation

The system was evaluated on four case studies of increasing complexity:

1. Desalting Process (Simple): Achieved 100% structural fidelity (F1 = 1.00) across all metrics (Units, Streams, Connections, Materials).
2. Merox Sweetening (Moderate): Achieved 100% fidelity, successfully inferring implicit connections (e.g., between prewash and reactor) not explicitly drawn.
3. Atmospheric Distillation (Complex): High performance (Connection Consistency = 0.93). Minor deviations were attributed to simulator interface limitations regarding side-draw connections rather than diagram misinterpretation.
4. Aromatic Production (Industrial Scale): High performance (Stream Consistency = 0.96, Connection Consistency = 0.98). The system handled dense recycle loops and complex interconnections, with only minor errors in highly cluttered regions.

Ablation Studies:

• Removing the Descriptor Agent or Normalization Agent significantly degraded performance, especially in complex cases, proving the necessity of visual grounding and structural refinement.
• Disabling RAG caused total failure in cases requiring mixture components, highlighting the critical need for accurate component mapping.
• Model Benchmarking: Gemini 3 Flash outperformed open-weight alternatives (Qwen 3-VL, Qwen 3.5) in spatial reasoning and connectivity reconstruction, attributed to its "agentic vision" and early-fusion architecture.

5. Significance and Future Work

• Significance: This work demonstrates that AI can move beyond "reading" diagrams to "building" functional engineering models. It reduces the manual effort required for digital twin creation and process automation, making high-fidelity simulation more accessible.
• Limitations: Performance degrades with extremely dense diagrams, implicit semantics, and specific simulator interface constraints (e.g., complex distillation column templates).
• Future Directions:
  • Handling noisier, legacy, or scanned industrial documents.
  • Developing simulator-agnostic intermediate abstractions to support multiple platforms (e.g., Aspen Plus, gPROMS).
  • Enhancing self-correction loops with confidence-aware extraction and engineering rule retrieval.

In conclusion, SKETCH2SIMULATION establishes a viable pathway for automating the transition from visual engineering intent to executable process models, marking a significant step toward fully autonomous process systems engineering.