DUCTILE: Agentic LLM Orchestration of Engineering Analysis in Product Development Practice

This paper introduces DUCTILE, an agentic LLM orchestration framework that separates adaptive decision-making from deterministic tool execution to automate engineering analysis in product development, successfully handling input deviations in an aerospace case study while highlighting the emerging tension between task automation and the creation of exhausting supervisory roles.

The Gist

Here is an explanation of the paper "DUCTILE: Agentic LLM Orchestration of Engineering Analysis in Product Development Practice," translated into simple, everyday language with creative analogies.

The Big Problem: The "Brittle" Robot

Imagine you have a highly efficient robot chef in a restaurant. This robot is amazing at following a recipe to make a perfect lasagna. But here's the catch: the robot is brittle.

If the recipe says "use 2 cups of flour," the robot works perfectly. But if the supplier suddenly sends flour in a different bag, or if the recipe changes to "use grams instead of cups," or if the chef decides to swap the tomato sauce for a different brand, the robot doesn't just adapt. It breaks. It stops working, throws an error, and the kitchen grinds to a halt.

In the world of engineering (like building jet engines), this is exactly what happens. Engineers use scripts and software to automate calculations. But the real world is messy. Data formats change, units switch from metric to imperial, or a supplier sends a file in a weird format. Traditional automation breaks every time this happens, forcing engineers to stop their important work and spend hours fixing the "robot" (data wrangling).

The Solution: The "DUCTILE" Agent

The authors of this paper propose a new kind of helper called DUCTILE. Think of DUCTILE not as a robot that follows a rigid script, but as a smart, adaptable project manager.

The name stands for Delegated, User-supervised Coordination of Tool- and document-Integrated LLM-Enabled. That's a mouthful, so let's break it down with a metaphor:

Imagine a construction site.

• The Tools (The Bricks & Cement): These are the heavy, verified engineering software tools. They are like the bricklayers. They are strong, precise, and never make mistakes if you give them the right bricks. They are deterministic (they do exactly what they are told).
• The Agent (The Foreman): This is the LLM (Large Language Model). The Foreman doesn't lay the bricks. Instead, the Foreman reads the blueprints, talks to the suppliers, and tells the bricklayers what to do.
• The Engineer (The Site Manager): This is the human. They hire the Foreman, check the Foreman's plan, and give the final sign-off.

How DUCTILE works:

1. The Foreman reads the instructions: Instead of hard-coding rules like "If file is JSON, do X," the Foreman reads the company's design documents and the new data file.
2. The Foreman adapts: If the supplier sends a YAML file instead of JSON, the Foreman says, "Ah, this is different. I'll write a quick translator script to fix it before handing it to the bricklayers."
3. The Foreman supervises: The human Site Manager watches the Foreman's plan. If the plan looks weird, the Manager stops it. If it looks good, the Manager says "Go."
4. The Bricklayers do the work: The actual calculation happens in the trusted, verified software. The Foreman just orchestrates the flow.

The Real-World Test: The Jet Engine

The researchers tested this at GKN Aerospace, a company that makes parts for jet engines. They gave the agent a task: process new load data for a turbine part.

The Twist: The new data had four surprises that would have crashed a traditional robot:

1. File Format: It was in YAML instead of the expected JSON.
2. Units: It used Imperial units (pounds) instead of Metric (Newtons).
3. Naming: The parts were named "Left/Right" instead of the usual "Port/Starboard."
4. Method Change: The supplier added a hidden math rule (multiply one force by 1.04) that wasn't in the old data.

The Result:

• Traditional Robot: Would have crashed immediately.
• DUCTILE Agent: Read the documents, spotted all four issues, wrote the necessary code to fix them, and ran the calculation.
• The Humans: Two different engineers tried it. One let the agent do everything and checked the result at the end. The other asked the agent to do it step-by-step, checking after every move. Both got the exact same correct result.

Why This Matters (The "So What?")

This isn't just about making robots smarter; it's about changing how engineers work.

• The "Black Box" Fear: People worry that AI is a magic box where you put data in and get answers out, but you don't know how it happened. DUCTILE solves this. Because the agent writes the code and shows its "thinking" (like a scratchpad), the human can see exactly what the agent did. It's transparent.
• The "Human in the Loop": The paper emphasizes that the AI is not replacing the engineer's judgment. The AI is the "dexterous" worker (good at moving files, converting units, writing code), but the human is the "judgment" expert (knowing if the result makes physical sense).
• The Risk of Exhaustion: The authors warn of a potential downside. If the AI does all the "fun" problem-solving, the human might just become a bored supervisor, staring at screens, which can be exhausting. The goal is to use AI to remove the boring data cleanup so engineers can focus on the interesting design decisions.

In a Nutshell

DUCTILE is a new way to use AI in engineering. Instead of building rigid, breakable robots that fail when things change, we are building adaptable agents that act as a bridge between messy real-world data and precise engineering tools.

• Old Way: "If the file changes, the system breaks. Fix the code."
• DUCTILE Way: "The file changed? The agent reads the new format, figures out the fix, and gets the job done. The human just checks the work."

It turns engineering automation from a fragile glass house into a ductile (flexible) structure that can bend without breaking.

Technical Summary

Here is a detailed technical summary of the paper "DUCTILE: Agentic LLM Orchestration of Engineering Analysis in Product Development Practice."

1. Problem Statement

Engineering analysis in product development (particularly in aerospace) relies on a complex ecosystem of tools, data formats, and documented processes. Current automation strategies suffer from systemic brittleness:

• Rigid Interfaces: Traditional automation (scripts, KBE, PIDO platforms) relies on deterministic, pre-defined interfaces.
• Fragility to Change: When inputs change (e.g., file formats, unit systems, naming conventions, or methodology updates), these rigid pipelines break, requiring manual intervention or code rewriting.
• Knowledge Silos: To handle these breaks, organizations rely on expert engineers to manually adapt workflows, which concentrates critical knowledge in individuals and creates organizational vulnerability.
• The Gap: Existing AI approaches (like Generative Models) focus on generating design candidates or predicting performance, but they do not solve the "orchestration problem" of connecting heterogeneous, verified tools while following strict safety-critical processes.

2. Methodology: The DUCTILE Approach

The paper proposes DUCTILE (Delegated, User-supervised Coordination of Tool- and document-Integrated LLM-Enabled), an agentic orchestration framework. Its core innovation is the separation of adaptive orchestration from deterministic execution.

Core Architecture

• The Agent (Adaptive Layer): An LLM agent acts as an intermediary. It interprets natural language design practices, inspects input data, and adapts the processing path. It uses Thinking Mode (self-correction/reasoning) and Tool Calling to plan workflows.
• The Tools (Deterministic Layer): All actual engineering calculations are performed by verified, domain-certified tools (e.g., Python scripts, commercial solvers like ANSYS). The LLM does not perform the physics calculations; it only generates the code to call these tools.
• Human-in-the-Loop: Engineers supervise the agent, approve plans, and validate final outputs, retaining accountability for engineering judgment.

Technical Implementation

• Framework: Built using open-source agentic frameworks (e.g., Claude Code) to ensure inspectability and reproducibility.
• Context Window: The agent relies on a context window containing the task description, design practice documents (via MCP servers), tool documentation, and input data.
• Evaluation Metric (Pass-$k$): To address the stochastic nature of LLMs, the authors adopt a Pass-$k$ metric. An agent must successfully complete a task $k$ times in independent runs to be considered reliable. This is analogous to statistical basis levels (A-Basis, B-Basis) in aerospace material qualification.
• Requirements: The approach adheres to 12 specific requirements (R1–R12) covering inspectability, reproducibility, traceability, data governance, and human oversight.

3. Key Contributions

1. The DUCTILE Framework: A novel methodology for developing and executing LLM-based agentic automation that separates reasoning (LLM) from computation (verified tools).
2. Industrial Validation: A real-world case study at GKN Aerospace (a jet engine manufacturer) involving structural analysis of a Turbine Rear Structure (TRS).
3. Handling Variability: Demonstration that the agent can handle four distinct input deviations that would break traditional scripts:
   • File format change (JSON $\to$ YAML).
   • Unit system mismatch (SI $\to$ Imperial).
   • Naming convention change (Port/Starboard $\to$ Left/Right).
   • Methodology update (Application of a 1.04 correction factor).
4. Evaluation Protocol: A rigorous evaluation strategy using the Pass-$k$ metric and "LLM-as-a-judge" combined with human expert review to ensure reliability.

4. Results

The approach was tested on an industrial load processing task with two engineers using different supervision styles:

• Engineer 1 (Delegation): Delegated the full task to the agent. The agent identified all four deviations, formulated a 7-step plan, wrote a processing script, and executed it. The engineer reviewed the final output and approved it.
• Engineer 2 (Incremental Discovery): Interacted step-by-step (read inputs $\to$ process one case $\to$ verify). The agent adapted to this supervision style, and the final output was identical to Engineer 1's.
• Reliability: The agent was run 10 independent times on the same scenario with no changes to prompts or tools.
  • 100% Success Rate: All 10 runs passed both quantitative checks (load summary values) and qualitative checks (LLM-as-a-judge verification of the script logic).
  • Consistency: The agent correctly handled all four deviations in every run without human code intervention.

5. Significance and Implications

• Resolving Brittleness: DUCTILE absorbs input variability at the orchestration layer, preventing the "knock-on effects" that break downstream design activities in traditional pipelines.
• Preserving Rigor: By keeping calculations in deterministic, verified tools, the approach maintains the traceability and certification compliance required in safety-critical industries (aerospace/automotive).
• Knowledge Management: It forces the explicit documentation of tacit engineering knowledge (design practices), reducing organizational vulnerability when experts leave.
• Human-AI Interaction: The study highlights a tension in the future of engineering work. While agents can remove mundane data-wrangling tasks, they may create a new, exhausting "supervisory role" if not designed carefully. The authors argue for "supertools" that amplify human capability rather than replacing judgment.
• Scalability: The approach is designed to be model-agnostic (using in-context reasoning rather than fine-tuning), allowing organizations to swap underlying LLMs as technology evolves without retraining the system.

Conclusion

The paper demonstrates that LLM agents can effectively orchestrate complex, safety-critical engineering workflows by acting as an adaptive layer between engineers and verified tools. By separating the "what" (orchestration/planning) from the "how" (deterministic execution), DUCTILE offers a robust path forward for automating engineering analysis in dynamic, evolving product development environments.