🤖 AI

CAX-Agent: A Lightweight Agent Harness for Reliable APDL Automation

This paper introduces CAX-Agent, a lightweight harness designed to enhance the reliability of APDL automation by implementing a multi-layered architecture with a recovery ladder, demonstrating through empirical benchmarks that model-driven regeneration significantly outperforms rule-based or no-recovery strategies in task completion and intervention reduction.

Original authors: Chenying Lin, Yichen Hai, Yi He, Ran Wang, Haiyan Qiang, Liang Yu

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenying Lin, Yichen Hai, Yi He, Ran Wang, Haiyan Qiang, Liang Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but slightly impulsive architect (the AI) how to build a bridge using a very strict, old-fashioned construction manual (the software called MAPDL). The architect is great at writing the blueprints, but if the manual says "Error: Beam too thin," the architect might get confused, try to write a new blueprint that makes the same mistake, or just give up entirely.

This paper introduces CAX-Agent, a "construction site manager" designed to sit between the architect and the manual to make sure the bridge actually gets built.

Here is the breakdown of how it works and what the researchers found, using simple analogies:

1. The Problem: The "One-and-Done" Trap

Usually, when you ask an AI to write code for a simulation, it tries once. If the computer says "Error," the AI often panics or stops. The whole project fails.

The Paper's View: Without a manager to catch mistakes, a single error stops the entire pipeline.

2. The Solution: The "Agent Harness" (The Site Manager)

Instead of letting the AI run wild, the authors built a Harness. Think of this as a strict site manager who:

Watches the AI: It sees what the AI writes.
Checks the Manual: It runs the code and reads the error messages.
Decides the Next Move: If there's an error, the manager decides whether to fix it with a quick rule or ask the AI to try again.

The system is built in three layers:

The AI (The Architect): Writes the code.
The Harness (The Manager): Organizes the work, checks for errors, and manages retries.
The Solver (The Construction Crew): Actually builds the simulation.

3. The "Recovery Ladder" (How they fix mistakes)

When the construction crew hits a snag, the manager has a "ladder" of ways to fix it, starting from the easiest to the hardest:

Rung 1 (Rules): "Oh, the mesh is too big? Let's just automatically make the numbers smaller." (Like a pre-written cheat sheet).
Rung 2 (Model): "The error log says the beam is wrong. AI, read this error and rewrite the blueprint yourself."
Rung 3 (Context): "Let's give the AI more details about the problem."
Rung 4 (Human): "Okay, we're stuck. Call a human engineer."

4. The Experiment: A Race Between Three Teams

The researchers tested three different ways to handle errors on 50 simple bridge-building tasks (beams, plates, and cylinders). They ran each task three times to be sure.

Team A (No Recovery): The AI tries once. If it fails, it stops.
- Result: Like a driver who hits a pothole and immediately stops the car. They failed often.
Team B (Rule Only): The AI tries once. If it fails, the manager applies a strict, pre-written rule to fix it (e.g., "If error X, do Y").
- Result: Better than Team A, but the rules were too rigid. Sometimes the "cheat sheet" didn't fit the specific problem.
Team C (Model Only): The AI tries once. If it fails, the manager forces the AI to read the error message and rewrite the code itself, up to three times.
- Result: This team won by a landslide. The AI was smart enough to understand why it failed and fix it creatively.

5. The Results: Why "Model Only" Won

The researchers had two human judges look at the final blueprints (blindly, so they didn't know which team made them) and score them.

Success Rate: Team C (Model Only) succeeded 93% of the time. Team B (Rules) succeeded 77%, and Team A (No Recovery) only 69%.
Autonomy: Team C finished the job without needing a human to step in 84% of the time. Team B and Team A needed human help 100% of the time when they failed.
The "Human" Factor: The "Rule Only" team actually required a human to confirm the fix before trying again, which is why their "zero intervention" score was 0. The AI-driven team fixed things on its own.

6. The Catch (Limitations)

The authors are honest about the limits of their study:

Simple Tasks: They only tested simple, straight-forward structures (like straight beams). They didn't test complex, twisted, or "messy" real-world engineering problems.
Specific Tools: They only used one specific software (MAPDL) and one specific AI model.
The "Thin Wall" Issue: Even the winning team struggled with very thin, delicate parts that are hard to mesh (like a very thin sheet of metal). The AI sometimes couldn't figure out how to build the grid for those specific shapes.

The Bottom Line

This paper proves that if you want an AI to automate engineering simulations, you can't just let it write code and hope for the best. You need a manager (the Harness) that forces the AI to read its own mistakes and try again.

While simple "cheat sheets" (rules) help a little, letting the AI think about its own errors and rewrite its plan is the only way to get a system that works reliably without needing a human to hold its hand every time it stumbles.

Technical Summary: CAX-Agent for Reliable APDL Automation

Problem Statement

Large Language Models (LLMs) deployed for Mechanical Analysis and Design (MAPDL) finite-element simulation face significant reliability challenges. While LLMs can generate code, they often lack the structured execution control, tool encapsulation, and fault recovery mechanisms required for engineering workflows. In simulation pipelines, runtime errors—such as meshing failures, convergence issues, or missing results—are common even for structurally simple tasks. Without explicit recovery mechanisms, a single failure terminates the entire pipeline. Existing agent frameworks often expect the LLM to manage its own execution, which fails to provide the necessary engineering skeleton for robust, repeatable automation. There is a lack of controlled, empirical evidence comparing different recovery policies (e.g., rule-based vs. model-driven) under repeated conditions with human-validated outcomes.

Methodology

System Architecture: CAX-Agent

The paper introduces CAX-Agent, a lightweight, domain-native agent harness designed specifically for APDL automation rather than adapting a generic framework. The architecture is organized into three layers:

LLM Service Layer: Includes a local lightweight model (Qwen-27B) for fast first-pass generation and an external high-capability API (Claude Sonnet 4.6) for complex reasoning and repair.
Agent Harness (Orchestrator): This middleware manages tool lifecycles, workflow state, and fault diagnosis. Crucially, the orchestrator—not the LLM—owns retry budgets, tool dispatch, and stop conditions. It implements a Recovery Ladder that escalates from deterministic rule patching to model-driven regeneration, context enrichment, and finally human intervention.
Solver Backend: A unified interface connecting to MAPDL engines (PyMAPDL, CLI, or fallback modes).

Experimental Design

The study evaluates three distinct recovery strategies on a benchmark of 50 standard structural tasks (beams, plates, cylinders under static, modal, and thermal loading). The tasks are deliberately simple to isolate the effect of the recovery policy rather than task difficulty.

No Recovery: One-shot execution with no repair attempts.
Rule Only: Deterministic, log-to-patch correction using four string-transform rules (e.g., doubling element size for mesh failures, inserting time-stepping for convergence issues) without additional model reasoning.
Model Only: Error-log-conditioned model regeneration with bounded retries (up to 3 forced retry rounds) where the LLM reads the error log, diagnoses the failure, and regenerates the script.

Evaluation Metrics

Each of the 450 total case-runs (50 tasks × 3 strategies × 3 repeats) was evaluated using:

Task Score ( $t_i$ ): A human-assessed score (0–4) for correctness and completeness of the output image.
Autonomy Score ( $a_i$ ): A system-derived score (0–3) based on whether manual intervention was required.
Recovery Efficiency Score ( $e_i$ ): Based on retry counts and solver outcomes.
Total Score ( $q_i$ ): The sum of the above ( $t_i + a_i + e_i$ ), ranging from 0 to 10.
Completion Rate: Binary metric indicating if at least one post-processing image was produced.
Statistical Analysis: Pairwise comparisons using Cliff's $\delta$ (effect size) and Mann–Whitney U tests, with inter-rater agreement measured by quadratic weighted Cohen's $\kappa$ (0.84).

Key Contributions

CAX-Agent Architecture: A three-layer, MAPDL-native agent harness featuring a recovery ladder and an orchestrator-centric control model that separates semantic repair (LLM) from execution control (Harness).
Controlled Empirical Evaluation: A rigorous, repeated-run comparison of three recovery strategies on 50 standardized APDL tasks, utilizing blind human scoring and inter-rater validation.
Empirical Evidence: Demonstration that model-driven recovery substantially outperforms both rule-based repair and no-recovery baselines in completion rates, output quality, and autonomy.

Results

The Model Only strategy achieved superior performance across all metrics compared to Rule Only and No Recovery:

Completion Rate: Model Only (0.9267) significantly outperformed Rule Only (0.7733) and No Recovery (0.6933). The 95% confidence intervals for Model Only did not overlap with the baselines.
Task Quality: Model Only achieved a mean task score of 3.59/4, compared to 3.17/4 for Rule Only and 2.74/4 for No Recovery.
Total Score: Model Only scored 9.16/10, significantly higher than Rule Only (7.03/10) and No Recovery (5.60/10).
Zero-Intervention Rate: Model Only achieved 0.84, whereas Rule Only achieved 0.00. The paper notes that Rule Only's zero score reflects its design property requiring a human confirmation step before re-execution, preventing fully autonomous operation.
Effect Sizes: Large effect sizes were observed (Cliff's $\delta$ = 0.81–0.87) between Model Only and the other strategies, indicating a high probability that a randomly selected Model Only run outperforms a Rule Only run.

Failure Analysis: Residual failures were concentrated in static structural settings, particularly those involving thin-wall or mesh-sensitive geometric features. Deterministic rules were found insufficient for long-tail solver and script-path failures, whereas model-driven repair adapted to broader failure signatures.

Significance and Claims

The paper claims that CAX-Agent demonstrates that a lightweight, domain-native harness can transform MAPDL simulation from scattered, unreliable LLM calls into a standardized, traceable, and repeatable engineering workflow.

Key findings include:

Necessity of Recovery: Explicit recovery mechanisms are central to system reliability; without them, single failures terminate pipelines.
Superiority of Model-Driven Repair: While deterministic rules offer some improvement over no recovery, they do not reliably raise output quality or achieve full autonomy. Model-driven recovery is the most practical default strategy for current CAX deployment.
Orchestrator-Centric Control: Separating the LLM's reasoning from the harness's execution control (retry budgets, stop conditions) provides behavioral guarantees that prompting alone cannot match.
Methodological Contribution: The study provides a reproducible template for empirical harness evaluation (controlled ablation, repeated runs, multi-axis scoring) that can be applied to other harness components and simulation backends.

The authors maintain modesty regarding the scope, noting that the benchmark uses simple geometries and linear elasticity. They state that validation on larger, more diverse benchmarks and across multiple solver backends is required before practical deployment in complex industrial settings. However, the core finding—that model-driven recovery outperforms rule-based and no-recovery approaches under controlled conditions—is presented as robust and statistically significant.