Solver-Independent Automated Problem Formulation via LLMs for High-Cost Simulation-Driven Design

1. Problem Statement

In high-cost simulation-driven design (e.g., antenna, aerospace, robotics), engineers must translate ambiguous, natural language design requirements into precise mathematical optimization models (objective functions and constraints) to guide algorithms.

• The Bottleneck: This translation is time-consuming, relies heavily on scarce expert knowledge, and is prone to error.
• Limitations of Existing LLM Approaches:
  • Prompt-based methods: Often fail to accurately capture complex, domain-specific constraints due to vague natural language inputs.
  • Fine-tuning methods: Typically rely on solver feedback (running the optimization to verify if the generated model works) to filter training data. In high-cost simulation domains, running these solvers is computationally prohibitive, making data filtering impossible.
• Core Challenge: How to automatically generate high-quality training data and evaluate the resulting optimization formulations without incurring the high cost of running physics simulations (solvers).

2. Methodology: The APF Framework

The authors propose APF (Automated Problem Formulation), a solver-independent framework that uses Large Language Models (LLMs) to convert requirements into executable code. The framework consists of four key modules:

A. Unified Representation

• Requirements ($R$): Formalized as a tuple $(Z, M, C)$, where $Z$ is the evaluation subregion (e.g., frequency band), $M$ is the metric (e.g., radiation efficiency), and $C$ is the design intent (threshold or optimization goal).
• Test Instances ($I$): Defined as the performance response curves of design solutions derived from historical high-fidelity simulations.

B. Data Generation & Augmentation

• Equation Generation: LLMs are prompted to convert structured requirement tuples into mathematical equations (Python code).
• Data Augmentation: To improve generalization, the framework employs:
  1. Semantic Paraphrasing: Rewriting requirements to capture linguistic variations while keeping constants/units fixed.
  2. Order Permutation: Randomizing the order of requirements and equations to prevent the model from relying on positional cues.

C. Solver-Independent Evaluation (The Core Innovation)

Instead of running expensive simulations to verify if a generated equation is correct, APF uses a ranking consistency approach:

1. Test Instance Annotation: An LLM acts as a "Judge" to generate a reference ranking ($\pi_{LLM}$) of a set of test instances based on how well they satisfy the natural language requirements. This uses a listwise ranking strategy (evaluating all instances at once) to capture complex trade-offs.
2. Equation Evaluation: The generated mathematical equation is executed on the same test instances to produce a predicted ranking ($\pi_E$).
3. Quality Scoring: The quality of the equation is measured by the Spearman correlation coefficient between $\pi_E$ and $\pi_{LLM}$. High correlation implies the equation accurately captures the design intent.
4. Selection: Only high-quality samples (high correlation) are retained to form the training dataset ($D_{HQ}$).

D. Supervised Fine-Tuning (SFT)

The curated high-quality dataset is used to fine-tune open-source LLMs (e.g., Llama-3, Qwen, Mistral), enabling them to generate accurate, executable optimization models directly from natural language.

3. Key Contributions

1. Solver-Independent Data Pipeline: A novel framework that constructs high-quality fine-tuning datasets without requiring expensive solver feedback, overcoming the primary barrier in simulation-driven design.
2. LLM-as-a-Judge Evaluation: Introduction of a listwise ranking strategy where LLMs evaluate the alignment between generated equations and design intent using test instance rankings, replacing the need for physical simulations.
3. Data Augmentation & Selection: A robust strategy combining semantic paraphrasing and order permutation with strict correlation-based filtering to ensure training data reliability.
4. Industrial Validation: Successful application to a complex, real-world antenna design problem involving conflicting multi-band constraints.

4. Experimental Results

The method was validated on a three-layer filtering patch antenna design task, which involves optimizing radiation efficiency across five distinct frequency bands with heterogeneous and conflicting goals.

• Performance vs. Baselines:
  • APF significantly outperformed prompt-based baselines (GPT-4o, DeepSeek-V3, Chain-of-Experts, OptiMUS) and raw open-source models.
  • Key Metric: The fine-tuned Llama-3.1-8B model achieved an overall alignment score of 0.7976, compared to 0.6651 for GPT-4o and 0.2248 for the base Llama-3 model.
  • Fine-tuned models consistently matched or exceeded the performance of much larger proprietary models.
• Design Outcome:
  • Optimization runs using APF-generated formulations successfully met all design requirements (passband maximization, stopband suppression).
  • Formulations from other methods failed to meet critical constraints (e.g., passband efficiency or stopband limits), leading to suboptimal designs.
• Efficiency: The listwise evaluation method was 26x faster and 23x cheaper than pairwise comparison baselines while maintaining higher ranking correlation.
• Ablation Studies: Removing data augmentation or data selection modules resulted in significant performance drops, confirming the necessity of both components.

5. Significance

• Democratizing High-Cost Design: APF enables the automation of problem formulation in fields where simulation costs are too high for traditional reinforcement learning or solver-based filtering.
• Bridging the Intent-Model Gap: It effectively translates vague engineering intent into rigorous mathematical constraints, reducing the reliance on human experts for the initial modeling phase.
• Scalability: By removing the dependency on solver feedback, the framework offers a scalable path to automate problem formalization across various physics-based engineering domains (e.g., aerodynamics, structural optimization).
• Model Efficiency: Demonstrates that smaller, fine-tuned open-source models can outperform massive proprietary models in specific industrial tasks when trained on high-quality, solver-independent data.

Limitations: The current evaluation is limited to antenna design, and the method relies on the context window size of LLMs for processing test instances, which may constrain the scale of validation datasets. Future work aims to extend this to broader physics-based domains.