Technical Summary: Search Hardness-Aware LLM-Based Problem Formulation for Expensive Simulation-Driven Design

Problem Definition

Expensive simulation-driven design is a critical process in engineering (e.g., airfoil and microelectronic design) where the goal is to identify design parameters satisfying natural-language requirements within a limited budget of high-fidelity, time-consuming simulations. The design process typically involves two stages: problem formulation (converting natural-language requirements into a solvable mathematical optimization problem) and optimization (searching for parameters under that formulation).

While existing research heavily focuses on improving optimization algorithms (e.g., surrogate-assisted methods) under fixed problem formulations, it often assumes the formulation itself is correct and effective. However, the formulation defines the objectives and constraints, thereby shaping the search landscape and determining the search path from initial solutions to target solutions. Current automated formulation methods, often driven by Large Language Models (LLMs), primarily focus on aligning the mathematical formulation with the design intent (semantic correctness) but overlook whether the resulting formulation induces an efficient search process. In expensive design scenarios, different valid formulations can lead to search paths with vastly different simulation costs, making the discovery of search-efficient formulations essential.

Methodology: SHA-PF

The authors propose SHA-PF (Search Hardness-Aware Problem Formulation), a framework that uses initial simulation data to discover formulations that guide the optimizer toward target solutions with fewer evaluations.

1. Core Insight: Search Hardness in Initial Data

SHA-PF operates on the finding that initial evaluated designs contain partial search trajectories toward the target region.

• Frequently observed states represent regions easily covered by sampling; prioritizing them offers diminishing returns.
• Rarely observed states correspond to "hard yet promising" regions that are difficult for optimizers to reach naturally but are informative for guiding the search.
• The framework posits that a formulation assigning higher priority to these rare, promising paths will enable the optimizer to reach the target solution with fewer expensive evaluations.

2. Search Hardness Metric and Anchor Selection

To operationalize this insight, SHA-PF defines a search hardness metric $h(z)$ for observed satisfaction states $z$:
$$h(z) = \frac{|\{i \mid z_i = z\}|}{|\{i \mid z \succ z_i\}| + \epsilon}$$

• The numerator represents the rarity of the state (lower count = higher rarity).
• The denominator measures progress potential by counting initial observations dominated by $z$ (using Pareto dominance).
• A lower $h(z)$ indicates a state that is rare but strongly supported by dominance, representing a "hard yet promising" direction.
• An anchor state ($z_{anc}$) is selected from the non-dominated observed states by minimizing $h(z)$.

3. Formulation Search Objective

The framework searches the formulation space to find a formulation $F$ that ranks responses corresponding to the anchor state higher than other responses. The objective is:
$$F^* = \arg \max_{F \in \hat{\Phi}_R} \frac{1}{|P_{anc}|} \sum_{(i,j) \in P_{anc}} I[y_i \triangleright_F y_j]$$
where $P_{anc}$ contains pairs where $y_i$ corresponds to the anchor state and $y_j$ does not. This encourages the optimizer to follow the trajectory toward the hard but promising anchor.

4. Evolutionary Formulation Refinement

SHA-PF employs an evolutionary loop to search for the optimal formulation:

• Generation & Repair: An LLM generates a draft formulation from natural-language requirements. A second "repair" LLM checks and revises the draft to ensure it is executable and semantically aligned with the requirements.
• Evolutionary Search: A population of repaired formulations is evolved. Offspring are generated using specific operators (component recombination, constraint restructuring, signal reparameterization, and semantic exploration) applied to parent formulations.
• Selection: Each candidate is scored based on the formulation search objective defined above. The best-performing formulations are retained for the next generation.

Key Contributions

1. Theoretical Insight: The paper reveals that rarely observed but promising search paths in initial simulation data are more valuable for continued search than commonly observed paths.
2. Evaluation Criterion: It proposes an anchor-state-based formulation evaluation criterion that favors formulations prioritizing these hard yet promising paths, thereby encouraging the optimizer to explore them.
3. Framework (SHA-PF): The authors integrate this criterion with LLM-based generation, validity repair, and iterative evolutionary refinement to automatically discover efficient problem formulations.
4. Empirical Validation: The framework is validated on a real-world multi-objective benchmark (HBV rainfall-runoff calibration) and five expensive antenna design benchmarks.

Experimental Results

The authors evaluated SHA-PF against baselines including manual expert formulations ($F_{Expert}$) and direct LLM-generated formulations ($F_{LLM}$).

• HBV Calibration: SHA-PF reached the feasible region (satisfying all four hydrological criteria) at evaluation 127, significantly earlier than all baselines. Furthermore, the best feasible solution found by SHA-PF dominated those found by baselines in terms of Nash-Sutcliffe efficiency (NSE) and runoff coefficient error (ROCE).
• Antenna Design Benchmarks:
  • SHA-PF successfully found requirement-satisfying designs for all five antenna tasks (HAE, HSL, HSE, CSL, WHG).
  • Most baselines failed to find a satisfying design within the budget (300 HFSS simulations).
  • Where baselines succeeded (e.g., $F_{LLM-A}$ on HAE and CSL), SHA-PF required significantly fewer evaluations (e.g., 10 evaluations for CSL vs. 36 for the baseline) and less time.
  • On tasks where baselines failed entirely (HSL, HSE, WHG), SHA-PF succeeded with 110–137 fewer simulations than the budget limit.
• Cross-Optimizer Robustness: When tested with different optimizers (SCBO, CEBO, DSI), SHA-PF's formulations consistently outperformed baseline formulations, demonstrating that the efficiency gain is attributable to the formulation rather than a specific optimizer.
• Ablation Studies:
  • Removing Anchor Selection (replacing it with random selection) degraded performance, increasing simulation costs and causing failures on some tasks.
  • Removing Formulation Repair caused total failure on all antenna tasks, highlighting that raw LLM generation often produces executable but semantically misaligned formulations.
• LLM Backbone Sensitivity: SHA-PF remained effective across different LLM backbones (GPT-5, Claude, DeepSeek, etc.). While backbones affected convergence speed and API costs, lightweight models (e.g., GPT-5-Mini) achieved comparable design-search efficiency to larger models.

Significance and Claims

The paper claims that SHA-PF addresses a critical gap in expensive simulation-driven design: the assumption that a "correct" formulation automatically leads to an efficient search. By explicitly optimizing for search hardness using initial data, SHA-PF discovers formulations that guide optimizers along more efficient trajectories.

The authors state that their method significantly reduces the number of expensive simulation evaluations required to meet design requirements compared to both human-expert and direct LLM-based approaches. They emphasize that this approach does not require additional training data or solver-based verification (which is often impossible in expensive design), relying instead on the initial simulation responses to infer search landscape properties. The work suggests that future efficiency gains in simulation-driven design may depend as much on how the problem is formulated as on the optimization algorithm used to solve it.