🤖 machine learning

ShapeBench: A Scalable Benchmark and Diagnostic Suite for Standardized Evaluation in Aerodynamic Shape Optimization

This paper introduces ShapeBench, an open-source, scalable benchmark and diagnostic suite featuring 103 diverse aerodynamic shape optimization tasks and standardized baselines to enable fair, systematic evaluation of both classical and LLM-driven optimization methods, revealing significant performance variance across problem classes and the need for more general-purpose approaches.

Original authors: Shaghayegh Fazliani, Krissh Chawla, Jack Guo, Yiren Shen, Matthias Ihme, Madeleine Udell

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shaghayegh Fazliani, Krissh Chawla, Jack Guo, Yiren Shen, Matthias Ihme, Madeleine Udell

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to design the perfect airplane wing, a super-fast car, or a futuristic drone. You want to find the shape that uses the least fuel or creates the least drag. In the past, engineers had to test these shapes one by one, which was slow and expensive. Now, we have powerful computer programs (optimizers) and artificial intelligence (AI) to help us find the best shapes quickly.

But here's the problem: How do we know which computer program is actually the best?

Right now, it's like having a race where every runner runs on a different track, with different rules, and different judges. One program might look like a champion on a small, simple track (a 2D wing), but fail miserably on a complex, bumpy track (a full 3D drone). Because everyone tests their tools on different things, we can't fairly compare them.

Enter "ShapeBench."

Think of ShapeBench as a massive, standardized "Olympics" for aerodynamic design tools.

1. The Stadium (The Benchmark)

The authors built a giant digital stadium containing 103 different challenges. These aren't just simple tasks; they cover a huge variety of "athletes" (shapes):

2D Airfoils: Like a slice of a wing.
3D Wings & Planes: From small drones to massive blended-wing bodies.
Cars: Testing how air flows over vehicles.
Mixed-Variable Designs: Some challenges involve changing numbers (continuous) and others involve switching categories (like choosing between a T-tail or a normal tail).

Every tool entering this stadium has to run the exact same race, under the exact same time limit (budget), so we can see who truly wins.

2. The Fast Track vs. The Real Track (Surrogates vs. CFD)

Running a real physics simulation (like a wind tunnel in a computer) is incredibly slow and expensive. It's like running a marathon in full armor.

The Fast Track (Surrogates): To speed things up, ShapeBench uses "surrogate" models. These are AI models trained to guess the results of the real physics. They are like a sprinter in running shoes—super fast, but sometimes they might guess wrong if they get too creative.
The Real Track (CFD): For the final check, ShapeBench has a "High-Fidelity" mode. This is the real physics simulation. It's slow, but it's the truth.

ShapeBench lets researchers run the race on the Fast Track first to find a winner, and then double-check that winner on the Real Track to make sure they didn't cheat.

3. The "Hall of Shame" (The Diagnostic Suite)

This is one of the coolest parts. Sometimes, a computer program gets so good at "cheating" the Fast Track that it finds a shape that looks amazing on paper but is physically impossible in real life.

The Analogy: Imagine a car design that has a drag coefficient of zero because the computer decided to make the car's wheels disappear and the car float 1 inch off the ground. The computer says, "Great job! Zero drag!" But a human looks at it and says, "That's a flying carpet, not a car."

ShapeBench has a Diagnostic Suite (a smart referee) that looks at the winning designs and says, "Wait a minute. This shape is physically impossible. The computer is exploiting a loophole in the rules." It flags these "hallucinations" so researchers don't get fooled by fake wins.

4. The Big Surprise (The Results)

The authors ran the Olympics with many different types of "runners":

Classical Runners: Old-school math tools (like gradient-based methods).
Evolutionary Runners: Tools that mimic natural selection (like Particle Swarm Optimization).
AI Runners: New tools powered by Large Language Models (LLMs) that "think" about the design.

The Shocking Discovery:
There was no single champion.

A tool that won the "Airfoil" race often lost the "Car" race.
A tool that won the "Simple" race often failed the "Complex" race.
The ranking of who was "best" changed so much between tasks that the correlation was almost zero (0.013).

The Lesson: You cannot say "Tool X is the best optimizer." You can only say "Tool X is the best optimizer for this specific type of problem." If you pick a tool based on one small test, you might pick the wrong one for your actual job.

5. The New Contender (ShapeEvolve)

The authors also introduced a new runner called ShapeEvolve. It's an AI specifically trained to understand aerodynamics. It doesn't just guess numbers; it "reads" the physics, looks at the shape, and evolves better designs over time. In some of the hardest races (like the Collaborative Combat Aircraft), this specialized AI outperformed the classical tools, showing that for very complex, high-dimensional problems, AI might be the way forward.

Summary

ShapeBench is a tool to stop the confusion in aerodynamic design. It provides a fair, standardized way to test different optimization tools across a wide variety of shapes. It teaches us that one size does not fit all: the best tool depends entirely on the specific shape you are trying to design. It also warns us to be careful of AI tools that find "impossible" solutions by cheating the simulation, providing a safety net to catch those errors before they reach the real world.

Technical Summary: ShapeBench

Problem Statement

Aerodynamic Shape Optimization (ASO) has seen rapid advancements in optimizer capabilities, including gradient-based methods, evolutionary algorithms, Bayesian optimization, and emerging Large Language Model (LLM)-guided approaches. However, the field lacks standardized, scalable evaluation frameworks. Current benchmarks are often narrow (e.g., restricted to 2D airfoils), lack diversity in problem formulations, and fail to provide matched-budget comparisons. This limitation prevents fair cross-method evaluation and obscures how well algorithms generalize across different shape classes and optimization regimes. Furthermore, the reliance on surrogate models to reduce computational costs introduces risks of approximation errors and "surrogate exploitation," where optimizers find designs that maximize surrogate predictions but are physically implausible or infeasible. There is a critical need for a unified benchmark that spans diverse aerodynamic tasks, supports fidelity-gap analysis, and includes rigorous diagnostic tools to validate design credibility.

Methodology

The authors introduce ShapeBench, an open-source benchmark and diagnostic suite designed to standardize ASO evaluation. The methodology encompasses four core components:

Unified Task Suite: ShapeBench comprises 103 instantiated ASO tasks spanning eight aerodynamic shape categories: 2D airfoils, 3D delta wings, blended wing bodies (BWB), transonic swept wings, passenger cars, collaborative combat aircraft (CCA), and mixed-variable conceptual designs (CERAS and Supersonic Transport Aircraft). Tasks cover single/multi-objective, single/multi-point, and mixed-variable optimization regimes.
Paired Simulation Pipelines: Each task is paired with a fast, pre-trained surrogate model for efficient search (e.g., NeuralFoil for airfoils, VortexNet for delta wings, Transolver-based models for CCA and cars). Where feasible, a high-fidelity Computational Fluid Dynamics (CFD) pipeline is available for final verification, enabling systematic fidelity-gap analysis.
Standardized Baselines and Optimizers: The framework provides a reproducible evaluation protocol with matched computational budgets. It includes established classical optimizers (Adjoint/IPOPT, L-BFGS-B, PSO, CMA-ES, Bayesian Optimization) and general-purpose LLM optimizers (OpenEvolve, ShinkaEvolve). Additionally, the authors introduce ShapeEvolve, a domain-specialized LLM evolutionary method that integrates ASO-specific prompts, aerodynamic feedback, and a secondary LLM sampler for adaptive local search.
Diagnostic Suite: To address surrogate exploitation, ShapeBench includes a two-stage diagnostic pipeline. The first stage performs deterministic checks (feasibility, geometric validity, aerodynamic plausibility). The second stage utilizes an LLM integrative judge to analyze design plots and flow fields, generating auditable reports that flag unphysical designs and suggest mitigation strategies.

Key Contributions

Cross-Domain Evaluation Suite: The release of ShapeBench, covering 103 tasks across eight shape categories, enabling systematic cross-task comparison of optimizers under a unified API.
Empirical Evidence of Non-Generalization: Documentation of substantial instability in optimizer rankings across different shape categories and problem formulations, with a mean pairwise Spearman $\rho \approx 0.013$ . This demonstrates that conclusions drawn from single-task experiments do not reliably generalize.
Novel Surrogate and Dataset: The development of COCOANet, a Transolver-based surrogate for Collaborative Combat Aircraft (CCA) tasks, trained on a new open dataset of 3,570 high-fidelity CFD simulations. This addresses a setting with limited public training data.
Diagnostic Framework: A diagnostic suite that moves beyond scalar reward values to evaluate design credibility, identifying failure modes such as boundary collapse and geometric over-deformation.
Specialized Baselines: The introduction of ShapeEvolve, a rigorous reference for LLM-driven optimizers, alongside standardized baselines for classical methods, facilitating fair head-to-head comparisons.

Results

Experiments conducted on ShapeBench reveal several critical insights:

Ranking Instability: Optimizer performance is highly task-dependent. For instance, Bayesian Optimization and PSO perform well on CERAS and delta-wing tasks but rank poorly on CCA tasks, while LLM-driven methods show the opposite trend. No single optimizer dominates across all categories.
Budget Dependence: Optimizer rankings shift over the course of optimization; no method maintains the top rank across the entire evaluation budget.
Surrogate Exploitation: In tasks like the 3D car design (DrivAerStar) and blended wing body (BWB), optimizers frequently converged to designs with extremely low drag coefficients that were geometrically unrealistic (e.g., rear ground clearance collapse). The Diagnostic Suite successfully identified these as artifacts of surrogate exploitation rather than genuine physical improvements.
Fidelity Gaps: Comparisons between surrogate predictions and high-fidelity solvers (e.g., XFOIL vs. NeuralFoil) revealed fidelity gaps, particularly in post-stall regimes or at the edges of the training distribution, where errors could reach up to ~18% in lift-to-drag ratios.
ShapeEvolve Performance: ShapeEvolve demonstrated competitive performance, often converging faster than classical methods in complex, high-dimensional spaces like the CCA task, where it significantly outperformed classical baselines.

Significance

The paper posits that ShapeBench is essential for advancing ASO research by providing a reproducible, comparable, and comprehensive evaluation environment. Its significance lies in:

Correcting Overfitting: By exposing the instability of single-task conclusions, it discourages the overfitting of methods to specific, narrow problem setups.
Ensuring Physical Validity: The diagnostic suite provides a mechanism to distinguish between genuine algorithmic improvements and surrogate artifacts, ensuring that high-scoring designs are physically credible.
Facilitating LLM Integration: By offering matched-budget comparisons and specialized baselines, it enables rigorous assessment of LLM-driven methods against established classical optimizers.
Community Standardization: The release of tasks, surrogates, training data, and evaluation scripts supports reproducible research and lowers the barrier for integrating new tasks and methods into the ASO ecosystem.

The authors conclude that while ShapeBench significantly improves reproducible ASO evaluation, challenges remain, particularly regarding the availability of high-fidelity verification for all settings and the inherent vulnerability of surrogate-driven optimization to exploitation. Future work aims to expand high-fidelity coverage and strengthen uncertainty controls.