FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs

This paper introduces FEM-Bench, a structured benchmark based on computational mechanics tasks designed to rigorously evaluate the ability of large language models to generate scientifically valid finite element method code, revealing that even state-of-the-art models struggle to consistently solve these nontrivial problems.

The Gist

Imagine you are trying to teach a brilliant, well-read robot how to be a structural engineer. You don't just want it to write code that looks like it works; you want it to write code that actually understands the laws of physics, like gravity, tension, and how materials bend.

This paper introduces FEM-Bench, a "final exam" designed specifically to test if Large Language Models (LLMs)—the AI brains behind tools like ChatGPT—can do this kind of serious scientific engineering.

Here is a breakdown of the paper using simple analogies:

1. The Problem: The "Calculator" vs. The "Engineer"

Think of current AI models as incredibly fast calculators. If you ask them to write a simple program to add numbers or sort a list, they are great. But if you ask them to simulate how a bridge collapses under a heavy truck, they often fail.

Why? Because building a physics simulation isn't just about writing code; it's about:

• Understanding the rules: Knowing exactly how forces move through a beam.
• Connecting the dots: Taking tiny pieces of a puzzle (small parts of a structure) and snapping them together perfectly to make a whole picture.
• Checking the work: Writing a test to prove the simulation isn't lying.

The authors realized there was no standard "driver's test" for AI in this specific field. Existing tests check if AI can write a website or solve a math riddle, but not if it can build a scientifically valid model of the physical world.

2. The Solution: FEM-Bench (The "Driving Test")

The authors created FEM-Bench, a collection of 33 specific challenges based on a first-year graduate course in computational mechanics.

• The Analogy: Imagine a driving test. You don't just ask the driver to "drive." You ask them to parallel park, merge onto a highway, and navigate a roundabout.
• The Tasks: In FEM-Bench, the "driving" involves things like:
  • Calculating how a 3D beam bends when you push it.
  • Turning a smooth, continuous shape (like a curved bridge) into a digital grid of tiny triangles (called "meshing").
  • Solving complex equations to see if a structure will buckle (collapse) under pressure.

3. The Twist: Two Parts to the Test

The benchmark doesn't just ask the AI to write the code. It asks for two things:

1. The Code: The actual simulation program.
2. The Test: A set of "check-up" rules (unit tests) that the AI must write to prove its own code works.

The Metaphor: It's like asking a student to not only build a bridge out of popsicle sticks but also to write a checklist proving the bridge won't fall down. If the student builds a bridge that looks cool but collapses when you put a weight on it, they fail. If they build a bridge that holds, but they can't write a test to prove it, they also fail.

4. The Results: The AI is Smart, But Not There Yet

The authors ran the top 10 AI models (including the newest ones from Google, OpenAI, and Anthropic) through this exam. Here is what they found:

• The Easy Stuff: The AIs are great at the basics. They can easily handle simple, straight-line problems (like a single wooden beam). It's like they can parallel park perfectly.
• The Hard Stuff: When the problems get complex—like dealing with twisting forces, curved shapes, or predicting when a structure will buckle—the AIs start to stumble.
  • The "Knowledge Gap": Sometimes the AI simply didn't know the specific formula for a complex physical phenomenon. It was like a driver who knows how to drive a car but doesn't know the rules of a roundabout.
  • The "Assembly Gap": Sometimes the AI knew the pieces but couldn't put them together correctly. It was like having all the Lego instructions but snapping the wrong bricks together.
  • The "Testing Gap": Even when the AI wrote a perfect simulation, it often failed to write the tests to prove it was correct. Writing the "checklist" was harder than building the "bridge."

The Score:

• The best model (Gemini 3 Pro) got about 90% of the simple tasks right.
• However, on the hardest tasks (those requiring complex physics without help), no model could solve them consistently.
• Interestingly, the AI was often better at writing the code than writing the tests to verify that code.

5. The "Cheat Sheet" Experiment

The researchers tried to see if they could help the AI by giving it a "cheat sheet" (a system prompt with extra instructions).

• Result: When they gave the AI the specific, complex formulas it was missing, it suddenly got much better at solving the hard problems.
• The Lesson: The AI isn't "stupid"; it just lacks specific, deep knowledge about certain physics formulas. It can't "invent" the math of a collapsing bridge on the fly, but if you hand it the formula, it can use it perfectly.

Summary

FEM-Bench is a reality check for AI in science. It shows that while AI is getting very good at general coding, it still struggles to be a reliable, independent engineer for complex physical problems. It can follow instructions and build simple models, but it cannot yet reliably reason through the deep, messy, and precise laws of physics required to simulate the real world without human help.

The paper concludes that we need benchmarks like this to track progress. As AI gets smarter, the "driving test" will need to get harder to keep measuring real improvement.

Technical Summary

Technical Summary: FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs

Problem Statement

As Large Language Models (LLMs) advance in their ability to reason about the physical world, a critical gap remains in the rigorous evaluation of their capacity to generate scientifically valid physical models. While existing benchmarks assess general programming logic, software engineering skills, or mathematical reasoning, none specifically evaluate an LLM's ability to perform the physical modeling, numerical discretization, and structured computational reasoning required for advanced scientific computing. In fields like computational mechanics, a program that runs but produces non-convergent, unstable, or physically impossible results holds no scientific value. The discipline requires strict adherence to governing equations, numerical consistency, and geometric transformations, yet current benchmarks fail to capture these domain-specific constraints. There is a need for a benchmark that can distinguish between mere code generation and the implementation of physics-grounded, numerically consistent algorithms.

Methodology

The authors introduce FEM-Bench, a diagnostic benchmark designed to evaluate LLMs on their ability to generate correct Finite Element Method (FEM) and Matrix Structural Analysis (MSA) code. Unlike large-scale training datasets, FEM-Bench is a curated suite of 33 introductory but nontrivial tasks aligned with a first graduate course on computational mechanics.

Benchmark Structure

• Task Design: Each task is a self-contained Python module featuring a reference implementation, a set of pytest-style unit tests, and known incorrect implementations ("expected failures") designed to test discriminative power.
• Domains: The suite spans three domains:
  • FEM 1D: Fundamental discretization and linear elasticity.
  • FEM 2D: Higher-order interpolation, multidimensional quadrature, and geometric mappings.
  • MSA 3D: Local element routines, coordinate transformations, global assembly, and eigenvalue problems (buckling analysis).
• Tiered Difficulty: Tasks are classified using a CC/H/T identifier. CC denotes the conceptual challenge (linear vs. geometric nonlinearity), H is the number of helper functions in the reference, and T indicates the tier of helper functions provided to the model (T0: none provided; T1: all provided; T2/T3: partial or none). This design isolates domain knowledge deficits (inability to construct missing components) from compositional reasoning deficits (inability to chain provided components).
• Evaluation Pipeline: The framework evaluates two capabilities:
  1. Function Correctness: Generated code is executed against curated verification inputs and compared to reference outputs using recursive numerical matching.
  2. Test-Suite Evaluation: Generated unit tests must pass on the reference implementation and fail on all provided "expected failure" implementations. The primary metric is the Average Joint Success Rate.

Experimental Setup

The authors evaluated ten state-of-the-art LLMs (including Gemini 3 Pro, GPT-5, Claude Opus 4.5, and open-weight models like Qwen3 and Llama 4) across the 33 tasks. Inference was conducted with low temperature (0.1) and high reasoning settings where supported. The authors also conducted a preliminary study using the GEPA prompt optimization algorithm to determine if performance gaps were due to prompt engineering or inherent domain knowledge deficits.

Key Results

• Performance Ceiling: No model successfully completed the entire FEM-Bench 2025 suite. The best-performing model for function writing, Gemini 3 Pro, solved 30/33 tasks at least once in five attempts and 26/33 tasks consistently (5/5). The best model for unit test writing, GPT-5, achieved an Average Joint Success Rate of 73.8%.
• Task Difficulty Gradient: Models reliably reproduced foundational building blocks (e.g., 1D linear elasticity, basic mesh generation) but struggled significantly as tasks introduced geometric nonlinearity or required reasoning beyond direct formula application.
  • Unsolved Tasks: Three tasks were not solved even once by any model: MSA_3D_assemble_global_geometric_stiffness_T3, MSA_3D_elastic_critical_load_T3, and MSA_3D_local_geometric_stiffness_T0. These tasks require generating complex geometric stiffness matrices without helper functions.
• Error Analysis: Failures were categorized into three deficits:
  1. Domain Knowledge Deficit: Inability to recall specific mechanics formulas (e.g., geometric stiffness coupling terms).
  2. Compositional Reasoning Deficit: Failure to correctly chain provided helper functions.
  3. Algorithmic Fidelity Deficit: Indexing errors, sign convention mismatches, or incomplete routines.
• Prompt Optimization: GEPA optimization revealed that generic reasoning instructions (e.g., "think carefully") did not improve performance. Significant gains were only observed when the optimized system prompts explicitly injected specific domain knowledge (e.g., the exact formulation of the geometric stiffness matrix), confirming that the primary bottleneck is a lack of specific domain knowledge rather than general reasoning capability.

Significance and Claims

The paper claims that FEM-Bench establishes a structured foundation for evaluating AI-generated scientific code, moving beyond simple syntax correctness to assess physical validity and numerical consistency.

• Diagnostic Utility: The benchmark effectively identifies specific failure modes in scientific computing, distinguishing between models that can follow instructions and those that possess the necessary physical reasoning capabilities.
• Current Limitations: The results indicate that while state-of-the-art LLMs are increasingly capable of contributing to structured numerical tasks, they are not yet ready to autonomously implement or verify advanced scientific computing workflows involving geometric nonlinearity or complex eigenvalue problems without external scaffolding or explicit domain knowledge injection.
• Future Direction: The authors posit that future iterations of FEM-Bench will incorporate increasingly sophisticated tasks to track progress. They suggest that bridging the current gap may require integrating LLMs with external tools for symbolic reasoning, automated code execution, and authoritative retrieval, rather than relying solely on pre-trained knowledge.

The work serves as a baseline for the community, highlighting that the "last mile" of scientific code generation requires a level of precision and domain-specific reasoning that current models struggle to achieve in a single-pass, zero-shot setting.