💻 computer science

ThermoQA: A Three-Tier Benchmark for Evaluating Thermodynamic Reasoning in Large Language Models

The paper introduces ThermoQA, a three-tier open-source benchmark of 293 thermodynamics problems evaluated against CoolProp 7.2.0 ground truth, which reveals that while frontier LLMs achieve high accuracy, their reasoning capabilities degrade significantly with task complexity and exhibit notable variability across runs.

Original authors: Kemal Düzkar

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kemal Düzkar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new engineer to design a power plant. You have two candidates: Candidate A is a human encyclopedia who has memorized every single page of the steam tables in the library. Candidate B is a brilliant problem-solver who understands how steam behaves, even if they haven't memorized every specific number.

For a long time, we've tested AI models (like the ones powering chatbots) with multiple-choice quizzes. It's like asking, "Is the sky blue? A) Yes, B) No." Both candidates could guess the right answer.

But this new paper, THERMOQA, changes the game. Instead of a quiz, it hands the candidates a complex, real-world engineering problem and says: "Show me your work, step-by-step, and calculate the exact efficiency of this power plant."

Here is the breakdown of what the paper found, using simple analogies.

1. The Three Levels of the Test

The researchers created a "video game" with three levels of difficulty to see how the AI handles different types of thinking:

Level 1: The Flashcard Test (Property Lookups).
- The Task: "What is the energy of water at 400°C and 5 MPa?"
- The Skill: Pure memory. It's like looking up a word in a dictionary.
- The Result: The AI models were great at this. They acted like super-fast encyclopedias.
Level 2: The Component Test (Component Analysis).
- The Task: "Here is a turbine and a pump. Calculate how much energy is lost as heat and how much work is produced."
- The Skill: Multi-step logic. You can't just look up the answer; you have to do a chain of math, like following a recipe where if you mess up step 2, the whole cake burns.
- The Result: The models started to stumble. Some got the math wrong; others forgot the physics rules.
Level 3: The Boss Battle (Full Cycle Analysis).
- The Task: "Design a complete power plant cycle involving a gas turbine, a steam turbine, and a heat exchanger. Calculate the total efficiency."
- The Skill: System thinking. This is the hardest level. It requires connecting dozens of steps together.
- The Result: This is where the "memorizers" failed and the "thinkers" won.

2. The Big Surprise: Memorization $\neq$ Intelligence

The most shocking discovery was that being good at Level 1 did not mean you were good at Level 3.

The "Gemini" Effect: One model (Gemini) was the absolute champion at Level 1 (memorizing facts). But when the test got harder, its performance dropped significantly. It was like a student who aced the vocabulary test but failed the essay because they didn't understand the concepts.
The "Claude" Effect: Another model (Claude Opus) was slightly slower at memorizing facts but was a master at the complex, multi-step reasoning. It climbed to the top spot once the test got hard.

The Lesson: Just because an AI can recite facts perfectly doesn't mean it understands how those facts work together.

3. The "Supercritical" Trap

The researchers found a specific "trap" in the test: Supercritical Water.

The Analogy: Imagine you are learning to drive on a straight, empty road (standard water). You drive perfectly. But then you are asked to drive on a winding, icy mountain pass where the road rules change instantly (supercritical water).
The Result: The AI models crashed. They had memorized the "straight road" data from textbooks, but when the physics got weird and non-linear (near the critical point), their "guessing" logic failed spectacularly. One model was off by 27% on a single calculation. This proved they were just pattern-matching, not actually "understanding" the physics.

4. The "Refrigerator" Blind Spot

The AI models were trained mostly on data about water and steam (because that's what power plants use). They knew almost nothing about R-134a, the refrigerant used in your home air conditioner and fridge.

The Analogy: It's like a chef who is a master at cooking steak but has never seen a vegetable. When asked to make a salad, they tried to cook the lettuce like a steak.
The Result: The models performed terribly on refrigeration cycles, revealing a huge gap in their training data.

5. The "Consistency" Problem

The researchers asked the same AI to take the test three times.

The Stable Model: One model (GPT-5.4) gave almost the exact same answer every time. It was like a reliable calculator.
The Unstable Model: Another model (DeepSeek) gave a great answer the first time, a mediocre answer the second, and a wild guess the third.
Why it matters: If you are building a bridge or a nuclear plant, you need an engineer who gives you the same answer every time. You don't want an engineer who gets lucky 50% of the time.

6. Token Efficiency: The "Thinking" Cost

The paper also measured how much "brain power" (computing tokens) the models used.

One model (Claude) spent a massive amount of "thinking time" to get the right answer. It was thorough but expensive.
Another model (Gemini) gave a quick answer with very little "thinking," but it was slightly less accurate.
The Trade-off: It's the difference between a student who spends 3 hours solving a math problem to get it right, versus one who guesses in 5 minutes.

The Bottom Line

THERMOQA is a wake-up call for the AI world.

Don't be fooled by flashcards: High scores on simple fact-recall tests don't mean an AI can do complex engineering.
Real-world physics is hard: When things get weird (like supercritical fluids) or involve real-world materials (like refrigerants), current AI models struggle.
Consistency is key: For engineering, a model that is 90% accurate but always 90% accurate is better than one that is 95% accurate one day and 70% the next.

The authors have released all their questions and code for free, inviting everyone to try to build an AI that can actually think like an engineer, not just read like one.

1. Problem Statement

Engineering thermodynamics is a quantitative discipline where errors propagate through multi-step calculations (e.g., a 2% error in enthalpy can lead to a 10% error in cycle efficiency). Existing Large Language Model (LLM) benchmarks fail to adequately evaluate this domain because:

Lack of Coherence: General science benchmarks (e.g., GPQA, MMLU) scatter thermodynamics across broad categories without isolating it as a coherent engineering domain.
Limited Scope: Dedicated benchmarks like UTQA focus only on ideal gases and multiple-choice questions, while others (e.g., Loubet et al.) are limited to ideal fluids and lack real-fluid properties (water, refrigerants) or exergy analysis.
Guessing vs. Reasoning: Multiple-choice formats allow models to guess or pattern-match, whereas engineering thermodynamics requires retrieving accurate property data, applying conservation laws, and performing consistent multi-step algebraic reasoning.

THERMOQA addresses these gaps by providing a rigorous, open-ended benchmark that tests the full scope of an engineering thermodynamics course, from property lookups to complex cycle analysis with real working fluids.

2. Methodology

Benchmark Design

The dataset consists of 293 open-ended questions divided into three tiers of increasing complexity:

Tier 1: Property Lookups (110 Q): Models must report thermodynamic properties (enthalpy, entropy, volume, etc.) and phase regions given state specifications. It covers 8 categories including subcooled liquid, wet steam, supercritical fluids, and inverse lookups.
Tier 2: Component Analysis (101 Q): Models analyze specific devices (turbines, compressors, pumps, heat exchangers, etc.) with given inlet/outlet conditions and isentropic efficiency. Analysis depth varies: (A) Energy balance, (B) Energy + Entropy, (C) Energy + Entropy + Exergy.
Tier 3: Cycle Analysis (82 Q): Full cycle calculations for 10 variants (Rankine, Brayton, Vapor-Compression, Combined-Cycle Gas Turbine). These involve 15–60 numerical steps per problem.

Working Fluids: The benchmark tests four fluid contexts: Water (using IAPWS-IF97), R-134a (using Helmholtz EOS), Air (using NASA 7-coefficient polynomials for variable specific heat), and combined Air+Water systems.

Ground Truth Generation

Programmatic Computation: All 293 answers are generated programmatically using CoolProp 7.2.0 and NASA polynomials, cross-verified against NIST reference data (<0.01% deviation).
No Human Annotation: This eliminates human error and ensures reproducibility.
Scoring:
- Tier 1: Per-property scoring with ±2% relative tolerance or ±0.5 absolute tolerance.
- Tiers 2–3: Step-weighted scoring. The final score is a weighted average of correct steps, where weights ( $w_i$ ) reflect engineering importance (e.g., final efficiency values carry higher weights than intermediate lookups). This penalizes models that retrieve inputs correctly but fail the core engineering analysis.

Evaluation Protocol

Models: Six frontier LLMs were evaluated: Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro, Grok 4, DeepSeek-R1, and MiniMax M2.5.
Protocol: Zero-shot evaluation (no few-shot examples) with reasoning modes enabled. Each model ran three independent times per tier to measure consistency (standard deviation).
Extraction: A two-pass extraction pipeline (regex + LLM re-extraction) ensures that reasoning tokens are captured and formatted correctly for scoring.

3. Key Contributions

Tiered Benchmark Design: Demonstrates that property memorization (Tier 1) and multi-step reasoning (Tier 3) are separable skills, evidenced by ranking reshuffles across tiers.
Programmatic Ground Truth: The first benchmark to use CoolProp for all reference values, ensuring high-fidelity real-fluid physics without human bias.
Multi-Run Consistency Analysis: Introduces standard deviation ( $\sigma$ ) as a distinct evaluation axis, quantifying reasoning reliability (ranging from $\pm0.1\%$ to $\pm2.5\%$ ).
Natural Discriminators: Identifies specific subsets (Supercritical water, R-134a, Variable-cp Brayton cycles) that produce the widest performance spreads, offering diagnostic value for developers.
Open-Source Release: The dataset, code, and model responses are publicly available under CC-BY-4.0/MIT.

4. Key Results

Overall Performance

Leaderboard: Claude Opus 4.6 leads with a composite score of 94.1%, followed by GPT-5.4 (93.1%) and Gemini 3.1 Pro (92.5%). The performance spread between top and bottom models is 21.1 percentage points.
Cross-Tier Degradation: There is a significant drop in performance as complexity increases.
- Claude Opus 4.6 shows the most robust reasoning, degrading only 2.8 pp from Tier 1 to Tier 3.
- MiniMax M2.5 degrades drastically by 32.5 pp (85.2% $\to$ 52.7%), indicating it relies heavily on memorization rather than reasoning.

Critical Findings

Supercritical Water (Tier 1): This category is the strongest discriminator. Models memorize textbook steam tables but fail to interpolate near the critical point ( $T_c \approx 374^\circ C$ ). The performance spread here is 44.5 pp (GPT-5.4 at 89.5% vs. MiniMax at 45.0%).
Real Fluids (R-134a): All models struggle with refrigerants (44–63% accuracy) compared to water (75–98%), highlighting a training data bias toward steam tables.
Component Difficulties: Compressors are the hardest component (55–75% accuracy) due to the counter-intuitive division formula for isentropic efficiency ( $w_{in} = \Delta h / \eta$ ), which models frequently reverse. Pumps are near-trivial ( $\ge 97.5\%$ ).
Consistency: GPT-5.4 is the most consistent ( $\sigma = \pm0.1\%$ on Tier 3), while DeepSeek-R1 shows high variance ( $\sigma = \pm2.5\%$ on Tier 2).
Token Efficiency: Opus invests significantly more tokens (up to 53K) for marginal gains over Gemini (which uses <3K tokens), suggesting diminishing returns on "thinking" for certain models.

Error Patterns

Cascading Errors: Errors propagate strongly ( $\phi = 0.94$ correlation). A mistake in an isentropic exit state ( $h_{2s}$ ) invalidates all subsequent cycle calculations.
Missing vs. Wrong: The dominant error mode is "missing" (omitting steps) rather than "wrong" (calculating a value outside tolerance).
Systematic Failures: Models often apply constant- $c_p$ formulas even when variable specific heats are explicitly requested, or reverse efficiency formulas for compressors vs. turbines.

5. Significance and Implications

Separation of Skills: THERMOQA proves that high accuracy on property lookups does not guarantee engineering reasoning capability. This challenges the assumption that scaling laws alone solve complex domain reasoning.
Training Data Bias: The poor performance on R-134a and supercritical fluids reveals a gap in training data, which is critical given the ubiquity of refrigeration cycles in the real world.
Engineering Reliability: The introduction of consistency metrics ( $\sigma$ ) is vital for engineering applications where deterministic outputs are required. A model with 90% $\pm 0.1\%$ is more useful than one with 91% $\pm 2.5\%$ .
Future Directions: The authors suggest future work should include tool-augmented evaluation (allowing access to CoolProp) to isolate reasoning errors from retrieval failures, and multimodal evaluation for interpreting T-s and P-h diagrams.

In conclusion, THERMOQA establishes a new standard for evaluating LLMs in quantitative engineering domains, highlighting that while frontier models excel at memorization, they still struggle with the nonlinear interpolation and multi-step logical consistency required for real-world thermodynamic design.