Superstudent intelligence in thermodynamics

Technical Summary: Superstudent Intelligence in Thermodynamics

Problem Statement
Thermodynamics is traditionally regarded as a rigorous academic hurdle in engineering education, characterized by high failure rates and the requirement for students to creatively combine fundamental principles rather than rely on pattern recognition. Historically, achieving a top grade (Grade A) in such exams has been considered proof of exceptional human intellectual capability. The core problem addressed in this study is the rapid evolution of Large Language Models (LLMs) and their ability to solve complex, non-routine engineering problems that were previously thought to be beyond the reach of artificial intelligence. Specifically, the authors investigate whether a state-of-the-art reasoning model can outperform human students in a standard university thermodynamics exam without prior training or specialized prompt engineering.

Methodology
The study employed a zero-shot prompting approach, utilizing OpenAI's reasoning model, o3, released in early 2025. The methodology involved the following steps:

• Test Material: The authors used the exact same engineering thermodynamics exam administered to 90 students at RPTU Kaiserslautern in Spring 2025. Unlike previous studies that used simplified, text-only problems tailored for LLMs, this exam included graphical inputs and required graphical outputs.
• Prompting Strategy: No special prompt engineering was applied. A standard zero-shot prompt was used: "You are an expert in thermodynamics. Solve the exam given below... If you are asked for plots, draw them." The exam consisted of three distinct problems.
• Execution: The model was run three independent times. The model received the text of the problems sequentially (Problem 1, then 2, then 3).
• Evaluation: The answers generated by o3 were graded using the exact same rubric and criteria applied to the human students. The evaluation focused on problem analysis, formulation of applicable equations, combination of principles, and numerical solutions.

Key Results
The performance of the o3 model was described as "breathtaking" and marked a significant departure from previous LLM capabilities:

• Overall Performance: In zero-shot mode, o3 solved all problems correctly, achieving a score in the range of the best human scores observed in over 10,000 similar exams since 1985.
• Comparison to Students: The model outperformed all 90 students. While the human failure rate was 58% and only one student achieved a Grade A, o3 achieved a score superior to that single top student.
• Problem-Specific Breakdown:
  • Problem 1: Solved without errors.
  • Problem 2: Solved with high accuracy; minor point deductions were due to issues with graphical representation rather than thermodynamic logic.
  • Problem 3: The only major error occurred here, relating to the caloric properties of a constant-density fluid, a fairly special topic.
• Historical Context: Previous studies (Loubet et al., 2025) conducted in late 2024 showed that models like GPT-4o and Llama 3.1 scored below 50% on advanced thermodynamic problems. The jump to o3 represents a qualitative shift, with scores exceeding 90% on problems more demanding than those in the previous "advanced" test set.

Key Contributions

1. Empirical Evidence of a Turning Point: The paper provides empirical data demonstrating that machines can now excel in complex tasks (solving a difficult thermodynamics exam) that have historically served as a proxy for human intellectual superiority.
2. Benchmarking Against Human Peers: Unlike many AI benchmarks that compare models against each other, this study directly compares an AI model against a cohort of human students in a real-world academic setting, establishing a new baseline for machine intelligence in engineering.
3. Analysis of Reasoning Capabilities: The results suggest that the improvement from earlier reasoning models (o1) to o3 is driven by enhanced reasoning capabilities rather than just the nature of the problems, as the exam included graphical elements and required creative application of principles.

Significance and Claims
The authors frame these findings as a fundamental turning point in both engineering practice and education:

• Definition of Intelligence: The paper argues that if a machine can successfully imitate a "very bright student" in a rigorous exam, it should be considered intelligent, drawing a parallel to the Turing test. The authors describe the performance not merely as "sparks" of artificial general intelligence (AGI) but as a "blaze."
• Impact on Engineering Work: The authors predict a shift toward mixed teams of human engineers and "AI colleagues." They highlight challenges in task allocation, trust verification, and decision-making responsibility, noting that the rapid pace of AI improvement may outstrip the ability to fully evaluate collaboration strategies.
• Impact on Education: The study suggests that the profile of the future engineer must shift from operational excellence to fundamental understanding. Since machines can handle routine calculations and problem-solving, human education must focus on:
  • Prompting and asking the right questions.
  • Verifying and trusting AI outputs.
  • Applying results to specific practical situations.
  • Maintaining deep disciplinary knowledge to validate machine-generated solutions.
• Epistemological Implications: The success of the model forces a re-evaluation of what "understanding" means in engineering. If a machine can solve these problems, does it "understand" thermodynamics? The paper posits that this question will become a major issue for the future epistemology of engineering.

The authors conclude that while extrapolating from academic exams to real-world engineering tasks is bold, the results indicate that highly skilled intelligent machines will become integral to the field very rapidly, necessitating immediate strategic development for human-machine cooperation.