💬 NLP

Investigating LLM's Problem Solving Capability -- a Study on Statics Questions

This study evaluates Large Language Models' capabilities in solving statics problems through a model distillation approach, revealing that while they perform well on text-only questions, their accuracy significantly declines when diagrams and multi-step reasoning are required due to challenges in consistently applying visual information rather than image recognition limitations.

Original authors: Tanner Culleton, Hung-Fu Chang

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tanner Culleton, Hung-Fu Chang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read student named "ChatGPT" who has read almost every book in the library. You want to test if this student can actually solve tough engineering homework problems, specifically in a subject called Statics (which deals with how forces balance on stationary objects, like bridges or cranes).

Here is what the researchers did, explained simply:

1. The "Copy-Paste" Problem

First, the researchers tried the obvious thing: they took real textbook questions and asked ChatGPT to solve them.

The Result: It was a disaster. The student got confused, especially when the problems involved 3D drawings. It would mix up directions (like saying "up" when it meant "right") and get the math wrong.
The Analogy: It was like asking a student to read a map written in a language they barely know. They knew the words, but they couldn't figure out the directions.

2. The "Reverse Engineering" Trick (Model Distillation)

Since the textbook questions were too hard, the researchers tried a clever trick called Model Distillation. Instead of asking the student to solve other people's questions, they asked the student to write its own questions.

The Logic: They figured, "If ChatGPT can write a good question, it probably knows the answer to that specific type of question because it 'learned' it from its own training data."
The Process: They asked ChatGPT to generate 50 statics problems. They noticed the student kept writing the same simple, 2D (flat) problems over and over. So, they picked the best 25 unique questions that ChatGPT seemed to understand well.

3. The Three Levels of the Test

The researchers created three different "exams" using those 25 questions to see how the student's brain worked:

Exam A (Text-Only): They asked ChatGPT to solve the questions it wrote, using only words.
- Result: Perfect Score (100%). The student knew the answers because it wrote the questions!
Exam B (With Pictures): They took those same word-only questions and drew simple diagrams for them.
- Result: Score dropped to 68%. The student could still read the words, but the picture threw it off. It struggled to connect the drawing to the math.
Exam C (The "New Numbers" Test): They took the diagrams from Exam B and changed all the numbers (e.g., changing a force from 10 lbs to 15 lbs).
- Result: Score dropped to 60%. This proved the student wasn't just memorizing the answers. It was trying to solve the problem fresh, but it kept making mistakes in the middle of the process.

4. What Did They Learn?

The researchers compared ChatGPT to another AI student named "Gemini."

The Big Discovery: The problem wasn't that the AI couldn't "see" the pictures. The problem was that it couldn't think in steps.
The Analogy: Imagine a chef who can read a recipe perfectly (Text-Only). If you hand them the ingredients and a picture of the dish (Diagram), they can still cook it. But if you ask them to cook a complex, multi-course meal where they have to adjust the recipe halfway through (Multi-step reasoning), they start dropping ingredients or forgetting the next step.
The Conclusion: The AI is great at simple, one-step math. But when it has to look at a picture, extract information, and then use that information in three or four different steps to get a final answer, it gets lost. It follows the right method but often gets the wrong number at the end.

Summary

The paper concludes that while AI is getting very good at reading and writing, it is still a bit clumsy when it comes to visual reasoning and long, complex chains of logic. It's like a student who knows the theory perfectly but freezes up when asked to apply it to a real-world drawing with changing numbers.

Note: The researchers only tested these specific engineering problems. They did not claim this applies to medical diagnoses, legal advice, or other fields, nor did they suggest how to fix the AI for the future. They simply reported how the AI performed on this specific set of engineering puzzles.

Technical Summary: Investigating LLM's Problem Solving Capability – A Study on Statics Questions

Problem Statement
Large Language Models (LLMs) have rapidly influenced education, demonstrating the ability to complete assignments and examinations across various subjects. However, existing research regarding their educational impact often relies on public or open problem datasets and lacks topic-specific analysis, particularly within mechanical engineering. There is a limited systematic investigation into how LLMs perform on specific problem types, such as statics, and whether their capabilities have evolved sufficiently to warrant updated educational assessments. Traditional evaluation methods, which involve directly prompting LLMs with textbook questions, have shown limitations, particularly with three-dimensional problems and diagram interpretation.

Methodology
To address these gaps, the authors adopted a model distillation-inspired approach rather than traditional direct testing. The study proceeded through the following phases:

Initial Assessment & Pivot: Initial tests using textbook questions from "McGraw Hill Connect" revealed that LLMs (specifically ChatGPT) struggled significantly with three-dimensional statics problems, often misassigning vector components between coordinate axes. Furthermore, when prompted to generate statics problems, the models produced repetitive, mostly two-dimensional, and straightforward questions.
Dataset Construction via Distillation: Recognizing that LLMs perform better on problems aligned with their internal training distribution, the researchers prompted ChatGPT to generate a large set of statics problems. Through an iterative process of generation and comparison, they filtered out repetitive instances to create a final Text-only Dataset of 25 unique, two-dimensional statics problems. These were categorized by difficulty:
- Conceptual (Easy): 8 questions testing definitions and qualitative reasoning.
- Direct Calculation (Medium): 7 questions requiring formula application.
- Multi-Step Application (Hard): 10 questions requiring multi-step reasoning and strategic setup.
Dataset Expansion: To evaluate visual reasoning and generalization, two additional datasets were constructed from the text-only set:
- Diagram-based Dataset: The text descriptions were manually converted into clean 2D diagrams using the Notability app.
- Value-Changed Diagram-based Dataset: The diagrams from the previous set were modified with new numerical values for forces, moments, and distances to test if the model was recalling solutions or genuinely reasoning.
Evaluation: The models (ChatGPT-5 and Gemini) were tested on all three datasets. Performance was measured by accuracy ( $A = N_c / 25$ ), defined as the number of correctly solved questions.

Key Contributions

Novel Evaluation Framework: The study introduces a model distillation perspective for evaluating LLMs in engineering education. Instead of testing models on external benchmarks where they may lack exposure, the study evaluates them on problems they likely generated or encountered during training, providing a clearer view of their internal reasoning capabilities.
Granular Performance Analysis: The research dissects LLM performance not just by overall score, but by problem type (Conceptual, Calculation, Multi-Step) and input modality (Text, Diagram, Value-Modified).
Identification of Reasoning Bottlenecks: The study isolates specific failure modes, distinguishing between image recognition limitations and multi-step reasoning deficits.

Results
The experimental results highlight a significant performance degradation as problem complexity and modality change:

Text-only Dataset: ChatGPT-5 achieved 100% accuracy, while Gemini achieved 60%. Notably, Gemini failed to solve any Multi-Step Application problems (0% accuracy).
Diagram-based Dataset: ChatGPT-5's accuracy dropped to 68%. While it maintained 100% accuracy on Conceptual questions, performance fell to 71.43% on Direct Calculation and 40% on Multi-Step Application.
Value-Changed Diagram-based Dataset: ChatGPT-5 achieved 60% accuracy. Performance on Multi-Step Application improved slightly to 60%, but Direct Calculation dropped to 42.8%.
Error Analysis: Across all tests, both models consistently followed correct solution procedures and methodologies. However, errors frequently occurred in the final numerical calculations or in the consistent application of extracted visual information across successive reasoning stages.

Significance and Claims
The paper claims that the observed decline in LLM performance when moving from text-only to diagram-based and multi-step problems is not primarily caused by limitations in image recognition. Instead, the authors conclude that the performance drop stems from difficulties in multi-step reasoning and the inability to consistently apply extracted visual information throughout the solution process.

The study suggests that while LLMs can approach near-expert performance on structured, text-based statics problems, they remain unreliable for complex, multi-step engineering tasks that require rigorous logical consistency. The authors posit that this insight is crucial for educators and curriculum designers to adapt instructional strategies and assessments, ensuring they promote conceptual understanding and higher-order thinking while mitigating inappropriate reliance on AI tools. The study acknowledges limitations, including the potential bias of using model-generated questions for evaluation and the modest dataset size (25 questions), but asserts that the findings provide a necessary baseline for understanding current LLM capabilities in mechanical engineering education.

1. The "Copy-Paste" Problem

2. The "Reverse Engineering" Trick (Model Distillation)

3. The Three Levels of the Test

4. What Did They Learn?

Summary

More like this