💻 computer science

Extended Reasoning Mode Improves Large Language Model Accuracy on the Orthopaedic In-Training Examination: A Paired Within-Model Comparison

This study demonstrates that utilizing extended-reasoning inference significantly improves GPT-5's accuracy on Orthopaedic In-Training Examination questions compared to standard mode, though the persistence of confident errors suggests these models should serve as supervised adjuncts rather than primary study resources for residents.

Original authors: Claire A. Donnelley, Stephanie Kaszuba, Peter Noback, Xuan Luo

Published 2026-09-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Claire A. Donnelley, Stephanie Kaszuba, Peter Noback, Xuan Luo

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every year, thousands of orthopaedic surgery residents in the United States take a rigorous exam to measure their knowledge of bones, joints, and muscles. This test, known as the Orthopaedic In-Training Examination, is a critical checkpoint in their training, helping program directors decide if a trainee is ready to become a board-certified surgeon. In recent years, these students have increasingly turned to artificial intelligence tools to help them study. These tools, called large language models, are computer programs trained on vast amounts of text that can answer questions, explain complex ideas, and even mimic the style of a medical textbook. They have become so capable that they can often pass medical exams themselves. However, a crucial question remained unanswered: does the way a student asks the computer to think change the quality of the answer? Specifically, does forcing the computer to pause and reason through a problem step-by-step before speaking produce better results than asking it to answer immediately?

A team of researchers set out to find the answer by putting a single, advanced artificial intelligence model through a series of tests using the actual questions from the 2024 orthopaedic exam. They did not compare different brands of computer programs against each other. Instead, they used the same program twice for every single question. First, they asked the model to answer in its standard mode, where it generates a response quickly. Then, they asked the exact same model the exact same question, but this time they switched it to an "extended reasoning" mode. In this second setting, the computer is instructed to spend more time internally breaking down the problem, weighing evidence, and thinking through the logic before it ever types out a final answer. The researchers wanted to see if this extra mental effort, which takes about a minute longer per question, would actually make the computer smarter.

The results were striking. When the model answered in the standard, quick mode, it got 79 percent of the questions right. This score was already impressive, placing the computer's performance roughly on par with a senior resident who has been in training for five years. But when the researchers switched the model to the extended reasoning mode, the accuracy jumped to 93 percent. This was not a case of the computer getting lucky on some questions and unlucky on others. The data showed a clear pattern: every question the computer got right in the fast mode, it also got right in the slow mode. The improvement came entirely from the questions it had previously missed; in the extended mode, it corrected nearly all of its mistakes. The researchers found that this boost in performance happened across almost every area of orthopaedic surgery, from hand injuries to spinal conditions, and it did not matter if the question included a picture of an X-ray or was just text.

Despite this dramatic improvement, the study uncovered a subtle but important warning for anyone using these tools. Even when the computer was wrong, it sounded just as confident as when it was right. In the few instances where the model still gave an incorrect answer, even after thinking deeply, it did not hesitate or express doubt. It would provide a detailed explanation and even cite medical literature to support its wrong choice, making the error sound authoritative. The researchers noted that if a student does not already know the correct answer, they might be easily misled by this false confidence. The computer is also capable of being tricked; if a user suggests a wrong idea, the model will often accept it and build a confident, detailed explanation around that mistake.

The study concludes that while these artificial intelligence tools are powerful, they are not perfect replacements for human teachers or verified study materials. The researchers suggest that for students using these tools, the best approach is to always use the extended reasoning setting, as the extra minute of thinking time significantly increases the chance of a correct answer. However, the output should always be treated as a draft that needs verification. The computer is a useful assistant for organizing study plans or summarizing concepts, but it cannot yet be trusted to judge its own accuracy. The findings show that the way we interact with these machines matters just as much as the machines themselves; a simple switch in settings can turn a good answer into a great one, but it cannot eliminate the need for human oversight.

Technical Summary: Extended Reasoning Mode Improves LLM Accuracy on the Orthopaedic In-Training Examination

Problem Statement
Orthopaedic surgery residents increasingly utilize Large Language Models (LLMs) for board preparation and studying. While existing literature has established that contemporary multimodal LLMs can achieve accuracy levels comparable to senior residents on standardized medical examinations, a critical gap remains in understanding the impact of inference-time reasoning depth within a single model. Prior studies have largely compared different model vendors (e.g., GPT vs. Gemini) while holding inference conditions constant. However, they have not isolated the effect of enabling "extended-reasoning" (or "thinking") modes versus standard response generation on an identical item set. This distinction is educationally significant because a resident using a consumer subscription typically does not switch vendors mid-session but can toggle between standard and extended reasoning modes. The extent to which this toggle influences accuracy and the nature of persistent failure modes in unsupervised settings remained unquantified.

Methodology
This study employed a cross-sectional paired benchmarking design to evaluate a single multimodal LLM (GPT-5) on the publicly released 203-item subset of the 2024 Orthopaedic In-Training Examination (OITE).

Experimental Design: Each of the 203 items was administered twice to the same model via the native ChatGPT multimodal interface between November 2025 and January 2026. The two configurations tested were: (1) Standard Mode (immediate response generation) and (2) Extended-Reasoning Mode (allocating additional inference time for multi-step reasoning).
Controls: All other conditions were held constant. Items were presented as screenshots containing the question stem, figures, and multiple-choice options. Prompts were standardized to "Select the best answer and provide an explanation." Sessions were cleared between items to minimize context retention.
Scoring: Responses were scored against the official American Academy of Orthopaedic Surgeons (AAOS) answer key. Rationales were compared to official explanations to identify answer-rationale discordance.
Statistical Analysis: Paired accuracy was analyzed using McNemar's exact test. Subspecialty differences were assessed using the Fisher-Freeman-Halton exact test. Significance was defined as $p < 0.05$ .

Key Contributions

Quantification of Intra-Model Performance: This study is the first to isolate the effect of inference-time reasoning depth within a single LLM on a fixed, high-stakes medical examination dataset.
Characterization of Failure Modes: The research provides a detailed taxonomy of errors that persist even under extended reasoning, specifically analyzing "confident errors" where the model provides incorrect answers with high fluency and apparent evidentiary support.
Subspecialty and Modality Analysis: The study evaluates performance variance across orthopaedic subspecialties (e.g., hand, trauma, spine) and distinguishes between image-containing and text-only items.

Results

Accuracy Improvement: Extended-reasoning mode significantly improved accuracy from 79% (161/203) in standard mode to 93% (189/203) in extended-reasoning mode. This represents an absolute improvement of 13.8 percentage points (95% CI 9.0–18.5).
Directionality of Improvement: The improvement was strictly unidirectional; every item answered correctly in standard mode remained correct in extended-reasoning mode. All 28 discordant pairs favored the extended-reasoning configuration ( $p < 0.001$ ).
Subspecialty Performance:
- Significant improvements were observed in Hand (50% to 83%, $p=0.031$ ) and Trauma (78% to 92%, $p=0.016$ ).
- No significant differences were found across subspecialties in either configuration ( $p=0.11$ standard; $p=0.68$ extended).
- Accuracy ranged from 50% to 100% in standard mode and 83% to 100% in extended-reasoning mode.
Image vs. Text: Accuracy did not differ significantly between image-containing and text-only items in either configuration ( $p=1.0$ for both).
Failure Behavior:
- Confidence: Incorrect answers were delivered with the same fluency and structure as correct ones, often without expressed uncertainty.
- Hallucination: In 2% of items, the model provided an answer that conflicted with its own rationale.
- Citation Errors: Misapplied or non-concordant citations were the least likely error type to be corrected by extended reasoning.
- User Interaction: When presented with a corrective premise by an investigator, the model incorporated the correction but generated a new, equally confident explanation based on the (incorrect) premise.

Significance and Claims
The paper claims that inference-time reasoning depth is a material determinant of LLM accuracy on orthopaedic board-style questions, a setting that residents can control with a single toggle. The findings suggest that while extended reasoning substantially reduces error frequency, it does not eliminate the risk of "confident errors."

The authors conclude that LLMs should be viewed as supervised adjuncts rather than primary study resources. The central caution is that the tool is least reliable for the learners least equipped to detect its failures, as the model offers no internal signal distinguishing correct from incorrect outputs. Consequently, the authors recommend that residents default to extended-reasoning configurations for board preparation but must treat all model outputs as unverified until cross-referenced with validated question banks, primary literature, or faculty instruction. The study explicitly states that examination performance does not equate to clinical judgment.

Technical Summary: Extended Reasoning Mode Improves LLM Accuracy on the Orthopaedic In-Training Examination

More like this