Comparison of the Diagnostic Reliability of ChatGPT in Letournel–Judet Acetabulum Fracture Classification Based on Judet Radiographs According to Orthopaedic Residents

Technical Summary: Diagnostic Reliability of ChatGPT-4o in Letournel–Judet Acetabular Fracture Classification

Problem Statement
Acetabular fractures represent complex orthopedic injuries requiring precise radiographic evaluation for surgical planning. The Letournel–Judet classification system is the standard framework for these injuries, yet it is known for moderate interobserver variability even among experienced trauma surgeons. While Artificial Intelligence (AI) and Large Language Models (LLMs) have demonstrated success in detecting simpler fractures and basic radiographic features, their efficacy in navigating the sophisticated, multi-step spatial integration required for complex acetabular fracture classification remains unexplored. This study addresses the gap in literature regarding whether current multimodal AI models can reliably serve as decision-support tools for this specific, high-complexity task.

Methodology
This retrospective observational study evaluated 184 anonymized cases of acetabular fractures treated surgically between May 2022 and December 2026 at Ankara Bilkent City Hospital.

• Data Source: Standard Judet radiographs (anteroposterior, iliac oblique, and obturator oblique views) were collected. Cases with non-acetabular fractures, incomplete imaging, or prior pelvic surgery were excluded.
• AI Evaluation: ChatGPT-4o was utilized as the primary AI model. A structured prompting strategy was employed, uploading a systematic radiographic checklist article (Brandser and Marsh) to the model. The model was instructed to answer seven specific binary questions regarding radiographic landmarks (e.g., disruption of the ilioischial line, presence of a spur sign, fracture of the posterior wall) individually before synthesizing a final Letournel–Judet classification.
• Human Evaluation: Two fourth-year orthopedic residents independently classified the same radiographs without access to CT scans or intraoperative data.
• Reference Standard: An experienced trauma surgeon with over a decade of pelvic trauma experience served as the reference standard. Crucially, the reference standard had access to multiplanar CT scans and intraoperative records, whereas the AI and residents were blinded to this advanced data.
• Statistical Analysis: Diagnostic accuracy was calculated against the reference standard. Interobserver agreement was measured using Cohen's kappa coefficient, and performance differences were analyzed using McNemar's test.

Key Results
The study yielded stark contrasts between human performance and the AI model:

• Overall Diagnostic Accuracy: ChatGPT-4o achieved a final classification accuracy of only 17.9% (33/184 cases). In comparison, the two orthopedic residents achieved accuracies of 88.6% and 91.8%, respectively.
• Agreement Metrics:
  • Residents showed "almost perfect" agreement with the reference standard (Kappa = 0.860 and 0.900).
  • ChatGPT-4o demonstrated "negligible" agreement with the reference standard (Kappa = -0.014) and only "slight" agreement with the residents (Kappa = 0.095–0.112).
  • The difference in performance between the residents and the AI was statistically significant (p < 0.001).
• Landmark Identification vs. Synthesis: While ChatGPT-4o showed moderate accuracy in identifying isolated landmarks (e.g., iliac wing fracture at 82.1%, spur sign at 86.4%), its overall checklist accuracy was 67.9%. The model struggled significantly with complex integration tasks, such as determining if a fracture divided the acetabulum into top/bottom or front/back halves (56.0% accuracy) and detecting posterior wall fractures (52.7% accuracy).
• Error Patterns: Confusion matrix analysis revealed a systematic bias where ChatGPT-4o frequently misclassified various fracture types (including posterior wall, posterior column, and T-type fractures) as "both-column" fractures. Notably, the model failed to correctly classify any true posterior column or anterior wall–column fractures.

Significance and Claims
The paper claims that current multimodal AI models, specifically ChatGPT-4o, are fundamentally inadequate for classifying complex acetabular fractures using the Letournel–Judet system. The authors assert that the model's failure is not merely a lack of visual recognition of individual landmarks but a failure in "hierarchical decision synthesis"—the inability to integrate sequentially acquired binary data into a coherent, global fracture pattern.

The study concludes that the use of this AI model as a diagnostic or decision-support tool in clinical practice for pelvic trauma is definitively not recommended due to its unacceptably low accuracy and inability to reliably integrate multiple radiographic landmarks compared to clinical residents. The authors suggest that while these tools show promise for future applications, particularly in training, they currently lack the spatial and anatomical synthesis capabilities required for complex trauma classification. The paper emphasizes that until these systems evolve to master such multi-step tasks, expert clinical oversight remains the only reliable standard.