Who Benefits From Sinus Surgery? Comparing Generative AI and Supervised Machine Learning for Predicting Surgical Outcomes in Chronic Rhinosinusitis

Technical Summary: Comparing Generative AI and Supervised Machine Learning for Predicting Surgical Outcomes in Chronic Rhinosinusitis

Problem Statement
Chronic Rhinosinusitis (CRS) is a heterogeneous condition where patient outcomes following endoscopic sinus surgery (ESS) vary significantly. While clinical guidelines support surgery for eligible patients, predicting which individuals will achieve a meaningful quality-of-life improvement remains challenging. The study addresses the need for pre-operative decision support to identify patients unlikely to achieve a Minimal Clinically Important Difference (MCID) in the Sino-Nasal Outcome Test-22 (SNOT-22), defined as a reduction of ≥8.9 points at six months. The core research question is whether models using only pre-operative clinical data can retrospectively identify patients who would have poor outcomes (i.e., those who should have avoided surgery), and how Generative AI (GenAI) systems compare to traditional supervised Machine Learning (ML) in this specific predictive task.

Methodology
The study utilized a prospectively collected, multi-center cohort of 524 patients who underwent ESS, derived from an NIH-funded study (NCT01332136). The dataset included 50 pre-operative attributes such as demographics, comorbidities, phenotype (CRSwNP vs. CRSsNP), objective disease burden (Lund–Mackay CT scores, endoscopic severity), and baseline SNOT-22 scores.

• Data Processing: Post-operative variables were strictly excluded to prevent data leakage. Features were harmonized, deterministically encoded, and missing values handled by removal. The dataset was split into an 80/20 stratified train-test set.
• Supervised ML Baselines: The authors evaluated five classifiers: Logistic Regression, Support Vector Machines (SVM), Naïve Bayes, Random Forest, and a custom Multi-Layer Perceptron (MLP). The MLP (one hidden layer, 400 neurons) was selected as the primary baseline due to its superior balance of accuracy and minority-class recall.
• Generative AI Benchmarking: Five state-of-the-art LLMs (ChatGPT/GPT-5 Thinking, MedGPT, Gemini 2.5 Pro, Perplexity Sonar, Claude Sonnet 4.5, and DeepSeek-V3.2) were tested in a zero-shot setting.
  • Protocol: Structured pre-operative data were serialized into prompts. Models were constrained to output a binary recommendation (0: surgery not recommended; 1: surgery recommended) and a confidence level.
  • Stability: Each case was run five times with fixed decoding parameters, and predictions were aggregated via majority vote.
  • RAG Analysis: A Retrieval-Augmented Generation (RAG) experiment was conducted with ChatGPT using a corpus of CRS guidelines to test if external knowledge improved performance.
• Evaluation Metrics: Performance was assessed using AUROC, F1 score, Average Precision (AP), calibration (Brier score, reliability curves), and decision-curve net benefit. Special attention was paid to the detection of the minority class (non-responders, Class 0).

Key Results

• Supervised ML Performance: The MLP achieved the highest overall performance with an accuracy of 85%, an AUROC of 0.66, and an F1 score of 0.83. Crucially, it demonstrated the best ability to identify non-responders (Class 0 recall of 0.45) while maintaining high precision for responders.
• GenAI Performance: All GenAI models underperformed the calibrated MLP in discrimination and calibration.
  • Discrimination: AUROC scores for GenAI models ranged from 0.36 (Perplexity) to 0.63 (Claude), all lower than the MLP's 0.66.
  • Bias: Most GenAI models exhibited a strong "treat-all" bias, failing to identify non-responders. For instance, MedGPT achieved 0% recall for the non-responder class (0/20), and Perplexity achieved 0% as well.
  • RAG Impact: Retrieval-Augmented Generation did not improve ChatGPT's discrimination (AUROC remained ~0.57) or minority-class detection, suggesting that guideline text reiterates heuristics already internalized by the models without adding patient-specific signal.
• Interpretability and Alignment: Despite lower predictive accuracy, the reasoning processes of the GenAI models aligned closely with clinical heuristics and the feature importance of the MLP. Both the LLMs and the MLP identified baseline SNOT-22, CT/endoscopy severity, polyp phenotype, and psychosocial comorbidities (e.g., depression, fibromyalgia) as critical predictors.

Key Contributions

1. First Comprehensive GenAI Benchmark: This work provides the first rigorous, head-to-head evaluation of multiple GenAI systems against tuned tabular ML models for predicting surgical outcomes in CRS using structured pre-operative data.
2. Standardized Evaluation Protocol: The authors established a reproducible framework for "tabular-to-GenAI" evaluation, including standardized zero-shot prompts, constrained binary outputs, and stability testing via multiple sampling runs.
3. Insight into GenAI Reasoning: The study reveals that while GenAI models lack the calibration and discrimination of supervised ML for this specific task, their emergent decision logic mirrors expert clinical reasoning and the feature importance of high-performing ML models.
4. Proposed Workflow: The paper proposes an "ML-first, GenAI-augmented" workflow. In this model, a calibrated tabular ML classifier performs the primary triage and risk estimation, while GenAI serves as a secondary layer to explain decisions in plain language, enhancing transparency and shared decision-making.

Significance and Claims
The paper claims that for structured, pre-operative prediction tasks involving clinical tabular data, calibrated supervised ML remains superior to current GenAI systems in terms of discrimination, calibration, and the ability to identify high-risk non-responders. The authors argue that GenAI should not replace ML for primary decision support in this context due to its instability and tendency toward over-recommendation. However, the study highlights the unique value of GenAI as an "explainer" that can translate complex model predictions into clinically familiar rationales, thereby supporting transparency and patient engagement. The findings suggest that the optimal path forward is a hybrid approach that leverages the reliability of ML for prediction and the communicative strengths of GenAI for explanation, rather than relying on GenAI as a standalone predictor.