Self-Reported Confidence of Large Language Models in Gastroenterology: Analysis of Commercial, Open-Source, and Quantized Models

1. Problem Statement

Large Language Models (LLMs) hold significant promise for clinical decision support but are prone to hallucinations, often presenting incorrect information with high confidence. In high-stakes medical domains like gastroenterology, this "overconfidence" poses severe safety risks.

• The Core Challenge: Current methods for uncertainty quantification (e.g., token-level probability, semantic difference assessments, and ML-based surrogates) are often computationally expensive, require specialized datasets, or lack generalizability.
• The Specific Gap: While "confidence elicitation" (asking the model to self-report its certainty) is a practical, zero-shot approach, previous research suggests LLMs fail to accurately represent their own uncertainty. There is a lack of systematic evaluation regarding how different model architectures (commercial vs. open-source), sizes, and quantization levels affect this self-reported confidence, particularly in specialized medical fields.

2. Methodology

The study conducted a systematic evaluation of 48 LLMs (including 6 quantized and 42 base models) across seven families: Qwen, Llama, Phi, Gemini, Gemma, Claude, and GPT.

• Dataset: The researchers utilized the 2022 American College of Gastroenterology (ACG) self-assessment examination, comprising 300 board-style multiple-choice questions.
  • Rationale: These questions are behind a paywall, minimizing the likelihood of data leakage from training sets. They cover diverse topics (liver, colon, endoscopy, etc.) and vary in difficulty based on historical test-taker performance.
• Experimental Design:
  • Prompt Engineering: Models were prompted to select an answer and provide a self-reported confidence score on a scale of 0 (least confident) to 10 (most confident).
  • Optimization: Parameters (temperature, max tokens, prompt structure) were optimized using a pilot set to maximize answer accuracy.
  • Extraction: A semi-automated pipeline using regex and LLM-based extraction (GPT-4o) was used to parse confidence scores from unstructured text. Human verification ensured 99% accuracy in extraction.
• Evaluation Metrics:
  • Accuracy: Percentage of correct answers.
  • Discrimination: Measured by AUROC (Area Under the Receiver Operating Characteristic curve), assessing the model's ability to distinguish between correct and incorrect answers based on confidence scores.
  • Calibration: Measured by Brier Score (mean squared error between predicted confidence and actual accuracy) and Expected Calibration Error (ECE). Lower values indicate better calibration.
  • Analysis: Correlations were analyzed between confidence and question difficulty/length.

3. Key Contributions

• Comprehensive Benchmark: This is one of the first studies to systematically compare commercial, open-source, and quantized LLMs specifically for confidence elicitation in a specialized medical subspecialty (Gastroenterology).
• Quantification of Overconfidence: The study provides empirical evidence that despite architectural advancements, LLMs consistently exhibit systematic overconfidence, where reported confidence scores significantly exceed actual accuracy.
• Model-Specific Insights: It identifies that newer, larger models (e.g., GPT-o1, Claude-3.5) show improved calibration but still fail to meet clinical safety thresholds. It also highlights that reasoning models (like GPT-o1) often truncate confidence scores due to token limits.
• Methodological Framework: The paper establishes a robust pipeline for extracting and analyzing self-reported confidence scores, including handling missing data and token truncation issues.

4. Key Results

• Performance vs. Confidence:
  • Overconfidence: All models reported confidence scores (mean range: 7.99–9.58) that were significantly higher than their actual accuracy (mean range: 30%–81%).
  • Top Performers: The highest accuracy was achieved by GPT-o1 preview (81.5%), followed by GPT-4o and Claude-3.5-Sonnet. However, even these top models were overconfident.
• Calibration and Discrimination:
  • AUROC: The best discrimination score was 0.626 (GPT-o1 mini), well below the 0.7 threshold often considered acceptable for clinical utility. No model achieved an AUROC > 0.7.
  • Brier Score: Only five models demonstrated Brier scores better than random guessing: GPT-o1-preview (0.157), Claude-3.5-Sonnet (0.202), GPT-4o (0.206), and two versions of Claude-3-Opus.
  • ECE: GPT-o1-preview had the lowest ECE (0.100), indicating the best calibration, though still imperfect.
• Impact of Difficulty and Length:
  • As question difficulty increased (lower percentage of human test-takers answering correctly), model accuracy decreased, but confidence scores remained stable. This indicates models do not dynamically lower their confidence when facing harder questions.
  • Question length had no significant impact on confidence scores.
• Model Families: Newer generations (e.g., Llama 3.3, GPT-o1, Claude 3.5) generally showed better calibration than older versions, suggesting that training data may be slowly incorporating uncertainty estimation skills, but the improvement is insufficient for clinical deployment.

5. Significance and Conclusion

• Clinical Safety Warning: The study concludes that self-reported confidence is currently insufficient for high-stakes medical applications. The persistent overconfidence and poor discrimination (AUROC < 0.7) mean clinicians cannot rely on a model's "certainty" to filter out hallucinations.
• Future Directions:
  • Targeted Training: The authors suggest that "emergent" skills in uncertainty quantification can be improved through specific training protocols (e.g., few-shot injection, in-context learning) rather than relying on general pre-training.
  • Methodological Shift: Relying solely on self-reporting is risky. Future frameworks should combine self-reported confidence with external verification methods or hybrid approaches (e.g., using BERT-based models for verification).
• Broader Implication: While LLMs are becoming more accurate, their ability to "know what they don't know" has not kept pace. Until models can accurately calibrate their uncertainty, their integration into clinical workflows requires rigorous human-in-the-loop oversight.

In summary: While newer LLMs are more accurate, they remain dangerously overconfident in medical contexts. The "self-reported confidence" metric, as currently implemented, fails to provide a reliable safety mechanism for clinical decision support.