Technical Summary: AI Consilium – A Benchmark of LLM Clinical Multidisciplinary Second Opinions

Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in generating differential diagnoses for complex clinical cases, a high-quality clinical second opinion requires more than diagnostic accuracy. It must be factually grounded, clinically coherent, actionable, and safe. Existing evaluations often rely solely on diagnostic agreement or treat multi-agent systems as monolithic "black boxes," failing to disentangle the specific contributions of model identity, response structure (free-form vs. structured), and the synthesis process itself. Furthermore, the assumption that multidisciplinary deliberation inherently improves outcomes lacks rigorous empirical validation, particularly when compared against compute-matched baselines and isolated role-specific opinions. There is a need for a benchmark that evaluates the full trajectory of a clinical second opinion—from independent role-specific analysis to multidisciplinary synthesis—while distinguishing between the quality of reasoning, the final conclusion, and the management plan.

Methodology

The authors developed a benchmark using 300 diagnostically resolved clinical cases from the NEJM–MedQA dataset (Case Records of the Massachusetts General Hospital). The study design involves a three-stage generation pipeline and a multi-dimensional evaluation framework.

1. Data and Pipeline Design

• Dataset: 300 unique cases were split into 110 cases for physician annotation and LLM Judge calibration, and 190 cases for held-out evaluation.
• Generation Pipeline:
  • Role Assignment: For each case, 1–5 clinical roles (e.g., Radiology, Endocrinology) are assigned, with one designated as primary.
  • Pre-Consilium Stage: Each role generates an independent second opinion under two conditions: Free-form (narrative) and Structured (schema-constrained with specific sections like syndromes, mechanisms, and management).
  • Interactive Consilium Stage: An interactive protocol synthesizes the role-specific opinions. Roles review a shared report iteratively (up to 5 rounds), voting "yes" or "no" with critiques. A moderator resolves dissent, and a finalizer produces a consolidated case-level report.
• Models: Eight distinct single-model pipelines were evaluated on the held-out set (DeepSeek-V4-Pro, Gemma-4-31B, GLM-5.2, GPT-5.6-Sol, GPT-OSS-120B, Qwen3.5-27B, Qwen3-235B, Qwen3-Next-80B). In the calibration set, a 3+1 procedure was used where three worker models generated opinions and a fourth (GLM-5.2) synthesized them.

2. Evaluation Framework

• Physician Reference: Six physicians with cross-disciplinary expertise rated the outputs using a 0–3 scale across eight response-quality criteria:
  1. Factual Correctness and Source Grounding
  2. Identification and Prioritization of Clinical Problems
  3. Completeness and Relevance of Data Use
  4. Coherence of Clinical Reasoning
  5. Quality of the Differential Diagnosis
  6. Quality of the Final Clinical Conclusion
  7. Justification and Specificity of Management
  8. Clinical Safety
  • Special Rule: Any fabricated clinically significant fact results in a score of 0 for factual grounding and caps related criteria at 1.
• Automated Judge: A physician-calibrated LLM Judge (GPT-OSS-120B) was optimized using evolutionary algorithms (GigaEvo) to minimize Mean Absolute Error (MAE) against physician ratings for each criterion independently.
• Metrics: The study analyzes model identity, response format, synthesis stage contributions, deliberation dynamics (consensus rounds), and inference costs (token usage).

Key Contributions

1. Dual-Checkpoint Benchmark: The first benchmark to evaluate standalone role-specific opinions and the final multidisciplinary synthesis as distinct clinical products on the same cases, allowing for the isolation of synthesis value.
2. Multidimensional Quality Framework: A physician-developed rubric separating factual grounding, reasoning, management, and safety, moving beyond simple diagnostic agreement.
3. Controlled Comparison: A design that independently varies model identity, response structure, and synthesis stages to disentangle their effects, including a comparison of compute-intensive multi-agent interaction against single-model baselines.
4. Calibrated Judge System: A criterion-wise calibrated LLM Judge that provides scalable evaluation of open-ended clinical second opinions, validated against physician ratings for specific dimensions.

Results

• Model Identity Dominance: Model identity was the strongest predictor of report quality. GPT-5.6-Sol led every evaluated component (role reasoning, role opinion, and final conclusions) in both free-form and structured formats. The performance gap between the best and worst models (approx. 0.4–0.5 points) significantly exceeded the differences caused by response format within a single model.
• Value of Synthesis: The complete structured synthesis achieved higher pooled Judge scores than the pre-consilium primary-role opinion across all shared criteria. The largest improvements were observed in problem prioritization, factual grounding, and reasoning coherence, indicating that the synthesis stage effectively integrates disparate perspectives into a more organized clinical account.
• Impact of Structure: Structured final reports showed higher overall means than free-form reports in seven of eight model pipelines. This advantage was driven primarily by a significant improvement in management (2.740 vs. 2.436), suggesting structured prompting enhances actionability and planning rather than uniformly improving all clinical dimensions.
• Deliberation Dynamics: The interactive protocol reached consensus in 97.8% of cases. However, longer review trajectories (more rounds) did not correlate with higher final scores. GPT-5.6-Sol achieved the highest quality scores with the fewest review rounds (averaging ~1.2 rounds), demonstrating that strong initial performance combined with rapid convergence is more effective than extended deliberation.
• Inference Costs: Structured pipelines required significantly more tokens (approx. 26.5k additional tokens per case) than free-form pipelines, highlighting a trade-off between the marginal gains in management quality and computational cost.

Significance and Claims

The paper claims that the AI Consilium benchmark successfully captures nuances of clinical second opinions that diagnostic accuracy alone misses. It demonstrates that:

1. Multidisciplinary synthesis adds value: The process of aggregating and revising role-specific opinions transforms isolated material into a higher-quality, integrated second opinion, particularly in prioritization and coherence.
2. Model choice is paramount: The identity of the underlying LLM is a more critical factor in determining the quality of a clinical second opinion than the specific response format or the complexity of the deliberation protocol.
3. Structure aids actionability: Structured prompting specifically improves the quality of management plans, making the output more actionable, though it comes with increased inference costs.
4. Efficiency of convergence: High-quality models can achieve consensus rapidly, suggesting that excessive deliberation rounds do not necessarily yield better clinical outcomes.

The authors conclude that while the benchmark provides a comprehensive framework for comparing future models and synthesis strategies, prospective studies are still necessary to establish the impact of these systems on actual clinician decision-making and patient outcomes. The study emphasizes evaluating clinical quality, procedural convergence, and inference cost as complementary outcomes rather than relying on diagnostic accuracy as a sole proxy.