ClinConsensus: A Consensus-Based Benchmark for Evaluating Chinese Medical LLMs across Difficulty Levels

1. Problem Statement

While Large Language Models (LLMs) show promise in healthcare, existing medical benchmarks suffer from critical limitations:

• Static and Isolated: Most benchmarks (e.g., USMLE-style exams) rely on single-round, multiple-choice questions focusing on factual recall rather than complex, multi-step clinical reasoning.
• Lack of Longitudinal Context: They fail to capture the continuum of care (prevention $\to$ treatment $\to$ long-term follow-up) and the dynamic nature of real-world patient management.
• Cultural and Systemic Gaps: Existing Chinese benchmarks often lack alignment with the specific realities of the Chinese medical system, including reimbursement rules, family-mediated decision-making, and local clinical guidelines.
• Safety vs. Accuracy: High accuracy on static tests does not guarantee safety or usability in real-world scenarios, where models may hallucinate, overlook critical constraints, or provide inconsistent advice over time.

2. Methodology

A. Dataset Construction: ClinConsensus

The authors curated a high-quality, expert-validated dataset comprising 2,500 open-ended clinical cases.

• Scope: Covers 36 medical specialties and 12 common clinical task types (e.g., differential diagnosis, medication safety, long-term monitoring).
• Care Stages: Distributed across three stages: Treatment (74.56%), Long-term Management (21.84%), and Prevention (3.60%).
• Complexity Stratification: Cases are categorized into three difficulty levels (L1, L2, L3) based on the number of clinical tasks and specialties involved, with L3 requiring complex causal reasoning and individualized planning.
• Expert Curation: Developed by a multidisciplinary team of Chinese clinical experts using real-world, de-identified cases transformed into natural language narratives.
• Evaluation Rubrics: Each case is paired with 30 expert-defined binary criteria (rubrics) covering clinical validity, evidence-based practice, and safety. Additionally, 73 consensus-level criteria were established to guide rubric formulation.

B. Quality Control

A two-stage filtering process ensures difficulty and validity:

1. Difficulty Filtering: Cases where leading LLMs (DeepSeek-v3, GPT-5, Gemini-2.5 Pro) scored $\ge$ 50% on the rubrics were discarded to ensure the benchmark tests non-trivial reasoning.
2. Expert Auditing: Senior clinicians randomly audited 20% of remaining cases for clinical correctness, internal consistency, and adherence to guidelines.

C. Evaluation Framework

To evaluate complex, open-ended responses, the authors propose a Dual-Judge Framework:

1. LLM-as-Judge: A high-capability proprietary model (e.g., GPT-4.1/5.1) acts as a grader.
2. Trained Judge: A lightweight, locally deployable model (8B parameter, distilled via Supervised Fine-Tuning) trained on expert annotations to replicate high-fidelity grading at lower cost.

• Input-Output Contract: The judge receives the full conversation context and a single rubric item, outputting a structured JSON decision (criteria_met: true/false) with an explanation.

D. Metric: Clinically Applicable Consistency Score (CACS@k)

The authors introduce CACS@k to address the limitations of average accuracy in high-stakes domains.

• Concept: Instead of averaging scores, CACS@k focuses on the consistency of generating clinically usable responses.
• Mechanism: It calculates the truncated area under the reverse-empirical cumulative distribution function (ECDF) of rubric-hit scores, starting from a clinical threshold $k$.
• Threshold Calibration ($k=7$): The threshold is empirically calibrated to the average performance of expert physicians (attending doctors) on a sample set. A score of $k=7$ represents the minimum "critical mass" of salient clinical information required for a response to be considered usable, acknowledging that experts do not always exhaustively list every possible consideration.
• Formula:
  $$\text{CACS@}k = \frac{100}{|D|(N-k+1)} \sum_{i \in D} \max(0, s_i - k + 1)$$
  Where $s_i$ is the number of rubric hits for case $i$, $N=30$ is the total rubrics, and $k=7$.

3. Key Contributions

1. ClinConsensus Benchmark: The first comprehensive, Chinese-specific benchmark covering the full continuum of care with 2,500 complex, open-ended cases across 36 specialties.
2. Scalable Evaluation Methodology: A dual-judge framework combining high-capability LLMs with a distilled, locally deployable judge model, enabling reproducible and cost-effective evaluation.
3. Novel Metric (CACS@k): A threshold-calibrated metric that prioritizes the consistency of clinically actionable outputs over average correctness, better aligned with physician judgment.
4. Comprehensive Analysis: A rigorous evaluation of 15 state-of-the-art LLMs revealing significant heterogeneity in clinical capabilities.

4. Experimental Results

The authors evaluated 15 leading LLMs (including GPT-5.2, ERNIE-5.0, Kimi-K2, Qwen, etc.) using ClinConsensus.

• Overall Performance: Top models (ERNIE-5.0, GPT-5.2, Kimi-K2, Qwen-Plus) achieved comparable aggregate CACS@7 scores (approx. 38.0), but performance dropped significantly for lower-tier models (e.g., LLaMA-405B at 14.20).
• Task Heterogeneity:
  • Kimi-K2 excelled in reasoning-heavy tasks (Differential Diagnosis, Causal Reasoning) and information extraction.
  • ERNIE-5.0 led in Evidence Retrieval and Long-term Follow-up.
  • GPT-5.2 performed best in Patient Education.
• Care Stage Challenges: The Treatment stage was the most difficult (Mean CACS@7 = 30.16), significantly lower than Prevention (38.43) and Long-term Management (37.17). This highlights that clinically actionable treatment planning remains a major bottleneck.
• Specialty Variance: Performance varied widely by specialty. Sports Medicine and Emergency Medicine saw high scores, while Organ Transplantation and Psychiatry showed the lowest performance, indicating domain-specific weaknesses.
• Trustworthiness: The automated graders (LLM-as-Judge and SFT-Judge) showed high agreement with human physicians (Macro-F1 $\approx$ 0.76–0.81), and the CACS@k rankings remained stable across different judges.

5. Significance and Conclusion

• Bridging the Gap: The study demonstrates that strong general language modeling capabilities do not automatically translate to safe, usable clinical behavior. Models often fail in longitudinal reasoning and complex treatment planning.
• Real-World Relevance: By incorporating Chinese-specific constraints (reimbursement, cultural context) and a continuum-of-care structure, ClinConsensus offers a more realistic assessment of AI readiness for clinical deployment.
• Future Direction: The benchmark serves as an extensible foundation for developing medical LLMs that are robust, clinically grounded, and capable of handling the safety-critical complexities of real-world healthcare. The authors plan to expand into multimodal and interactive evaluation settings.