Evidence-grounded AI for longitudinal clinical management in musculoskeletal care

Technical Summary: Evidence-Grounded AI for Longitudinal Clinical Management in Musculoskeletal Care

Problem Statement
Musculoskeletal diseases represent a leading cause of global disability, characterized by recovery, remodeling, and degeneration processes that unfold over months to years. Consequently, musculoskeletal care is fundamentally a longitudinal management problem rather than a series of isolated diagnostic or therapeutic decisions. Effective management requires clinicians to continuously integrate evolving patient evidence, external medical knowledge, and stage-specific functional goals. However, in routine practice, the evidence necessary for these decisions is often fragmented across visits, departments, and disparate hospital information systems. This fragmentation breaks continuous care pathways into isolated episodes, limiting individualized, evidence-based management.

Existing clinical Artificial Intelligence (AI) benchmarks and systems largely address bounded tasks, such as answering medical questions, conducting diagnostic dialogues, or selecting treatment options at a single point in time. These systems often fail to maintain consistent, traceable reasoning across successive clinical decision points, struggle to acquire missing information from realistic workflows, and lack the ability to anchor recommendations to real-time patient facts and external evidence simultaneously.

Methodology: The OrthoPilot System
To address these limitations, the authors present OrthoPilot, a hospital-deployable clinical AI agent designed for evidence-grounded longitudinal management. The system is built upon a 21-year longitudinal cohort of 183,472 patients from a high-volume tertiary academic hospital in Asia.

• Core Engine (CHEESE): At the system's core is the Clinical Holistic Evidence-to-Execution Synthesis Engine (CHEESE), a lightweight 32-billion-parameter Large Language Model (LLM). CHEESE is specialized for full-pathway reasoning and trained via reinforcement learning on expert care trajectories involving sequential decision-making tasks.
• Dual-Evidence Architecture (Tool Plaza): OrthoPilot operates on a "dual-evidence" paradigm linking two distinct worlds:
  1. In-Hospital Information World: Retrieves and organizes patient-specific multimodal evidence (X-ray, CT, MRI, laboratory results, pathology reports, medical orders, and consultation records) from real hospital information systems.
  2. External Knowledge World: Queries guidelines, PubMed literature, biomedical knowledge graphs, textbooks, and real-time web resources to ground recommendations in traceable, up-to-date medical evidence.
• Agent Architecture: The system employs a hierarchical multi-agent structure. A CHEESE-driven main agent defines reasoning strategies, detects missing evidence, and decomposes complex management questions into focused retrieval subtasks. Specialized sub-agents execute these tasks across the two evidence worlds, returning structured findings which the main agent synthesizes into evidence-grounded clinical recommendations. This allows the system to guide the care pathway, acquire key evidence at each decision point, and preserve reasoning consistency across the clinical timeline.

Benchmarking and Evaluation Framework
The authors developed OrthoBench, the first clinical-grade benchmark for decision-making across the musculoskeletal care continuum.

• Scope: Derived from 5,905 real patients, OrthoBench spans 1,000 disease codes and covers 11 tasks organized along the clinical timeline: four diagnostic reasoning tasks (Admission, Preoperative, Intraoperative, and Discharge diagnosis) and seven clinical management tasks (Perioperative assessment, Surgical planning, Orders, Discharge summary, Rehabilitation planning, and Multidisciplinary consultation).
• Stratification: The benchmark includes in-distribution (ID) and out-of-distribution (OOD) diseases, as well as rare and non-rare conditions, to test generalizability.
• Evaluation Metric (ORACLE): To evaluate open-ended clinical management tasks, the authors introduced Open Response Assessment for Clinical Language Evaluation (ORACLE). Unlike metrics based on lexical overlap, ORACLE decomposes clinical plans into physician-defined decision elements and scores the coverage of clinically meaningful evidence against physician-defined rubrics.

Key Results
The study conducted comprehensive retrospective and prospective evaluations:

1. Retrospective Benchmark Performance:

   • OrthoPilot significantly outperformed 22 baseline systems, including reasoning LLMs, general-purpose LLMs, medical domain-specific LLMs, and agent baselines.
   • On OrthoBench, OrthoPilot achieved 81.95% on closed-ended diagnostic tasks and 85.13% on open-ended management tasks, surpassing the next best baseline (CHEESE) by approximately 20% and 14%, respectively.
   • In external validation across 60 clinical centers (6,796 cases), OrthoPilot maintained its advantage, outperforming locally deployable open-source baselines in 57–59 centers for diagnostic tasks and 50–57 centers for management tasks.

2. Reader Study (Physician Comparison):

   • In a study involving 81 orthopaedic physicians (stratified by experience) and 307 patients, OrthoPilot alone achieved a mean overall score of 80.0%, exceeding Experts (65.4%), Specialists (59.5%), and Residents (56.2%).
   • When used as an assistant, OrthoPilot improved physician performance across all experience tiers, narrowing the performance gap between junior and senior physicians. AI-assisted Residents outperformed unassisted Experts by 7.3%.

3. Prospective Crossover Study:

   • In a study of 1,870 complex cases, OrthoPilot assistance increased full-chain management success from 16.6% to 27.2%.
   • Diagnostic accuracy improved significantly, particularly for admission diagnosis (rising from 65.7% to 77.8%), with the most marked improvements observed in junior physicians.
   • Surgical planning quality, rated by a blinded expert committee, shifted toward higher-quality categories, with the proportion of high-quality plans increasing from 25.0% to 34.0%.

4. Prospective Randomized Deployment:

   • In an 8-month randomized deployment involving 8,240 inpatients, OrthoPilot was embedded in routine hospital care.
   • Efficiency: The system increased cumulative cases per bed by 9.7% and significantly reduced the average length of stay (ALOS).
   • Workload: It reduced physician cognitive workload (mental demand, effort, frustration) and nursing documentation time.
   • Patient Experience: Patient-reported access to health information, communication quality, and overall satisfaction improved significantly.

Significance and Claims
The paper claims that OrthoPilot establishes a new paradigm for clinical AI, moving from models that predict isolated events to systems that execute longitudinal management across complete care pathways. The authors argue that the system's success stems from its ability to couple a clinically specialized reasoning backbone with hierarchical tool use, allowing it to acquire missing evidence and integrate it into management decisions dynamically.

Key contributions highlighted include:

• The demonstration that evidence-grounded sequential reasoning can translate retrospective gains into routine clinical workflows.
• The validation that such systems can improve decision-making robustness, reduce experience-dependent variation among physicians, and enhance system-level efficiency without compromising human-centered care aspects.
• The introduction of OrthoBench and ORACLE as necessary frameworks for evaluating full-pathway clinical management, moving beyond single-time-point performance metrics.

The authors conclude that while the system is not intended to store all medical knowledge in model parameters, its purpose is to acquire relevant evidence at run time and translate it into patient-specific next steps under clinician review, offering a governed design principle for AI in longitudinal care.