Structured Workflows and Retrieval-Augmented Generation Improve LLM Recommendations for Colorectal Liver Metastases: The METATRON Study

Technical Summary: Structured Workflows and Retrieval-Augmented Generation Improve LLM Recommendations for Colorectal Liver Metastases (The METATRON Study)

Problem Statement

Large Language Models (LLMs) show promise for clinical decision support but face significant limitations in complex oncology, including hallucinations, incomplete guideline adherence, and inconsistent performance across scenarios. While recent studies suggest LLMs can align with expert opinions, systematic reviews indicate they struggle with autonomous deployment in multidisciplinary settings. Specifically, the management of Colorectal Liver Metastases (CRLM) requires integrating tumor biology, systemic therapy response, technical resectability, and extrahepatic disease status. Current LLM applications often lack the structured reasoning architecture necessary to reproduce the sequential, adaptive cognitive processes of expert hepatobiliary surgeons during Multidisciplinary Tumor Board (MDT) discussions. It remains unclear whether structured clinical workflows and retrieval-augmented generation (RAG) can improve the concordance of LLM-generated recommendations with expert reference standards in this high-stakes domain.

Methodology

The study evaluated four inference strategies built upon a single base LLM (gpt-oss-120B, deployed locally) using a dataset of 110 synthetic CRLM clinical vignettes. These vignettes covered eight representative clinical scenarios, including synchronous/metachronous metastases, post-chemotherapy reassessment, and anatomically challenging cases. Crucially, selected cases deliberately omitted decision-critical variables to test the models' ability to recognize insufficient information.

The four strategies compared were:

1. LLM-baseline: Standard prompting without additional guidance or retrieval.
2. RAG-LLM: Baseline prompting augmented with a vector database containing 292 documents from international guidelines (ESMO, NCCN, AHPBA) and peer-reviewed literature.
3. WF-LLM (Workflow-guided): A strategy incorporating an expert-derived, tree-structured clinical reasoning architecture. This workflow, encoded in JSON, formalizes the hepatobiliary surgeon's cognitive process into four domains: Primary tumor assessment, Hepatic disease assessment, Extrahepatic disease assessment, and Anatomical feasibility. It operates via a two-phase process: (1) structured case analysis extracting specific values or flagging "Missing" data, and (2) recommendation generation based on this structured analysis.
4. WF-RAG-LLM: A hybrid strategy combining the structured workflow analysis with retrieval-augmented context.

Evaluation Framework:
Recommendations were assessed by three independent LLM judges (Claude Sonnet 4.6, GPT-4o, Llama 3.3 70B) against expert reference recommendations generated by a senior hepatobiliary surgeon. Evaluation occurred across five dimensions on a 1–5 Likert scale:

• Completeness: Coverage of major clinical domains.
• Step Concordance: Reproduction of the sequence of clinical decisions.
• Case Tailoring Concordance: Incorporation of patient-specific factors.
• Missing-data Concordance: Identification of critical missing information.
• MDT Decision Concordance: Alignment with the final treatment decision.

Additionally, a zero-shot Named Entity Recognition (NER) pipeline measured entity-level lexical concordance, and confusion matrices analyzed anatomical resectability classification (compliant vs. non-compliant).

Key Contributions

• Expert-Derived Reasoning Architecture: The study introduces a workflow-guided approach that formalizes expert clinical knowledge into a structured, hierarchical reasoning framework (JSON schema). Unlike simple prompting, this architecture constrains the LLM to explicitly evaluate specific decision domains (e.g., FLR adequacy, vascular involvement) before synthesizing a recommendation, mirroring the adaptive nature of human MDT discussions.
• Differentiated Impact of RAG and Workflow: The study demonstrates that RAG and structured workflows address complementary cognitive functions. RAG primarily enhances the recognition of missing information, while workflow guidance improves the organization and coherence of clinical reasoning.
• Granular Evaluation: The research moves beyond global performance metrics by analyzing scenario-specific performance (e.g., post-chemotherapy vs. metachronous cases) and specific surgical tasks like anatomical resectability classification.

Results

• Overall Performance: The WF-RAG-LLM strategy achieved the highest aggregated composite score (0.626 ± 0.070), outperforming RAG-LLM (0.559), WF-LLM (0.527), and the baseline (0.444).
• Dimension-Specific Gains:
  • Missing-Data Concordance: RAG-LLM achieved the highest score (0.486) in this dimension, significantly outperforming the baseline (0.130). This suggests retrieval mechanisms help models identify when information is insufficient.
  • Reasoning Dimensions: Workflow-guided strategies (WF-LLM and WF-RAG-LLM) showed the largest gains in Step Concordance, Case Tailoring, and MDT Decision Concordance.
  • Completeness: All strategies achieved high completeness scores, indicating that modern LLMs can generate comprehensive text even without structural guidance; the differentiator was the organization of that information.
• Scenario Variability: Performance varied significantly by clinical scenario. Metachronous metastases showed the highest alignment, while post-chemotherapy metastases were the most challenging, particularly for step concordance. The hybrid WF-RAG strategy showed the most significant improvement in these complex scenarios.
• Anatomical Resectability: Workflow-guided strategies improved the detection of anatomically non-compliant cases (True Positives increased from 1 in baseline/RAG to 6 in WF-RAG), raising the macro F1-score from ~0.50 to ~0.64. However, sensitivity remained insufficient (40%) for autonomous clinical use.
• Statistical Significance: The Claude Sonnet 4.6 judge detected statistically significant differences across all criteria, favoring WF-RAG-LLM. GPT-4o found significance in step concordance, missing-data, and MDT decision criteria. Llama 3.3 showed lower sensitivity, finding significance only in missing-data concordance.

Significance and Claims

The paper claims that the performance of LLM-assisted decision support in complex oncology is not solely a function of the foundation model's capability but is critically dependent on the inference strategy and the clinical reasoning architecture surrounding it.

• Complementary Mechanisms: The study posits that clinically informed reasoning architectures (workflows) and knowledge grounding (RAG) are complementary. Workflows organize the reasoning process to match expert cognition, while RAG enhances the model's ability to detect informational uncertainty.
• Shift in Focus: The authors argue that future development should shift focus from merely scaling foundation models to explicitly modeling expert clinical reasoning. The workflow acts as a "reproducible and auditable" alternative to unconstrained prompting.
• Safety and Limitations: The study modestly concludes that while WF-RAG-LLM achieves the highest concordance, it is not yet ready for autonomous deployment. Sensitivity in detecting anatomical non-compliance remains low, and the study was simulation-based using synthetic vignettes.
• Future Direction: The authors emphasize that the next step is prospective validation within real MDTs to assess clinical utility, workflow efficiency, and patient outcomes, rather than immediate clinical implementation. The findings suggest that "clinically informed reasoning architectures" are a necessary component for reliable AI in multidisciplinary decision support.