🤖 AI

GuideSkill: Evolving Executable LLM Agent Skills for Guideline-Grounded Clinical Reasoning

GuideSkill introduces a model-agnostic framework that compiles clinical practice guidelines into executable diagnostic functions, which are then refined through case data to significantly enhance LLM accuracy and skill coverage in clinical reasoning without requiring backbone model updates.

Original authors: Lang Cao, Yuhao Shen, Tianyang Luo, Simo Du, Hao Peng, Yue Guo

Published 2026-07-30

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Lang Cao, Yuhao Shen, Tianyang Luo, Simo Du, Hao Peng, Yue Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive, complex mystery. You have a brilliant detective (a Large Language Model, or LLM) who is great at reading clues, guessing suspects, and telling a good story. But this detective sometimes gets distracted, misses the fine print, or forgets the strict rules of the law. On the other hand, you have a thick, dusty rulebook (Clinical Practice Guidelines) that contains the exact, step-by-step laws for solving every type of crime. The problem is, if you just hand the rulebook to the detective, they might read it but not actually follow the rules, or they might get confused by the long paragraphs. What if, instead of just reading the book, we could turn those rules into a set of magical, self-checking tools? Imagine giving the detective a special "Truth-O-Meter" for every suspect that instantly flashes green or red based on the evidence, rather than asking them to guess. This is the world of medical AI: trying to combine the creative, storytelling power of smart computer brains with the rigid, life-saving precision of medical rules.

The paper "GuideSkill" introduces a clever new way to do exactly this. The researchers built a system called GuideSkill, which acts like a translator that turns boring medical rulebooks into a library of "executable skills." Think of these skills as little computer programs or "checklists" that can be run automatically. Instead of the AI just reading a guideline and hoping it understands, GuideSkill compiles the rules into code that can actually check a patient's symptoms and spit out a score: "This evidence strongly supports Disease A," or "This evidence rules out Disease B."

The system works in two cool stages. First, there's GuideSkill-Zero. This is the "starter pack." The researchers took existing medical guidelines and turned them into these executable skills. It's like taking a cookbook and turning every recipe into a robot chef that can only cook if you give it the exact right ingredients. This alone was already better than just letting the AI read the guidelines. But the real magic happens in the second stage, GuideSkill-Evo. Here, the system learns from real patient stories. If the "starter pack" didn't have a rule for a rare disease, or if a rule was a bit too vague, the system looks at thousands of real cases, figures out what the experts actually did, and updates or adds new skills to the library. It's like the robot chefs going to culinary school, tasting real dishes, and learning new recipes that weren't in the original book.

When a new patient comes in, the system does a team-up. The AI detective first makes a list of possible suspects (a "differential diagnosis"). Then, the GuideSkill library runs its automated checks on each suspect. It combines the detective's gut feeling with the hard, mathematical scores from the skills to pick the winner. The results are impressive: on four different medical tests, this team-up was much more accurate than the AI working alone or just reading the guidelines. In fact, by adding the "learning from cases" step, the system covered 99.5% of the diseases it was tested on, compared to only 56.5% with the initial rules. Even better, a human doctor reviewed the new skills and said they were medically sound and trustworthy. The paper suggests that this approach—keeping the rules as separate, checkable tools rather than trying to memorize them inside the AI's brain—is a powerful way to make medical AI safer and more reliable, without needing to retrain the AI's entire brain every time a new rule is discovered.

Technical Summary: GuideSkill

Problem Statement

Clinical diagnosis is a multistep process requiring the synthesis of patient information, generation of differential diagnoses, and the application of evidence-based thresholds to support or refute candidates. While Clinical Practice Guidelines (CPGs) encode the necessary diagnostic criteria and conditional logic, Large Language Models (LLMs) typically fail to execute these rules correctly. Current approaches often rely on retrieving guideline text (RAG) or absorbing guidelines through training, but these methods do not guarantee that criterion-level evaluations are applied rigorously to specific patient cases. Furthermore, existing systems struggle to handle diagnoses outside the initial guideline corpus or to combine flexible candidate generation with explicit, disease-specific rule application without updating model parameters.

Methodology

The paper introduces GuideSkill, an external reasoning layer that transforms CPGs from passive references into an evolvable library of executable, disease-specific diagnostic skills. The framework operates in three stages:

1. Task Formulation

Given a patient case $x$ , the system predicts a final diagnosis $d^*$ . The process involves:

Candidate Generation: An LLM generates a ranked set of differential diagnoses $C(x)$ .
Skill Execution: For each candidate $d$ , a corresponding executable skill $g_d$ is retrieved from a library and executed on the patient case to produce an evidence score $s_{skill}(x, d)$ .
Fusion: The final score $S(x, d)$ is a weighted combination of the LLM's ranking score $s_{LLM}(x, d)$ and the skill-based evidence score:
$S(x, d) = \alpha s_{LLM}(x, d) + (1 - \alpha) s_{skill}(x, d)$
where $\alpha \in [0, 1]$ balances the two signals.

2. Skill Initialization (GuideSkill-Zero)

This stage constructs the initial skill library from curated clinical guidelines:

Curation: Raw guideline documents are filtered and curated to remove noise and non-actionable content.
Extraction & Compilation: Recommendations are extracted, grouped by ICD-10 category, and compiled by an LLM into executable Python functions.
Scoring Logic: Each skill maps patient features to one of four support tiers (Confirmed, Strongly Suggestive, Compatible, Not Supported), normalized to a score in $[0, 1]$ . Contradictory evidence can downgrade the support level.

3. Skill Evolution (GuideSkill-Evo)

To address the limitation that guidelines cover only a finite set of diagnoses, the library is evolved using labeled patient cases:

Rationale Generation: For training cases, the LLM generates diagnostic rationales summarizing the evidence supporting the ground-truth diagnosis.
Distillation & Optimization: Rationales for the same diagnosis are aggregated to distill new disease-specific rules. These rules are used to either optimize existing skills (if the diagnosis is already covered) or generate new skills (if the diagnosis is absent from the initial library).
Result: This process expands the library's coverage and refines the decision rules to better align with real-world heterogeneous presentations without modifying the LLM backbone parameters.

4. Inference

During inference, the LLM proposes a differential diagnosis. GuideSkill retrieves the relevant skills, extracts the specific features required by each skill from the patient narrative, executes the skills to obtain evidence scores, and fuses these scores with the LLM's ranking to determine the final prediction.

Key Contributions

Agentic Skill Execution: The paper formulates guideline-grounded diagnosis as the execution of agentic skills, developing a pipeline that compiles CPG recommendations into executable ordinal scorers indexed by disease.
Case-Conditioned Evolution: It introduces a mechanism to refine covered skills and add previously uncovered diagnoses using labeled cases, expanding the library's scope outside the LLM's parameters.
Comprehensive Evaluation: The authors evaluate the approach across four heterogeneous diagnostic-reasoning benchmarks and four distinct LLM backbones (proprietary and open-weight), separating the contributions of guideline initialization from the gains achieved through evolution.

Results

The evaluation spans four benchmarks (MedCaseReasoning, ER-Reason, MIMIC-CDM-FI, MedThink-Bench) and four backbones (GPT-5.4, Claude-Sonnet-4.6, MedGemma-27B, Qwen3.5-9B).

Performance Gains: GuideSkill-Zero consistently outperforms guideline RAG across all backbones. GuideSkill-Evo achieves the highest macro-average accuracy for every backbone, improving over direct inference by an average of 18.49% relative to the strongest baseline.
Coverage: Evolution increases gold-label skill coverage from 56.5% (GuideSkill-Zero) to 99.5% (GuideSkill-Evo), effectively filling gaps in the initial guideline corpus.
Generalization: On MedThink-Bench, a benchmark excluded from the evolution training set, GuideSkill-Evo achieves the best or tied-best accuracy across all backbones, demonstrating transferability to unseen data.
Comparison to Training-Based Methods: On Qwen3.5-9B, GuideSkill-Evo outperforms the strongest parameter-update baselines (including fine-tuning with guidelines, cases, and reinforcement learning) by 11.16% without updating the backbone.
Clinical Validity: Expert evaluation of 10 skills (5 guideline-initialized, 5 case-derived) indicated that 95.2% of initial rules were guideline-consistent, and 9/10 final skills were accepted with only minor revisions. The overall quadratic-weighted $\kappa$ between clinician judgments and skill outputs was 0.60.

Significance

The paper posits that GuideSkill demonstrates the viability of external executable skills as a model-agnostic mechanism for clinical reasoning. By externalizing diagnostic logic, the system allows for:

Reusability: The same diagnostic logic can be inspected, updated, and reused across different LLM backbones.
Transparency: Decision logic is explicit and interpretable, unlike the opaque weights of fine-tuned models.
Extensibility: The library can be expanded with new diagnoses and refined rules using labeled cases without the computational cost or data requirements of retraining the backbone model.

The authors conclude that while broader clinician validation is necessary before clinical deployment, the results support the use of executable skills to combine guideline-derived procedures with case-derived diagnostic patterns, offering a robust alternative to parameter-update methods for improving diagnostic accuracy and coverage.