From Guidelines to Guarantees: A Graph-Based Evaluation Harness for Domain-Specific Evaluation of LLMs

1. Problem Statement

The paper addresses the critical limitations of current Large Language Model (LLM) evaluation in specialized domains, particularly healthcare.

• The Coverage Gap: Existing benchmarks (often manually curated MCQA datasets) fail to provide comprehensive coverage of specific, complex clinical guidelines. They are resource-intensive to create and often miss nuanced relationships within the text.
• Contamination Risks: Static datasets are vulnerable to "surface-form contamination" (models memorizing exact questions during training) and "relationship-level contamination" (models memorizing facts from source documents without understanding the logic).
• Scalability & Validity: Manual curation does not scale with evolving guidelines, and post-hoc review of generated questions is often insufficient to guarantee clinical accuracy.

2. Methodology

The authors propose a Graph-Based Evaluation Harness that transforms structured clinical guidelines into a dynamic, queryable knowledge graph. The framework operates in three main stages:

A. Graph Construction

• Source Material: The authors used the WHO IMCI (Integrated Management of Childhood Illness) handbook, an 80-page document containing flowcharts and checklists.
• Schema Design: The graph consists of five node types and four edge types:
  • Nodes: Condition, Symptom, Treatment, FollowUp, and Severity.
  • Edges: INDICATES (Symptom→Condition), TREAT (Condition→Treatment), FOLLOW (Condition→FollowUp), and TRIAGE (Condition→Severity).
• Curator Role: Crucially, the graph was manually curated by a clinical expert (a board-certified physician with 15+ years of experience). Automated extraction (PDF parsers/LLMs) failed to capture the conditional logic of visual flowcharts. This manual curation ensures the graph itself is the "source of truth," inheriting validity from the expert rather than relying on post-generation review.

B. Dynamic Query Instantiation

Instead of using a static dataset, the system generates evaluation questions on-demand via graph traversal:

• Traversal Logic: For every "Condition" node, the system traverses connected nodes to instantiate five specific question types (e.g., Condition → Symptom, Symptom → Condition, Condition → Treatment).
• Combinatorial Variation: To ensure robustness and prevent memorization, the system varies:
  • Templates: Multiple phrasing templates per relationship type.
  • Parameters: Randomized patient ages (constrained to valid ranges for the specific condition).
  • Distractors: An age-stratified sampling algorithm selects incorrect options (distractors) from conditions sharing the same age range, ensuring clinical plausibility while maintaining variability.

C. Contamination Resistance

The framework mitigates contamination through two mechanisms:

1. Surface-Form Resistance: By generating questions dynamically with randomized ages, distractors, and templates, the probability of a model encountering the exact same question during training is minimized.
2. Temporal/Versioned Evaluation: Because the graph is a structured representation, the harness can be re-run against updated guidelines. This allows evaluators to test if models have learned generalizable clinical reasoning or merely memorized relationships from a specific version of a guideline.

3. Key Contributions

1. Graph-Based Harness: Introduction of a framework that guarantees complete coverage of guideline relationships, contamination resistance via combinatorial variation, and validity inherited from expert-constructed graph structures.
2. Transformation Method: A methodology for converting structured clinical flowcharts into a knowledge graph that supports systematic, automated evaluation.
3. Dynamic Evaluation: A shift from static datasets to on-demand query instantiation, enabling continuous regeneration of evaluation data as guidelines evolve.
4. Empirical Insights: Demonstration that aggregate benchmarks obscure systematic weaknesses in clinical reasoning that this granular approach reveals.

4. Experimental Results

The framework was applied to evaluate five LLMs (three closed-source frontier models: Claude Sonnet 4.6, o4-mini, GPT-5.2; and two open/domain models: GPT-OSS-20B, MedGemma-4B) on 432 generated questions.

• Overall Performance: Frontier closed-source models achieved 66–68% accuracy, outperforming open-weight models (57%) and the domain-finetuned MedGemma-4B (50%).
• Granular Capability Gaps:
  • Strengths: All models performed best on Symptom → Condition questions (64–82%), indicating strong recognition of clinical indicators.
  • Weaknesses: Models struggled significantly with Treatment and Follow-up protocols (often <60%), revealing gaps in applying clinical management logic.
• Template Sensitivity: A template ablation study showed that phrasing significantly impacts performance. For example, specific follow-up templates yielded accuracy as low as 14% for some models, while others reached 90%. This highlights the necessity of using multiple templates to avoid phrasing artifacts skewing results.
• Model Scale vs. Domain Tuning: Surprisingly, the domain-finetuned MedGemma-4B underperformed larger general models, suggesting that in this specific setting, model scale and general reasoning capacity dominate over specialized fine-tuning.

5. Significance and Future Implications

• Scalable Infrastructure: The approach shifts the labor burden from authoring thousands of individual questions to a one-time graph construction effort. Once the graph is built, it can generate infinite variations of valid evaluation instances.
• Accountability & Separation of Concerns: The framework clearly separates roles:
  • Graph Constructors (Domain Experts) ensure clinical validity.
  • Harness Operators (Engineers) manage the generation pipeline.
  • Model Developers consume results without needing access to the underlying graph, enabling third-party auditing.
• Generalizability: While tested on WHO IMCI, the schema is applicable to any domain with structured decision logic (e.g., legal compliance, regulatory guidelines, other medical protocols like Malaria or HIV guidelines).
• Limitations: The system relies heavily on the accuracy of the initial manual graph construction (single-expert bias) and currently evaluates only MCQA, which may not capture the full complexity of real-world differential diagnosis or visual diagnostic elements.

Conclusion: The paper presents a paradigm shift from static benchmarking to dynamic, graph-grounded evaluation. By anchoring evaluation in expert-verified knowledge graphs, it provides a scalable, contamination-resistant, and clinically valid foundation for assessing LLMs in high-stakes domains.