Technical Summary: VETSCORE

Problem Statement

Large Language Models (LLMs) are increasingly applied in veterinary medicine, yet they frequently produce outputs that are potentially harmful or factually inaccurate. While recent approaches advocate for "attributable text generation" (including citations and source excerpts) to improve reliability, the mere presence of citations does not guarantee that the generated claims are faithful to the provided sources. Furthermore, existing fact verification methods typically treat all claims equally. In high-stakes domains like veterinary medicine, this is a critical flaw: an unverified claim regarding a drug dosage poses a significantly higher risk of patient harm than an unverified claim regarding the publication year of a study. Current evaluation metrics fail to distinguish between these levels of risk, potentially allowing high-risk hallucinations to pass undetected while penalizing low-risk inaccuracies disproportionately.

Methodology: VETSCORE

The authors present VETSCORE, a multi-step evaluation pipeline designed to assess the faithfulness of long-form veterinary QA outputs to their cited source excerpts, weighted by the potential harm of each claim. The method extends the standard "decompose-then-verify" paradigm with risk-adjusted scoring.

The pipeline consists of four distinct stages:

1. Output Segmentation: The generated text is split into segments based on delimiters (empty lines or previous citations). Each segment is associated with its corresponding source excerpts.
2. Claim Decomposition: An LLM decomposes each text segment into individual atomic claims. To ensure context is preserved (e.g., resolving anaphora), the decomposer is provided with the preceding text of the output up to the last header.
3. Dual-Scoring Mechanism:
   • Fact Verification: An LLM judge evaluates whether each atomic claim is supported by the provided source excerpts. This yields a binary score ($v_i \in \{0, 1\}$), where 1 indicates support and 0 indicates lack of support or contradiction.
   • Harm Potential Scoring: Independently, an LLM judge assigns a harm potential score ($w_i \in [1, 5]$) to each claim. This score reflects the severity of harm if the claim were incorrect and acted upon by a veterinary practitioner. The scale ranges from 1 (no clinical consequence, e.g., study metadata) to 5 (direct patient harm, e.g., incorrect drug dosage or contraindication).
4. Risk-Weighted Aggregation: The final score for a segment is calculated by aggregating the verification and harm scores. The authors define a local penalty based on the unverified potential, normalized by the total harm potential of the segment. The formula allows for controlling the contribution of high-harm claims via an exponent $p$ (set to 2.426 in experiments). The final risk-adjusted verification score is $1 - p_{global}$, representing the fraction of total harm potential that is supported.

Dataset and Evaluation

To validate VETSCORE, the authors constructed a meta-evaluation dataset:

• Inputs: 67 diverse queries from a production veterinary QA system, covering various disciplines (gastroenterology, cardiology, etc.) and species.
• Generation: Six different LLMs (including GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, and open-weight models) generated answers with citations based on retrieved PubMed sources.
• Human Annotation: Veterinary experts (practitioners and students) annotated 1,200 segments containing 4,986 claims. They performed two tasks: binary fact verification and 1–5 harm potential scoring.
• Aggregation: Due to moderate inter-annotator agreement on harm scoring ($\alpha = 0.462$), the authors used Item Response Theory (specifically the Partial Credit Model) to adjust for rater severity and derive aggregate scores.

Key Results

The authors evaluated VETSCORE using nine different judge models (ranging from proprietary models like GPT-5.6 Sol to smaller open-weight models like Gemma 4 31B) against the human annotations.

• Correlation with Experts: VETSCORE achieved high Spearman correlations with human experts. Fact verification correlations reached up to 0.783 (Gemini 3.6 Flash), and harm potential correlations reached 0.763 (Gemini 3.1 Pro).
• Model Efficiency: The modular approach proved effective even with smaller models. Gemma 4 31B (31B parameters) achieved competitive correlations with much larger proprietary models, suggesting the method does not require massive LLMs for effective evaluation.
• Ablation Studies: Removing the harm potential component (Verification-only baseline) significantly reduced the correlation with human combined scores in absolute terms, confirming that risk-weighting is essential for clinical relevance. Joint evaluation (combining verification and scoring in one prompt) degraded performance compared to the modular approach.
• Consistency: Proprietary models and Gemma 4 31B showed high consistency across runs, whereas some open-weight Mixture-of-Experts (MoE) models exhibited higher variance.

Significance and Contributions

The paper claims three primary contributions:

1. VETSCORE Pipeline: An efficient, explainable, and modular verification pipeline that provides risk-weighted faithfulness scores specifically for veterinary long-form QA. Unlike previous methods, it explicitly penalizes unverified high-risk claims more heavily than low-risk ones.
2. Expert-Validated Dataset: The release of a meta-evaluation dataset annotated by veterinary professionals, which demonstrates that the proposed approach correlates highly with human expert judgment.
3. Feasibility with Small Models: The demonstration that the modular decomposition and risk-weighting approach allows for effective evaluation using smaller, more efficient LLMs, making the system scalable.

The authors emphasize that VETSCORE addresses the specific need for "risk-informed" evaluation in clinical domains. By decomposing outputs and weighting claims by harm potential, the system offers fine-grained explainability, allowing practitioners to identify exactly which high-risk claims lack support. The paper concludes that while the method is currently focused on excerpt-based faithfulness, its risk-adjusted framework is a critical step toward safer, evidence-based AI in veterinary medicine.