Validating Large Language Model Extraction of Actuarial Variables from Unstructured Claims Documents

Technical Summary: Validating Large Language Model Extraction of Actuarial Variables from Unstructured Claims Documents

Problem Statement
Actuarial functions such as reserving, ratemaking, and claims management have historically relied on structured numerical data. However, the claims lifecycle generates vast amounts of unstructured text (medical records, adjuster notes, transcripts) containing predictive information that remains largely untapped due to the high cost, inconsistency, and lack of scalability of manual processing. While Large Language Models (LLMs) offer a solution for handling varied formats and contextual disambiguation without extensive engineering, the literature lacks rigorous validation frameworks for extracting structured actuarial variables from these documents. Specifically, there is no established standard for how to validate such extraction or what reliability thresholds are appropriate for deployment in high-stakes actuarial applications.

Methodology
The authors conducted a formal validation study of a two-stage LLM pipeline designed to extract 14 core actuarial variables (e.g., claim severity, injury severity, litigation risk) from workers' compensation claims.

• Extraction Pipeline:
  • Stage 1 (Document-Level): Processes individual documents (medical records, notes, etc.) using tailored prompts. Outputs include a mandatory rationale with evidence citation, a rationale type (verbatim vs. derived), and a confidence score [0, 1]. This enforces structured chain-of-thought reasoning.
  • Stage 2 (Claim-Level Synthesis): Aggregates Stage 1 extractions using compound scoring that weights document authority (e.g., settlement notes weighted higher than phone transcripts), confidence scores, temporal decay, and variable-specific importance. Outputs are constrained to controlled vocabularies.
• Data: Validation utilized 20 synthetic claims generated from Synthea FHIR bundles. These bundles provided known ground-truth clinical characteristics, which were augmented with additional document types via LLM generation. This approach ensured shareable, privacy-compliant data with known truth values.
• Validation Protocol: Adapted from the QUEST framework (Tam et al.), two independent clinical reviewers (PharmDs) scored the LLM output on 20 claims across 10 dimensions using a 5-point Likert scale. Dimensions covered Quality/Accuracy, Safety/Ethics, and Understanding/Reasoning.
  • Disagreements of two or more points triggered a third-reviewer adjudication.
  • Claims with three or more divergent dimensions underwent full adjudication by the third reviewer blind to the primary scores.
• Analysis: Inter-rater reliability was measured using quadratic weighted Cohen's kappa (κw) on 200 paired assessments. The study also analyzed exact agreement rates, divergence proportions, and rater asymmetry in adjudicated items.

Key Contributions

1. Pipeline Architecture: Description of a two-stage extraction pipeline with design features (mandatory rationales, confidence scoring, controlled vocabularies) intended to reduce hallucination risk and create regulatory audit trails.
2. Validation Protocol: A formal, dimension-level validation framework adapted for actuarial variable extraction, reporting inter-rater reliability across specific rubric dimensions and principles.
3. Rater Asymmetry Characterization: Identification of systematic scoring asymmetry between reviewers, a practical feature of validation design often overlooked in actuarial literature.

Results

• Overall Reliability: The overall quadratic weighted kappa was 0.53, indicating moderate agreement. Exact agreement occurred on 51.5% of items, while 19.0% of items diverged by two or more points.
• Dimensional Variation:
  • Strong Agreement: Dimensions regarding Relevance (75% exact) and Appropriateness (45% exact) showed strong alignment, though some kappa values were zero due to range restriction (reviewers consistently scoring high).
  • Weak Agreement: Dimensions assessing failures (Missing Information, Contextual Understanding, Hallucination) showed the highest divergence. Missing Information had a negative kappa (−0.08), indicating agreement below chance, with 40% of items diverging by two or more points. Hallucination had the lowest exact agreement (20%) among substantive dimensions.
• Rater Asymmetry: Reviewer 1 consistently scored lower than Reviewer 2. In adjudicated items, Reviewer 1 was further from the adjudicated score (mean absolute distance 1.87 vs. 0.80) and scored below the adjudicated value in 12 of 15 items, suggesting a systematic conservative bias rather than random error.
• Confidence Scores: The pipeline's self-reported confidence scores were uniformly high (interquartile range 0.85–0.90) regardless of variable type, limiting their utility as a triage mechanism for distinguishing uncertain extractions.

Significance and Claims
The paper concludes that current LLM extraction capabilities are sufficiently reliable for triage applications (e.g., flagging claims for elevated human review) where the cost of an incorrect flag is low. However, the authors assert that direct input into reserve models or loss development calculations warrants additional validation due to weaker performance on omission and hallucination dimensions.

The study emphasizes that the observed rater asymmetry implies that calibration exercises using anchor claims with pre-specified scores must precede formal validation scoring in future studies. Furthermore, the restricted range of confidence scores suggests a need for post-hoc calibration to create effective filtering thresholds. The authors position this work as a replicable template for future actuarial LLM validation, applicable to various lines of business beyond workers' compensation.