💬 NLP

Claim Automation using Large Language Model

This paper demonstrates that a locally deployed, Low-Rank Adaptation (LoRA) fine-tuned Large Language Model significantly outperforms general-purpose models in generating accurate, structured corrective-action recommendations for insurance claims, offering a reliable and governable solution for automating decision-making in regulated domains.

Original authors: Zhengda Mo, Zhiyu Quan, Eli O'Donohue, Kaiwen Zhong

Published 2026-02-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhengda Mo, Zhiyu Quan, Eli O'Donohue, Kaiwen Zhong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you work at a massive insurance company. Every day, thousands of car owners call in with broken parts. They describe the problem in their own words: "My AC is blowing hot air and making a weird grinding noise," or "The engine light came on after I hit a pothole."

In the past, human experts (called claim adjusters) had to read these messy, unstructured stories and figure out exactly what repair to authorize. It was slow, tiring, and prone to human error.

This paper is about teaching a super-smart AI robot to do that first step for them. But here's the catch: they didn't just ask the robot to "guess." They built a specialized, secure, and highly trained version of the robot specifically for this job.

Here is the story of how they did it, explained with some everyday analogies.

1. The Problem: The "Generalist" vs. The "Specialist"

Imagine you hire a brilliant, world-famous chef (a General Large Language Model) to fix a specific, weird mechanical problem with a 1998 Toyota.

The Chef's Strength: They know how to cook anything. They can write a poem, explain quantum physics, or make a perfect soufflé.
The Chef's Weakness: If you ask them to fix a carburetor, they might give you a recipe for a sandwich instead. Or they might suggest a repair that sounds logical but is actually dangerous for that specific car. They also might talk too much, giving you a 10-page essay when you just need a 3-word instruction.

In the insurance world, using these "Generalist" AI chefs is risky. They might hallucinate (make things up), violate privacy rules by sending data to the cloud, or just give answers that don't fit the strict rules of the insurance company.

2. The Solution: The "Apprentice" Training

The authors decided to stop hiring the Generalist chef and instead train a Specialist Apprentice.

The Classroom: They took a powerful AI model (called DeepSeek-R1) and put it in a "classroom" filled with 2 million real car repair records.
The Lesson: They didn't just show the AI the questions; they showed it the answers that real human mechanics wrote. They taught the AI: "When a customer says 'grinding noise,' the correct answer isn't 'buy a new car,' it's 'replace the brake pads.'"
The Technique (LoRA): Imagine the AI is a giant encyclopedia. You can't rewrite the whole encyclopedia (that's too expensive and slow). Instead, they used a technique called LoRA (Low-Rank Adaptation). Think of this as adding a sticky note or a custom index to the encyclopedia. The AI keeps all its general knowledge, but now it has a special, tiny cheat sheet that tells it exactly how to speak "Insurance Mechanic."

3. The Rules: The "Strict Foreman"

One of the biggest problems with AI is that it can be chatty and unpredictable. If an AI writes a paragraph when the computer system needs a single code, the whole system breaks.

The authors built a Governance Framework (a set of strict rules):

Local Deployment: The AI lives on the insurance company's own servers, not on the open internet. It's like keeping the recipe book in a locked safe in the kitchen, not posting it on a public blog. This keeps customer data private.
Structured Output: They trained the AI to act like a robotic foreman. It doesn't get to write a story; it must fill out a specific form. If the form isn't filled out perfectly, the AI knows it failed.

4. The Results: From "Maybe" to "Yes"

They tested four different versions of the AI:

The Raw AI: Just the base model. It was messy, often refused to follow rules, and gave wrong answers. (Like the Generalist Chef trying to fix a car).
The Prompted AI: They tried to tell the AI what to do using a very strict prompt (like shouting instructions). It got slightly better but still made mistakes and didn't follow the format.
The Instruction-Tuned AI: A version trained to follow orders better. It followed the format 86% of the time, but still made errors in the actual repair advice.
The Fine-Tuned Specialist (The Winner): This is the one trained on the 2 million real claims.
- Format: It followed the rules 100% of the time.
- Accuracy: It guessed the correct repair 92% of the time (on clear cases).

The Analogy:

The Prompted AI is like a tourist who bought a phrasebook. They can ask for directions, but they might get the accent wrong or miss a nuance.
The Fine-Tuned AI is like a local taxi driver who has driven those specific streets for 20 years. They know exactly where to go, they know the traffic rules, and they get you there without asking for a map.

5. Why This Matters

This paper proves that for high-stakes jobs like insurance, you can't just "plug and play" a generic AI. You have to specialize it.

By training the AI on real-world data and forcing it to follow strict rules, they created a tool that:

Speeds up work: Adjusters don't have to read the whole story; the AI gives them the answer immediately.
Saves money: Fewer mistakes mean fewer bad repairs paid for.
Keeps secrets: The data never leaves the company's building.

In a nutshell: They took a smart but scattered AI, gave it a massive amount of specific training data, put it in a secure room, and taught it to fill out a specific form. The result is a reliable, super-fast assistant that helps insurance companies fix cars faster and more accurately.

1. Problem Statement

The paper addresses the limitations of applying Large Language Models (LLMs) to regulated, data-sensitive domains like insurance.

The Gap: Traditional actuarial modeling relies heavily on structured data, ignoring unstructured narrative text (e.g., claim descriptions, diagnostic notes) which contains crucial causal reasoning and operational intent.
Limitations of Current Approaches:
- Text-to-Feature Paradigm: Traditional NLP methods compress text into engineered numerical features, losing semantic nuance and pragmatic cues.
- General-Purpose LLMs: Off-the-shelf, API-based LLMs suffer from domain misalignment, lack of transparency (black-box inference), data privacy risks, and governance issues. They often produce "plausible" but operationally incorrect outputs or fail to adhere to strict schema requirements needed for automation.
The Goal: To develop a locally deployed, governance-aware LLM framework that automates the generation of structured corrective-action recommendations from unstructured warranty claim narratives, ensuring data privacy, auditability, and alignment with real-world operational logic.

2. Methodology

A. Framework Architecture

The authors propose a modular, intermediate-task automation framework with four core design principles:

Local Deployment: Uses an open-source model (DeepSeek-R1-Distill-Llama-8B) deployed on-premises to ensure data sovereignty and reproducibility.
Domain Adaptation: Fine-tunes the foundation model using millions of historical warranty claims to align internal representations with specific insurance terminology and action taxonomies.
Modular Intermediate Task: Instead of an end-to-end financial prediction (black box), the model performs a specific task: generating structured corrective-action summaries from unstructured "Complaint" and "Cause" narratives. This isolates linguistic understanding from downstream financial modeling.
Multi-Dimensional Evaluation: Combines automated semantic metrics with human evaluation to assess both practical utility and predictive accuracy.

B. Mathematical Formulation & Model

Task: Formulated as a conditional language generation problem $P(y|x)$ , where $x$ is the complaint/cause narrative and $y$ is the corrective action.
Architecture: Based on the DeepSeek-R1 decoder-only Transformer architecture, featuring:
- RoPE (Rotary Position Embeddings): For handling sequence order.
- PreNorm + RMSNorm: For training stability.
- SwiGLU Activation: In the Feed-Forward Network (FFN).
Fine-Tuning Strategy (LoRA):
- Uses Low-Rank Adaptation (LoRA) to inject trainable low-rank matrices ( $\Delta W = BA$ ) into the frozen weights of the pre-trained model.
- Placement: Applied to Query, Value, Output, Gate, Up, and Down projection matrices in both Attention and FFN layers.
- Training Objective: Minimized masked negative log-likelihood on the "Correction" tokens only, ignoring the input instruction tokens during loss calculation.
- Hyperparameters: Rank $r=32$ , scaling $\alpha=32$ , AdamW optimizer, FP16 mixed precision.

C. Dataset

Source: A proprietary corpus of ~2.0 million real-world automotive warranty claims from PCMI Corporation.
Structure: Each claim includes free-text narratives (Complaint, Cause, Correction) and structured metadata (VIN, make, model).
Preprocessing: A deterministic pipeline was used to handle noise, including removing duplicates, normalizing abbreviations (e.g., "r&r"), correcting misspellings, and filtering administrative text.

D. Evaluation Framework

The study employs a dual-dimensional evaluation strategy:

Usability:
- Format Compliance: Percentage of outputs strictly adhering to the predefined schema.
- Response Validity: Percentage of outputs containing actionable, parsable information.
Accuracy:
- Evaluated only on valid outputs.
- Metrics: A combination of automated metrics (Edit distance, BLEU, BERTScore, BLEURT, LLM-as-a-Judge) and Human Evaluation (Gold Standard).
- Human Scoring: Annotators rated semantic alignment on a 0.0–1.0 scale (Perfect Match to No Match).
- Correlation: Automated metrics were validated against human judgments using Chatterjee's $\xi$ , Spearman's $\rho$ , and Kendall's $\tau$ . BERT Cosine Similarity and LLM-as-a-Judge showed the highest correlation with human judgment.

3. Key Contributions

Governance-Aware LLM Workflow: Demonstrates a viable path for integrating LLMs into regulated industries by combining local deployment with domain-specific fine-tuning, addressing privacy and auditability concerns.
Superiority of Domain Fine-Tuning: Provides empirical evidence that fine-tuning on domain data fundamentally reshapes model behavior, outperforming both zero-shot prompting and instruction-tuned general-purpose models.
Novel Evaluation Taxonomy: Proposes a rigorous evaluation framework that separates usability (structural controllability) from accuracy (semantic correctness), highlighting that high accuracy is meaningless without format compliance in automation pipelines.
Distributional Realignment: Theoretically and empirically shows that fine-tuning minimizes the KL divergence between the model's output distribution and the real-world operational data distribution ( $P_{data}$ ), correcting the "plausible but incorrect" bias of general models.

4. Key Results

A. Usability (Format & Validity)

M1 (Base Model): 0% format compliance, 8% validity. (Unusable for automation).
M2 (Base + Prompting): 6.5% format compliance, 29.5% validity. (Prompting alone is insufficient).
M3 (Instruction-Tuned Qwen): 86.5% format compliance, 97.5% validity. (Better, but still has structural failures).
M4 (Fine-Tuned DeepSeek-R1): 100% format compliance, 100% validity.
- Significance: Only the fine-tuned model achieved deterministic structural control, a prerequisite for automated pipelines.

B. Accuracy (Semantic Correctness)

Non-Fine-Tuned Models: Clustered between 56% and 64% accuracy on valid subsets. They rely on generic internet knowledge, leading to "plausible" but operationally wrong actions.
Fine-Tuned Model (M4): Achieved 81.5% accuracy on the full set and 92.0% on the High-Quality (HQ) subset.
Comparison: The fine-tuned model significantly outperformed commercial general-purpose LLMs (even those with larger parameter counts), proving that domain alignment is more critical than model scale for specialized tasks.

C. Distributional Shift

Analysis of BERT cosine similarity distributions showed a systematic rightward shift for the fine-tuned model, indicating outputs are semantically closer to expert human references than any baseline.

5. Significance and Future Work

Practical Impact: The study validates that LLMs can be reliable building blocks for insurance automation if deployed locally and fine-tuned, moving beyond the "black box" of API-based solutions.
Theoretical Insight: It confirms that prompt engineering cannot override the underlying probability distribution of a base model; only fine-tuning can realign the model with specific operational logic and rare-event distributions.
Future Directions:
- Modular Integration: Expanding the framework to include downstream tasks like labor/parts estimation (using RAG) and reserving (using neural networks).
- Human-in-the-Loop: Addressing residual data quality issues (e.g., ambiguous narratives) by incorporating practitioner feedback for continuous online learning and targeted curation.

In conclusion, the paper argues that for high-stakes, regulated domains like insurance, domain-adaptive fine-tuning is not just an optimization but a necessity to achieve the structural reliability and semantic accuracy required for production-grade automation.