Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation

This paper introduces the AEPC-QA benchmark to evaluate 51 large language models on Quebec insurance regulations, revealing that while inference-time reasoning and retrieval-augmented generation can significantly boost accuracy, the latter's tendency to cause context distraction and the superior performance of generalist models over specialized French fine-tuned ones highlight critical stability challenges for autonomous deployment.

The Gist

Imagine you are trying to buy car insurance in Quebec, Canada. Thanks to new laws, you can do it entirely online without talking to a human agent. But the insurance contracts are huge, confusing, and written in complex legal language. Most people are left trying to decipher them alone, creating a dangerous "advice gap."

This paper asks: Can Artificial Intelligence (AI) step in to act as a reliable, legal-savvy guide for these people?

The researchers from Laval University built a massive test to find out. Here is the story of their experiment, explained simply with some analogies.

1. The Exam: "The Quebec Insurance Gauntlet"

To test the AI, the researchers created a secret, high-stakes exam called AEPC-QA.

• The Source: They took 807 questions from official, paper-only study guides used to certify real human insurance agents. Because these guides are only on paper and not on the internet, the AI couldn't have "cheated" by memorizing them beforehand.
• The Difficulty: The questions are tricky. They don't just ask "What is insurance?" They ask complex scenarios like, "If a golfer gets hit by a ball, which legal defense is irrelevant?" It requires deep logic, not just guessing.

2. The Contestants: 51 Different AI Brains

They pitted 51 different Large Language Models (LLMs) against this exam. Think of these models as different types of students:

• The General Geniuses: Massive, smart models trained on everything (like a student who read every book in the library).
• The Specialists: Smaller models specifically trained on French or legal topics.
• The "Reasoners": A new type of AI that pauses to "think" step-by-step before answering, like a detective solving a puzzle.

They tested them in two ways:

1. Closed-Book: The AI has to answer from memory alone.
2. Open-Book (RAG): The AI is allowed to look up answers in a digital library of Quebec insurance laws before answering.

3. The Big Surprises (The Results)

🧠 Surprise #1: Thinking Beats Memorizing

The biggest winners weren't the models that had memorized the most laws. They were the "Reasoners" (like the o3 and o1 models).

• The Analogy: Imagine a student who memorized the dictionary vs. a student who knows how to solve a math problem. When the question is tricky, the one who knows how to think wins. The top AI scored nearly 79% correct, proving that logic is more important than memory for legal advice.

📚 Surprise #2: The "Knowledge Equalizer" vs. The "Distraction"

This is where it gets weird.

• The Equalizer: For some AI models that were "dumb" about Quebec law, giving them the digital library (RAG) was a miracle. Their scores jumped by over 35%. It's like giving a student a cheat sheet, and suddenly they ace the test.
• The Distraction: But for other very smart models, the library was a trap. When they tried to read the extra documents, they got confused, panicked, or started refusing to answer because the text looked "too legal" or "too risky."
  • The Metaphor: Imagine a brilliant chef who can cook a perfect steak. If you suddenly hand them a 50-page manual on "How to Cook a Steak" while they are cooking, they might get so distracted by the instructions that they burn the steak or refuse to cook at all. This is called "Context Distraction."

🇫🇷 Surprise #3: The "Specialist" Trap

You might think a model specifically trained on French insurance laws would win. It didn't.

• The massive, general-purpose models (which speak many languages and know many topics) crushed the small, specialized French models.
• The Lesson: Being a "specialist" in one narrow area didn't help as much as being a "generalist" with strong reasoning skills. The small French models were so focused on the words that they missed the logic.

4. The Verdict: Are We Ready?

The paper concludes that while AI is getting very good (approaching human expert levels), we aren't ready to let them run the show alone yet.

• The Risk: If an AI gives bad advice, the insurance company is legally liable (just like a human agent).
• The Problem: The "Context Distraction" issue means that sometimes, giving an AI more information makes it worse. This is dangerous for real-world use.
• The Solution: We need a "Human-in-the-Loop." AI should do the heavy lifting and suggest answers, but a human must double-check them, especially when the AI is confused by the extra documents.

Summary in One Sentence

AI is becoming smart enough to understand complex insurance laws, but it's currently too unstable to trust blindly; it needs a human supervisor to make sure it doesn't get distracted by its own library or give dangerous advice.

Technical Summary

Here is a detailed technical summary of the paper "Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation."

1. Problem Statement

The digitization of insurance distribution in Quebec, accelerated by Bill 141, has removed the requirement for human agents, creating a significant "advice gap." Consumers are left to interpret complex, multi-page financial contracts without professional guidance. While Large Language Models (LLMs) offer a scalable solution for automated advisory services, their deployment in high-stakes financial domains is hindered by the need for strict legal accuracy.

• The Challenge: Standard LLMs often produce "plausible" but legally incorrect answers (hallucinations), which can lead to coverage gaps, financial loss for policyholders, and legal liability for insurers.
• The Gap: Existing legal benchmarks and models are heavily biased toward Common Law jurisdictions (US/UK) and English languages. They lack the specific reasoning patterns required for Civil Law systems (like Quebec's Civil Code) and the Francophone context, rendering them unsuitable for this specific use case.

2. Methodology

A. The AEPC-QA Benchmark

To rigorously evaluate models, the authors introduced AEPC-QA, a private, gold-standard benchmark:

• Source: Derived from 25 official, print-only regulatory certification handbooks used by the Autorité des marchés financiers (AMF).
• Scale: 807 multiple-choice questions with four options and a single objectively correct answer.
• Characteristics: The dataset contains ~96,000 tokens in questions alone, with a readability score (Flesch-Kincaid) of 64.55, reflecting professional certification difficulty.
• Data Integrity: Because the source material is print-only and not indexed online, the risk of data contamination (models having seen the answers during pre-training) is minimal.

B. Experimental Setup

The study evaluated 51 LLMs across two distinct paradigms:

1. Closed-Book Generation: Assessing the model's intrinsic parametric knowledge of Quebec insurance law without external context.
2. Retrieval-Augmented Generation (RAG):
   • Corpus: The "Quebec Automobile Insurance Expertise Reference Corpus" (QAIERC), containing 168,600 sentences of official legislation and contracts.
   • Architecture: A dense retriever (text-embedding-ada-002) retrieves the top-5 relevant chunks, which are compressed to reduce noise before being injected into the LLM prompt.
   • Goal: To determine if external context can compensate for a lack of domain-specific pre-training.

C. Evaluation Protocol

• Metrics: Accuracy on multiple-choice questions.
• Validation: Stratified 10-fold cross-validation.
• Baselines: Five naive baselines (Random Sampling, Always A/B/C/D) were established to define the performance floor (~25% accuracy).
• Statistical Analysis: Z-tests were used to determine the statistical significance of performance differences between Closed-Book and RAG settings.

3. Key Contributions

1. AEPC-QA Benchmark: The first rigorous, private benchmark specifically for Quebec insurance regulation, ensuring alignment with industry standards and minimizing data contamination.
2. Comprehensive Benchmarking: A systematic evaluation of 51 models (proprietary and open-source) in a zero-shot setting, explicitly excluding models trained on Common Law/English data to ensure jurisdictional relevance.
3. Empirical Analysis of RAG: A deep dive into how RAG acts as a performance amplifier for some models while acting as a "context distraction" for others.

4. Key Results & Findings

A. The Supremacy of Inference-Time Reasoning

• Finding: Models utilizing Chain-of-Thought (CoT) reasoning at inference time significantly outperformed standard instruction-tuned models.
• Top Performers: o3-2025-04-16 (78.68% with RAG) and o1-2024-12-17 (75.18% with RAG) secured the top spots.
• Insight: The bottleneck in legal QA is not just knowledge retrieval but the application of logic to novel scenarios. Interestingly, even for French tasks, these models often perform intermediate reasoning in English before generating French output.

B. RAG as a "Knowledge Equalizer" vs. "Context Distraction"

• Equalizer Effect: RAG dramatically boosted models with weak parametric knowledge. For example, DeepSeek-reasoner jumped from 36.30% (Closed-Book) to 71.77% (RAG).
• Context Distraction: Conversely, high-performing models suffered catastrophic regressions when exposed to RAG.
  • Example: Gemini-2.5-pro dropped from 74.65% (Closed-Book) to 14.06% (RAG).
  • Causes:
    1. Format Non-compliance: Models retrieved relevant info but failed to adhere to strict single-letter output constraints, generating verbose explanations that were scored as zero.
    2. Safety Refusals: Rigid safety alignments triggered refusals (e.g., "I cannot provide legal advice") when the complex RAG prompt contained liability descriptions.

C. The Specialization Paradox

• Finding: Smaller, domain-specific French fine-tuned models (e.g., French-Alpaca-Llama3-8b-it) underperformed massive generalist models.
• Result: The specialized French model achieved only 0.08% accuracy with RAG, ranking last.
• Insight: For complex financial advisory, abstract reasoning capabilities (generalist) are more critical than surface-level vocabulary alignment with specific legal jargon.

D. The Proprietary Gap

• Proprietary models (e.g., OpenAI, Anthropic) consistently outperformed open-weight models by approximately 20 percentage points. Open models currently lack the necessary reasoning density and multilingual robustness for autonomous deployment in high-liability scenarios.

5. Significance and Implications

• Architectural Shift: The study suggests that future financial AI should prioritize inference-time reasoning over parametric memorization of laws.
• Deployment Risks: The "Context Distraction" phenomenon reveals that simply adding a RAG layer is not a guaranteed solution; it can actively degrade the performance of capable models. Rigorous, model-specific calibration is required before autonomous deployment.
• Ethical & Legal Constraints:
  • Jurisdictional Bias: Deploying Common Law-trained models in Civil Law jurisdictions is an ethical failure that risks providing legally invalid advice.
  • Human-in-the-Loop: Given the 21% error rate of even the best models and the legal precedent (e.g., Moffatt v. Air Canada) holding corporations liable for AI errors, fully autonomous deployment is deemed premature.
• Future Directions: Research must focus on distilling reasoning capabilities into smaller models, developing robustness interventions to mitigate context sensitivity, and evolving evaluation protocols to include generative justification and multi-turn dialogue rather than just multiple-choice selection.