💬 NLP

ActuBench: A Multi-Agent LLM Pipeline for Generation and Evaluation of Actuarial Reasoning Tasks

This paper introduces ActuBench, a multi-agent LLM pipeline that automates the generation and evaluation of advanced actuarial reasoning tasks aligned with the IAA syllabus, demonstrating that independent verification significantly improves item quality while revealing that locally-hosted open-weights models offer superior cost-performance trade-offs and that open-ended evaluation is essential for accurately discriminating top-tier model capabilities.

Original authors: Jan-Philipp Schmidt

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jan-Philipp Schmidt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a massive, ultra-difficult exam for the world's most precise accountants and risk managers: Actuaries. These are the math wizards who calculate the odds of everything from car crashes to life expectancy to set insurance prices.

The problem is, writing these exams is incredibly hard, expensive, and slow. You need human experts to write questions that are tricky but fair.

Enter ActuBench. Think of this not as a single person writing a test, but as a high-tech, automated factory run by a team of AI robots working together to build, check, and grade these exams.

Here is how the paper explains this system, broken down into simple concepts:

1. The Factory Floor: A Team of Specialized Robots

Instead of asking one AI to do everything (which often leads to mistakes), the authors built a pipeline with four distinct "agents" (AI roles), each with a specific job:

Agent A (The Architect): This is the smartest, most expensive robot. Its job is to look at a learning goal (like "calculate pension risk") and draft the actual exam question. It's the creative writer.
Agent B (The Trickster): Once the question is written, Agent B's job is to invent the "wrong answers" (distractors). These need to look plausible enough to fool a student who is halfway there, but clearly wrong to an expert.
Agent C (The Strict Inspector): This is the most important innovation. Agent C never writes anything. It only looks at what A and B produced. It acts like a strict teacher grading a draft. If the question is confusing or the wrong answers are too obvious, Agent C sends it back for a fix.
- The Analogy: Imagine a writer (A) and a joke-writer (B) trying to make a comedy sketch. Agent C is the director who says, "That joke doesn't land," or "The setup is unclear." Because C didn't write the sketch, it doesn't have "blind spots" and can spot errors the writers missed.
The Assistant (The Librarian): A cheap, fast robot that fetches information from Wikipedia to make sure the facts are real and tags the questions by topic (e.g., "Health Insurance" or "Pensions").

The Magic: This "Inspector" role is the paper's big breakthrough. By separating the writer from the checker, they found that the AI caught 60% of the mistakes on the very first try, fixing them automatically without a human needing to step in.

2. The Two Types of Exams

The researchers didn't just test the AI on one kind of question. They ran two different types of tests to see how the AI really thinks:

The Multiple-Choice Test (MCQ): The AI sees a question with four options (A, B, C, D) and picks one.
- The Catch: This is like a "recognition" test. Even if the AI doesn't know the answer, it might guess correctly by eliminating the silly options.
The "Judge" Test (Open-Ended): The AI has to write out the answer and explain its reasoning from scratch, with no options to choose from.
- The Catch: This is a "derivation" test. The AI can't guess; it has to actually do the math and logic.

3. The Big Surprises (The Results)

The team tested 50 different AI models (from big companies like Google, OpenAI, and Anthropic, plus some open-source ones). Here is what they found:

Surprise #1: The "Cheap" Winners.
You might think the most expensive, super-smart AI models would win easily. But on the multiple-choice test, a few models tied for first place (98% accuracy). However, the cost to run them varied wildly.
- The Analogy: It's like two cars finishing a race in the same time. One is a $200,000 Ferrari (expensive AI), and the other is a $20,000 Toyota (open-source AI). The Toyota got you to the finish line for a fraction of the price.
- The Winner: A model running on a standard home computer (Gemma) or a cheap cloud service (Cerebras) performed almost as well as the most expensive ones.
Surprise #2: The "Thinking" Mode isn't Always Worth It.
Some companies offer a "Reasoning Mode" where the AI takes extra time to "think" before answering.
- The Result: On multiple-choice questions, this extra thinking only gave a tiny boost (a few percentage points) but cost 2x to 5x more money. It was like paying for a luxury upgrade that didn't actually make the car go much faster.
Surprise #3: The "Multiple Choice" Trap.
This is the most important finding. The AI models looked very similar on the Multiple-Choice test (all scoring near 100%). But when they took the Open-Ended "Judge" test, the rankings changed completely.
- The Analogy: Imagine a student who memorizes the answer key. On a multiple-choice test, they get an A+. But on a test where they have to show their work, they fail.
- The "Multiple Choice" format inflated the scores, hiding who was actually good at deep reasoning. Only the "Judge" test revealed the true difference between the top-tier AI and the rest.

4. Why This Matters

The authors built a public website where anyone can look at these questions and see how different AIs answered them.

The Takeaway:
If you just need an AI to pick the right answer from a list (like a quick check), you don't need the most expensive, "thinking" AI. You can use a cheaper, open-source model running on your own computer and get great results.

But, if you need the AI to solve a complex problem from scratch (like a real-world actuarial calculation), you do need the top-tier, expensive "reasoning" models. The cheap models can recognize the answer, but they struggle to derive it without help.

In short: ActuBench is a new, automated factory that builds better exams for AI, proving that while cheap AI is great at guessing, the expensive "thinking" AI is still the only one that can truly solve the hardest problems.

1. Problem Statement

Actuarial science is a highly technical domain requiring precise mathematical reasoning, deep familiarity with financial theory, and sensitivity to regulatory frameworks. The International Actuarial Association (IAA) Education Syllabus defines the body of knowledge for qualified actuaries. However, creating high-quality assessment items (questions) aligned with this syllabus is expensive, slow, and requires scarce subject-matter experts.

While Large Language Models (LLMs) have advanced in general reasoning, there is a lack of English-language benchmarks specifically aligned with the IAA syllabus to evaluate their capabilities in actuarial tasks. Existing studies either lack benchmarking, focus on narrow subdomains, or target non-English datasets. Furthermore, static benchmarks suffer from data contamination and rapid saturation as models improve.

2. Methodology: The ActuBench Pipeline

The authors propose ActuBench, a multi-agent LLM pipeline designed to automatically generate, verify, and evaluate actuarial assessment items. The system is built on a four-role agent architecture separated by adapters to ensure distinct functions and cost optimization.

A. Generation Pipeline (Multi-Agent Workflow)

The pipeline processes IAA learning objectives through a linear sequence:

Agent A (Drafting): Bound to a strong "reasoning-mode" model (e.g., claude-sonnet-4-6:thinking). It extracts keywords, fetches relevant Wikipedia excerpts via API, summarizes them, and drafts the question stem, correct answer, and rationale.
Agent B (Distractor Construction): Bound to a non-reasoning, creative model (e.g., claude-sonnet-4-6). It generates three plausible but incorrect distractors, each with a rationale explaining the specific misconception it targets.
Agent C (Independent Verifier): The core methodological differentiator. Bound to a separate, potentially critical model (e.g., o3:reasoning), it never generates content. It independently verifies the item stem/answer and the distractor ensemble.
- Verification Logic: Checks for ambiguity, factual errors against the source notes, and underspecification.
- Repair Loop: If Agent C flags an error, it triggers a bounded one-shot repair loop. Agent A or B produces a single revision, which is re-verified. If it fails again, the item is persisted with a failure flag rather than discarded.
Auxiliary Agent: A cost-optimized, fast model (e.g., gpt-5.4-mini) handles Wikipedia note summarization and topic labeling (e.g., Life, Non-life, Pensions).

B. Evaluation Protocol

The pipeline produces two complementary benchmarks, each containing 100 items drawn from disjoint sets to prevent cross-contamination:

MCQ Benchmark: 100 items selected as the "empirically hardest" (lowest collective accuracy) from a larger pool. Models must select a single letter (A-D) without explanation.
Judge Benchmark: 100 items (restricted to quantitative calculation) where models provide open-ended free-text answers. A separate "Judge" LLM scores the response against the ground truth, extracting the final answer and reasoning.

3. Key Contributions

Multi-Agent Generation with Independent Verification: The separation of drafting, distractor creation, and verification into distinct agents (specifically the "Verifier-First" role of Agent C) significantly improves item quality. Agent C flags the majority of items on the first pass, demonstrating that verification is more reliable than self-verification.
Dual Evaluation Modes: The paper introduces a comparison between MCQ (recognition/elimination) and LLM-as-Judge (derivation) modes, revealing that they produce meaningfully different rankings.
Live, Accessible Benchmark: A web interface (https://actubench.de/en/) allows public inspection of all 200 items and 50 model responses without requiring a repository checkout, enhancing reproducibility.
Comprehensive Empirical Evaluation: Evaluation of 50 language models from 8 providers (Anthropic, OpenAI, Google, xAI, DeepSeek, Mistral, Cohere, and open-weights) on the hardest actuarial tasks.
Cost-Performance Analysis: Identification of models that sit on the cost-performance Pareto front, specifically highlighting the viability of locally hosted open-weights models.

4. Key Results

A. Cost-Performance Landscape

Frontier Saturation: Four models (claude-opus-4-6:thinking, gpt-5-mini, o3:reasoning, o4-mini:reasoning) achieved 98% accuracy on the MCQ benchmark. However, their costs varied by an order of magnitude (from ~$0.09 to >$1.50 per run).
Pareto Dominance:
- Cerebras-hosted gpt-oss-120b (open-weights) achieved 97% accuracy at near-zero cost.
- Locally hosted Gemma 4 (via Ollama on consumer hardware) achieved 85% accuracy at zero marginal cost.
- Conclusion: For structured MCQ tasks, flagship reasoning models are often not cost-effective compared to optimized open-weights models.

B. MCQ vs. Judge Divergence

The "Scaffold" Effect: The MCQ format inflates performance ceilings. While top models scored ~98% on MCQ, their best Judge-mode score dropped to 87%.
Differentiation: Removing the multiple-choice scaffold (Judge mode) revealed a 10–15 percentage point gap between the strongest reasoning models and non-reasoning models.
Rank Correlation: The rankings between MCQ and Judge modes showed only moderate agreement ( $\rho=0.68$ ). MCQ favors elimination skills, while Judge mode tests genuine derivation.

C. Sector Performance

Accuracy varied by actuarial sector. ADS-AI (Data Science/AI) had the highest mean accuracy (87%), while Non-life had the lowest (73%).
Reasoning Modes: Enabling "thinking" modes provided a modest average gain (~3.5 pp) but was highly variable. One pair (claude-opus-4-6) saw a +13 pp gain, while others saw negligible or negative returns.

5. Significance and Implications

For the Actuarial Community: ActuBench provides the first English-language, IAA-aligned benchmark, enabling practitioners to assess LLM readiness for tasks like risk modeling and regulatory compliance.
For LLM Benchmarking: The study challenges the reliance on static MCQ benchmarks. It demonstrates that Judge-mode evaluation is necessary to discriminate between top-tier models on complex reasoning tasks.
For Deployment Strategy:
- Structured Tasks (MCQ/Classification): Organizations can achieve near-state-of-the-art accuracy using locally hosted open-weights models (e.g., Gemma 4) or low-cost hosted endpoints, avoiding expensive flagship reasoning models.
- Open-Ended Derivation: For tasks requiring genuine reasoning and derivation, reasoning-mode variants or frontier models are cost-justified, as the performance gap widens significantly in Judge mode.
Methodological Insight: The "Independent Verifier" architecture is a robust pattern for automated content generation, effectively catching hallucinations and logical errors that self-verification misses.

6. Limitations

Statistical Power: With 100 items per benchmark, small rank differences (e.g., <5%) may not be statistically significant.
Data Grounding: The pipeline relies solely on Wikipedia, which may lack coverage for specific non-English regulations (e.g., Solvency II) or specialized consulting methodologies.
Language: Currently English-only; a German version is proposed for the DAV (German Actuarial Association) ecosystem.
Judge Bias: While mitigated by ground-truth answers, the LLM-as-Judge still carries potential biases regarding verbosity and style.

In summary, ActuBench establishes a rigorous framework for evaluating LLMs in high-stakes professional domains, proving that automated multi-agent pipelines can generate high-quality assessments and that cost-effective, locally hosted models are viable for many actuarial applications.