INS-ActBench: A Comprehensive Benchmark for Assessing Professional Actuarial Capability of Large Language Models

Technical Summary: INS-ActBench

Problem Statement
While Large Language Models (LLMs) have demonstrated potential in financial reasoning, existing benchmarks typically evaluate domain knowledge, numerical reasoning, long-context understanding, and tool use in isolation. This fragmented approach fails to assess the integrated professional workflows required in high-stakes fields like actuarial science, where decisions must be context-grounded, computationally reproducible, and numerically verifiable. Current actuarial studies are often limited to individual applications, and financial benchmarks do not systematically evaluate the progression from domain knowledge to case-based reasoning and finally to verifiable professional execution.

Methodology
The authors introduce INS-ActBench, a comprehensive benchmark constructed from 12,050 question-and-answer pairs derived from public examination materials released by 16 professional actuarial associations. The construction pipeline involved systematic question selection, task restructuring, format standardization, and multi-stage expert verification.

The benchmark is structured around a progressive capability framework comprising three subsets:

1. INS-Act-Know: Evaluates standardized actuarial knowledge (concepts, principles, formulas) across six subcategories: Probability and Statistics, Economics and Finance, Actuarial Mathematics, Life Insurance, Non-life Insurance, and Actuarial Modeling. Questions are standardized into single-answer multiple-choice formats.
2. INS-Act-Case: Evaluates long-context reasoning over realistic insurance business cases. It utilizes company backgrounds, financial statements, and operating information (up to 175K tokens) to test evidence localization and judgment under complex conditions. Questions are converted to multiple-answer multiple-choice formats.
3. INS-Act-Practice: Evaluates tool-oriented practice where models must translate actuarial requirements into executable workflows. This includes Spreadsheet (SS) tasks (generating Python code to manipulate Excel files) and R-code (R) tasks (generating R code for statistical analysis). Both require numerically verifiable outputs with a 0.2% relative error tolerance.

The evaluation covered nine representative LLMs (six proprietary, three open-source) and a baseline of five human actuarial experts. Human experts used offline tools (Excel, R, textbooks) without internet or LLM assistance.

Key Results
Experiments reveal a clear capability boundary between standardized knowledge mastery and professional work competence:

• Knowledge vs. Execution: Frontier proprietary LLMs (e.g., GPT-5.5, Gemini-3.1-Pro) perform strongly on INS-Act-Know, often exceeding the sampled human baseline. However, their performance declines substantially in INS-Act-Case and INS-Act-Practice, where human participants remain more consistent.
• Tool Usage: In INS-Act-Practice, error analysis indicates that failures are primarily concentrated in "modeling and calculation errors" (pure actuarial knowledge issues) and "function usage errors" (mixed tool/knowledge issues), rather than simple tool call failures. Models can often initiate tool workflows but struggle to translate them into correct actuarial procedures.
• Numerical vs. Non-Numerical: Frontier models handle formula-based numerical reasoning well when steps are well-defined, but weaker models show significant accuracy drops on numerical questions compared to non-numerical ones.
• Jurisdictional Sensitivity: Performance varies significantly across different actuarial associations (SOA, IFoA, Others). Models perform better on standardized knowledge but show greater inconsistency in case analysis and practice tasks across different regulatory contexts, suggesting a reliance on specific training distributions rather than generalized regulatory reasoning.

Significance and Contributions
The paper claims three primary contributions:

1. First Large-Scale Actuarial Benchmark: INS-ActBench is the first benchmark to span knowledge, case reasoning, and tool-based practice, addressing the gap in evaluating integrated professional workflows.
2. Systematic Construction Pipeline: The authors developed a pipeline to transform raw examination materials from 16 associations into 12,050 standardized instances, ensuring reproducibility and minimizing data contamination.
3. Empirical Evidence of Capability Gaps: The study provides reproducible evidence that while LLMs are increasingly capable of standardized actuarial calculation and knowledge retrieval, reliable professional assistance requires stronger contextual integration, tool-based execution, and consistency across diverse regulatory settings.

The authors position INS-ActBench as a foundation for developing actuarial LLMs toward reliable professional assistance, emphasizing that future progress must move beyond exam-style knowledge performance toward trustworthy decision support in real-world insurance workflows. The benchmark is designed to identify failure modes in reasoning, tool use, and jurisdiction-sensitive interpretation, rather than to serve as a basis for real financial decisions or professional qualification.