BankMathBench: A Benchmark for Numerical Reasoning in Banking Scenarios

1. Problem Statement

While Large Language Models (LLMs) are increasingly adopted in digital banking for customer service, they exhibit significant deficiencies in core banking numerical reasoning. Existing models struggle with:

• Multi-step computations: Calculating total payouts, comparing products with varying interest rates, and handling early repayment conditions.
• Domain-specific logic: Misinterpreting product types (e.g., simple vs. compound interest), applying conditions incorrectly, and failing at basic arithmetic involving exponents or geometric progressions.
• Benchmark Gaps: Existing mathematical benchmarks (e.g., GSM8K, MATH) focus on academic curricula, while financial benchmarks (e.g., FinQA) target financial statements and securities analysis. Neither adequately covers everyday banking scenarios (deposits, savings, loans) requiring condition-specific reasoning.

2. Methodology: BankMathBench Dataset

The authors propose BankMathBench, a domain-specific dataset designed to evaluate and improve LLM performance in realistic banking contexts.

A. Dataset Structure

The dataset contains 13,839 problems in both Korean and English, categorized into three difficulty levels:

1. Basic: Single-product reasoning (deposits, savings, loans) requiring multi-step calculations (arithmetic, progressions, decimals).
2. Intermediate: Multi-product comparisons (e.g., Deposit A vs. Savings B), requiring separate computations for each product followed by a comparative analysis.
3. Advanced: Complex multi-condition scenarios involving variable interest rates, early withdrawal penalties, tax deductions, and specific repayment schedules (e.g., equal principal vs. equal principal-and-interest).

B. Automated Generation Pipeline

The dataset was generated via a fully automated pipeline using GPT-4o and o1-mini, followed by manual verification by banking professionals. The pipeline consists of three stages:

1. Question Generation: Uses prompt engineering to create realistic scenarios with randomized parameters (amounts, rates, terms) to ensure diversity. Includes a data augmentation step where numerical values are perturbed or masked to force contextual inference.
2. Solution Generation & Verification: A two-tiered strategy ensures accuracy:
   • Basic: Generates mathematical expressions and LaTeX.
   • Intermediate/Advanced: Generates executable Python code.
   • Verification: Only pairs where two independent solution methods yield consistent results are retained.
3. Reasoning Generation: Produces step-by-step reasoning traces integrating natural language and mathematical expressions, tagged with <calc> and <result> for tool integration.

C. Expert Validation

Two banking professionals with over three years of experience reviewed the dataset for practical relevance, logical soundness, and format consistency, ensuring the data reflects real-world bank consultations.

3. Key Contributions

• First Comprehensive Banking Benchmark: Introduces the first dataset specifically targeting everyday banking numerical reasoning (deposits, savings, loans) across three difficulty tiers.
• Reproducible Generation Pipeline: Provides a parameterized, automated pipeline that can be adapted to different national financial contexts by adjusting variables like tax policies and currency units.
• Tool-Augmented Training Framework: Demonstrates a novel approach combining Supervised Fine-Tuning (SFT) with Tool-Augmented SFT, where models are trained to generate formulas that are automatically executed by external calculators.

4. Experimental Results

The authors evaluated various open-source (e.g., LLaMA-3, Qwen, DeepSeek) and closed-source (GPT-4o, Claude, Gemini) models on BankMathBench.

• Zero-Shot Performance: Existing models performed poorly, especially as difficulty increased. For example, on the Advanced level, most open-source models scored near 0% accuracy, and even top-tier closed-source models (GPT-4o, Claude) dropped significantly (e.g., GPT-4o: 6.3% on Advanced).
• Impact of Fine-Tuning (SFT): Training on BankMathBench yielded substantial improvements.
  • Basic Level: Average accuracy increased by 57.6% (Korean) and 57.6% (English) for tool-augmented models.
  • Intermediate Level: Average accuracy increased by 75.1% (Korean) and 72.7% (English).
  • Advanced Level: Average accuracy increased by 62.9% (Korean) and 63.4% (English).
• Tool-Augmented SFT: Integrating external tools (calculators) during fine-tuning provided the most significant gains, particularly for the Intermediate level where aggregating multiple computed results is required. This approach reduced cumulative numerical errors to nearly zero.
• Model Size & Architecture: While larger models (7B/8B) generally performed better, lightweight models (e.g., 1.5B/2.1B) showed surprising competitiveness when fine-tuned on this specific domain, suggesting that domain-specific data is more critical than sheer scale for banking tasks.

5. Significance and Conclusion

• Reliability in Finance: The study highlights that current LLMs are unreliable for financial advice without domain-specific training and tool integration. BankMathBench provides a necessary standard for evaluating this reliability.
• Path to Robust Banking Assistants: The results suggest that combining domain-specific fine-tuning with tool-augmented reasoning is the most effective path toward deploying accurate, computation-grounded digital banking assistants.
• Future Directions: The authors plan to expand the dataset to include funds and insurance and to incorporate explicit assessments of reasoning quality beyond just numerical accuracy.

In summary, BankMathBench bridges the gap between general mathematical reasoning and the specific, high-stakes numerical requirements of the banking industry, demonstrating that targeted data generation and tool integration can significantly enhance LLM capabilities in this critical domain.