FIRE: A Comprehensive Benchmark for Financial Intelligence and Reasoning Evaluation

This paper introduces FIRE, a comprehensive benchmark designed to evaluate large language models on both theoretical financial knowledge through exam questions and practical business reasoning via 3,000 curated scenario questions, accompanied by a systematic evaluation matrix and results from state-of-the-art models including the authors' XuanYuan 4.0.

The Gist

Imagine you are hiring a new financial advisor. You have two ways to test them:

1. The Written Test: You give them a stack of textbooks and ask them to answer multiple-choice questions about interest rates, tax laws, and investment theories.
2. The Real-World Simulation: You put them in a chaotic office where a client is panicking about a stock crash, a bank is asking for a risky loan, and a regulator is auditing their files. You ask them to actually solve these problems and make decisions.

For a long time, the tech world only cared about Test #1. They built Artificial Intelligence (AI) models and asked them to pass finance exams. If the AI got an "A" on the test, everyone assumed it was a genius banker.

The Problem:
Just because someone can ace a written exam doesn't mean they can handle a real crisis. The authors of this paper realized that existing AI tests were like asking a pilot to recite the manual but never actually letting them fly the plane in a storm. The old tests were too simple, too repetitive, and didn't cover the messy, complicated reality of actual banking and investing.

The Solution: FIRE
The team created FIRE (Financial Intelligence and Reasoning Evaluation). Think of FIRE as the ultimate "Driver's License" for AI in finance. It's a massive, two-part challenge designed to see if an AI is just a "bookworm" or a true "expert."

Here is how FIRE works, using simple analogies:

Part 1: The "Knowledge Vault" (The Written Test)

• What it is: The team gathered over 14,000 questions from the world's toughest real-world finance exams (like the CFA for analysts, CPA for accountants, and FRM for risk managers).
• The Analogy: Imagine a library containing every single question ever asked on a bar exam or a medical board exam. The AI has to answer these to prove it knows the rules, the math, and the vocabulary.
• The Goal: To check if the AI has the theoretical brainpower to understand finance.

Part 2: The "Simulation Lab" (The Real-World Test)

• What it is: This is the big innovation. The team created 3,000 realistic scenarios based on actual jobs in banks, insurance companies, and investment firms.
• The Analogy: Instead of asking, "What is a credit risk?" (Theory), they ask: "A client wants a loan for a new factory, but their cash flow looks shaky and the economy is slowing down. Do you approve the loan? If yes, what conditions do you set? If no, how do you explain it to the client without losing them?"
• The Matrix: They organized these questions like a giant spreadsheet.
  • Rows: Different industries (Banking, Insurance, Stocks, Crypto).
  • Columns: Different tasks (Making a decision, designing a product, fixing a customer complaint, catching fraud).
• The Goal: To see if the AI can actually do the job, not just talk about it.

How They Grade the AI

• For the Written Test: It's easy. Right or Wrong. (1 or 0).
• For the Real-World Test: This is tricky because there isn't always one "right" answer.
  • The Solution: They built a special "AI Judge." Imagine a senior bank manager who has a strict checklist (a rubric). The AI Judge reads the AI's answer and checks: Did they spot the risk? Did they follow the law? Was the tone professional? It gives a score based on how well the AI followed the rules of the game.

What They Found (The Plot Twist)

The researchers tested the smartest AI models available (including their own new model, XuanYuan 4.0) against FIRE.

1. The Good News: The AIs are brilliant at the written test. They scored incredibly high on the 14,000 exam questions. They know the definitions and the formulas perfectly.
2. The Bad News: The AIs struggled in the Simulation Lab. When faced with messy, real-world scenarios, their performance dropped significantly. They often missed subtle risks or gave generic advice that wouldn't work in a real bank.

The Takeaway:
The paper concludes that current AI is like a student who memorized the entire dictionary but has never held a conversation. They know the words of finance, but they haven't learned the wisdom of finance yet.

Why This Matters:
FIRE is a tool to stop banks from trusting AI just because it passed a quiz. It forces developers to build AI that can actually handle the stress, nuance, and danger of real money management. It's the difference between a robot that can recite the safety manual and a robot that can actually save the plane when the engine fails.

Technical Summary

1. Problem Statement

Despite the rapid advancement of Large Language Models (LLMs) in general domains, their application in the financial sector faces significant evaluation challenges. Existing benchmarks suffer from three primary limitations:

• Superficial Adaptation: Most benchmarks merely adapt standard Natural Language Processing (NLP) tasks to financial contexts, failing to assess the ability to solve authentic, complex business problems.
• Coarse Categorization: Existing evaluations often lack fine-grained taxonomies, leading to task overlap while neglecting critical subdomains (e.g., specific risk management or compliance scenarios).
• Misalignment with Business Value: Current metrics often measure superficial terminology understanding rather than practical operational efficacy (e.g., credit approval efficiency or fraud detection), making the Return on Investment (ROI) of AI deployment difficult to quantify.

There is a critical gap between theoretical financial knowledge and the ability to operationalize that knowledge in real-world, multi-step financial workflows.

2. Methodology

The authors introduce FIRE, a comprehensive benchmark developed through a collaboration between Du Xiaoman Technology, Tsinghua University, and Renmin University of China. The methodology is structured around two complementary evaluation categories and a novel matrix-based framework.

A. The FIRE Framework

The benchmark consists of 17,000+ total evaluation items divided into two distinct categories:

1. Theoretical Knowledge Assessment (14,000+ Questions)

• Source: Curated from 14 major professional financial qualification exams, including international certifications (CFA, FRM, CIA, CISA, CMA) and Chinese domestic certifications (CPA, CCBP, AFP, Fund/Futures Practitioner, Economist).
• Focus: Tests deep understanding of financial concepts, regulatory frameworks, and ethical standards.
• Evaluation: Binary scoring (1 for correct, 0 for incorrect) based on exact match with ground-truth answers.

2. Practical Skills Assessment (3,000 Questions)

• Source: Real-world financial business scenarios collected from industry operations.
• Evaluation Matrix: A two-dimensional matrix is used to ensure granular coverage:
  • Vertical Axis (Sectors): 8 sectors (Banking, Insurance, Securities, Fund Management, Futures, Trust, FinTech, General Finance) further broken down into 17 business subcategories.
  • Horizontal Axis (Functions): 4 functional pillars (Insight & Decision, Product Design & Marketing, Service & Operation, Risk & Compliance).
• Question Types:
  • 1,000 Closed-Form Questions: Paired with high-quality reference answers for objective, automated exact-match evaluation.
  • 2,000 Open-Ended Questions: Lack ground-truth answers. These are evaluated using a Problem-Wise Rubric approach.
    • Rubric Generation: Specific evaluation rubrics are generated for each task to define scoring criteria.
    • Scoring Model: A dedicated scoring model (trained via Supervised Fine-Tuning and Reinforcement Learning with Verifiable Rewards) is used to grade open-ended responses against these rubrics, addressing the instability of standard "LLM-as-a-Judge" methods.

B. Baseline Model: XuanYuan 4.0

To provide a strong in-domain baseline, the authors developed XuanYuan 4.0, a 36B-parameter dense model.

• Training Pipeline:
  1. Continued Pre-training (CPT): Integrated with curated financial corpora and an annealed training schedule.
  2. Self-Regularization: Uses KL-divergence relative to a reference model to prevent drift.
  3. Supervised Fine-Tuning (SFT): Utilizes high-fidelity datasets in math, STEM, and agentic tasks.
  4. Reinforcement Learning (RLVR): Optimized using the DAPO algorithm on internal financial business scenarios to align outputs with complex financial logic.

3. Key Contributions

• FIRE Benchmark: The first comprehensive benchmark that systematically bridges the gap between theoretical financial knowledge and practical, scenario-based reasoning.
• Financial Application Scenario Matrix: A novel, principled framework for decomposing financial tasks into specific sectors and functional pillars, serving as a roadmap for data curation and model alignment.
• Advanced Evaluation Protocol: Introduction of a rubric-based, automated scoring pipeline for open-ended financial tasks, overcoming the instability of generic LLM evaluators.
• Strong Open-Source Baseline: Release of XuanYuan 4.0, demonstrating that domain-specific alignment can allow smaller models (36B) to compete with much larger proprietary models.
• Open Resources: Public release of the benchmark questions, evaluation code, and the XuanYuan 4.0 model architecture details.

4. Experimental Results

The authors evaluated state-of-the-art proprietary models (Gemini 3.0 Pro, GPT-5.2, Claude 4.5), open-source models (DeepSeek V3.2, Kimi K2, Qwen3), and financial-specialized models on FIRE.

• Theoretical Knowledge (Exams):
  • Models generally perform well on certification exams (averaging 80–90%+ for top models).
  • XuanYuan 4.0 achieved the highest performance among open-source models (92.15% average), comparable to the proprietary leader Gemini 3.0 Pro (91.43%).
• Practical Scenarios (Real-World Tasks):
  • A significant performance drop was observed compared to exam results.
  • GPT-5.2 led proprietary models (77.46% average).
  • XuanYuan 4.0 (79.08% average) outperformed its backbone (Seed-OSS-36B) and other open-source baselines, achieving performance parity with GPT-5.2 despite having significantly fewer parameters.
• Key Insight: There is a performance decoupling. Models excel at standardized knowledge retrieval (exams) but struggle with complex, operational financial reasoning. Domain-specific training (as seen in XuanYuan 4.0) is crucial for bridging this gap.

5. Significance

• Industry Impact: FIRE provides a rigorous standard for financial institutions to quantify the ROI of AI deployment, moving beyond "chatbot" capabilities to assessing real-world decision-making and risk control.
• Research Direction: The results highlight that current LLM architectures have a fundamental limitation in translating theoretical knowledge into actionable intelligence. Future research must focus on domain-specific alignment and operational reasoning rather than just scale.
• Security & Compliance: By emphasizing regulatory compliance and risk management scenarios, FIRE encourages the development of safer, more reliable financial AI systems.
• Cost Efficiency: The success of XuanYuan 4.0 suggests that specialized fine-tuning can enable smaller, more cost-effective models to handle complex financial tasks, facilitating broader adoption in the industry.