Technical Summary: RealFin – How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?

1. Problem Statement

Current financial Large Language Model (LLM) evaluations suffer from a critical flaw: they operate under a closed-world assumption, presuming that every question is fully specified and admits a single correct answer. In real-world financial practice, however, problems are often under-specified. Key premises (e.g., regulatory standards, time horizons, accounting rules, or macroeconomic assumptions) are frequently implicit or missing.

When faced with such "condition-missing" questions, current LLMs tend to:

1. Over-commit: They confidently generate answers by silently filling in missing gaps with learned defaults or training data biases.
2. Fail to Abstain: They lack the epistemic awareness to recognize that a question is logically underdetermined and that no valid answer can be derived from the provided information.
3. Misinterpret Ambiguity: This failure is particularly pronounced in Chinese, where context-dependent terminology can mask missing conditions, leading models to hallucinate plausible but unjustified conclusions.

The paper argues that reliable financial AI must not only answer correctly when information is sufficient but also identify missing conditions and abstain when a question cannot be answered.

2. Methodology

A. The REALFIN Benchmark

The authors introduce REALFIN, a bilingual (English/Chinese) benchmark designed to evaluate financial reasoning under incomplete information.

• Dataset Construction: Starting from 1,178 professional, well-posed exam-style questions (CFA-style for English, CPA-style for Chinese), the authors manually created paired "condition-missing" versions.
• Condition Removal: They systematically removed logically necessary premises while ensuring the remaining text remained linguistically plausible. Removed conditions fell into four categories:
  1. Initial Assumptions: (e.g., interest rate regimes, inflation expectations).
  2. Intermediate Linking Conditions: (e.g., specific valuation models like CAPM/DCF, hedging logic).
  3. Constraint-Defining Information: (e.g., financial covenants, seniority attributes).
  4. Standard-Selection Cues: (e.g., IFRS vs. GAAP, regulatory frameworks).
• Task Formulations: The benchmark evaluates models under three distinct settings:
  1. Original (Full-Condition): Standard QA with complete information.
  2. Revised (Condition-Missing): Questions with removed premises; models must either clarify or recognize the impossibility of a unique answer.
  3. None-of-the-Above (NOTA): The correct answer option in the Revised set is replaced with "None of the Above." This forces models to explicitly identify missing information rather than guessing the "most likely" option.

B. Experimental Setup

• Models Evaluated: 15 LLMs were tested, categorized into:
  • General-Purpose: (e.g., GPT-5.1-mini, Claude-3.5-Sonnet, DeepSeek-V3).
  • Finance-Specialized: (e.g., XuanYuan3-70B, Fin-R1-7B, DISC-FinLLM).
  • Reasoning-Enhanced: (e.g., DeepSeek-R1, GPT-OSS series, DianJin-R1).
• Evaluation Protocol: Zero-shot setting with a strict JSON output format requiring reason, answer, and confidence.
• Metrics: Accuracy on Original vs. Revised sets, and a novel Reasoning Accuracy metric (checking if the model's explanation correctly identifies the missing premise) vs. Answer Accuracy (selecting NOTA).

3. Key Contributions

1. RealFin Benchmark: The first dataset to systematically evaluate financial reasoning on under-specified problems, bridging the gap between academic benchmarks and real-world ambiguity.
2. Bilingual Evaluation: A comprehensive dataset covering English (CFA) and Chinese (CPA) financial domains, revealing significant cross-lingual disparities in model behavior.
3. New Evaluation Paradigm: Moving beyond simple answer accuracy to assess epistemic caution—the ability to detect missing information and refuse to answer when justified.
4. Comprehensive Model Analysis: A rigorous comparison of general, specialized, and reasoning-enhanced models, highlighting that domain-specific fine-tuning does not automatically solve the problem of over-commitment.

4. Key Results

A. Performance Degradation and Over-Commitment

• General Models: Surprisingly, general-purpose models (e.g., GPT-5.1-mini, Claude-3.5) often outperformed finance-specialized models on Revised tasks. However, this "success" was largely due to implicit assumption alignment—they guessed the missing condition correctly by chance or by relying on training data defaults, rather than recognizing the missing information.
• Finance-Specialized Models: Early financial models (e.g., DISC-FinLLM, FinGPT) performed poorly, often failing to recognize missing conditions and aggressively retrieving domain knowledge to force an answer, leading to near-zero accuracy in NOTA settings.
• Reasoning-Enhanced Models: Models like Fin-R1-7B showed superior performance. They were better at identifying missing premises and abstaining (selecting NOTA) when appropriate. However, they sometimes suffered from "over-reasoning," generating long hypothetical scenarios even after identifying the missing condition.

B. The "Dead Zone" in Chinese CPA

• A critical finding is the Chinese-specific failure mode. In Chinese CPA tasks, models (even large ones) exhibited a "dead zone" where they failed to answer condition-missing questions, often scoring near 0% on complex calculations.
• Cause: Chinese financial terminology is highly context-dependent. When a condition is removed, the remaining text often still forms a coherent sentence that misleads the model into making aggressive, unjustified inferences.

C. The Gap Between Answer and Reasoning

• Answer Accuracy vs. Reasoning Accuracy: In NOTA settings, many models selected the correct "None of the Above" option but failed to explain why (low reasoning accuracy).
  • Example: GPT-OSS-120B achieved ~89% Answer Accuracy on English NOTA but only ~20% Reasoning Accuracy. It selected NOTA as a "safe" option without understanding the logical gap.
• Confidence Mismatch: Models often displayed high confidence (e.g., >90%) even when their reasoning was flawed or when they were guessing.

D. Scaling Does Not Solve the Problem

• Increasing model size (e.g., from 20B to 120B parameters) improved answer accuracy but worsened the reasoning gap. Larger models became more confident in their unjustified assumptions, leading to higher over-commitment.

5. Significance and Implications

• Redefining Reliability: The paper argues that a reliable financial LLM must satisfy three criteria: (1) Answer correctly when information is sufficient, (2) Identify missing conditions when they are absent, and (3) Abstain when no valid answer exists.
• Risk of Automation: Current benchmarks that only measure answer accuracy on well-posed questions dangerously overestimate the reliability of LLMs in high-stakes financial decision-making.
• Future Directions: The authors propose a training pipeline combining Specialized SFT (for professional knowledge) and Logic-Driven RL (to penalize guessing and reward explicit identification of missing information). They also suggest moving from passive rejection to active clarification, where models ask users for missing data.

Conclusion: RealFin demonstrates that current LLMs, regardless of size or domain specialization, struggle to distinguish between "solvable" and "unsolvable" financial problems. True reliability in finance requires models that possess the humility to say "I don't know" when the necessary information is missing, rather than confidently hallucinating a solution.