Technical Summary: BizFinBench.v2

Problem Statement

Large Language Models (LLMs) are increasingly deployed in financial applications; however, a significant gap exists between their reported performance on existing benchmarks and their actual efficacy in real-world business scenarios. The paper identifies two fundamental limitations in current financial evaluation frameworks:

1. Disconnection from Real-World Business: Most benchmarks rely on simulated, synthetic, or general-purpose datasets that lack the complexity, noise, and specific workflow logic of authentic financial operations.
2. Static Offline Focus: Existing benchmarks predominantly evaluate static, offline tasks, failing to capture the "online capability" essential for financial services, such as real-time market data processing, dynamic risk monitoring, and instantaneous decision-making.

This disconnect leads to a misleading assessment of model readiness, where high benchmark scores do not translate to reliable performance in practical investment or advisory contexts.

Methodology

To address these gaps, the authors introduce BizFinBench.v2, the first integrated offline and online benchmark constructed from authentic user query-response data from both Chinese and U.S. equity markets.

Data Construction and Quality Control

• Source: The dataset comprises 28,860 questions derived from real user interactions on financial business platforms.
• Scenarios: The data is categorized into four core business scenarios:
  1. Business Information Provenance (BIP): Frontline inquiries requiring robust discernment of erroneous inputs and multi-turn interactions.
  2. Financial Logical Reasoning (FLR): Tasks involving quantitative computation, event logic, and counterfactual inference.
  3. Stakeholder Feature Perception (SFP): Analysis of user sentiment and financial reports.
  4. Real-time Market Discernment (RMD): Online tasks involving stock price prediction and portfolio management.
• Quality Control: A rigorous three-level mechanism was employed:
  1. Platform Clustering & Desensitization: Automated aggregation and removal of PII/sensitive data.
  2. Frontline Review: Manual screening by 10 professionals to filter invalid or duplicate pairs while retaining realistic perturbations.
  3. Expert Cross-Validation: Validation by six senior financial experts across three teams to ensure business alignment and compliance.

Task Design

The benchmark includes eight offline tasks and two online tasks:

• Offline Tasks: Anomaly Information Tracing, Financial Multi-turn Perception, Financial Data Description, Financial Quantitative Computation, Event Logic Reasoning, Counterfactual Inference, User Sentiment Analysis, and Financial Report Analysis.
• Online Tasks:
  • Stock Price Prediction (SPP): Predicting daily closing prices with a 1% tolerance interval.
  • Portfolio Asset Allocation (PAA): A simulation system restoring real-world trading rules (fees, latency, slippage) where models make hourly investment decisions over a specific period.

Evaluation Setup

• Models Evaluated: 21 LLMs, including 6 proprietary models (e.g., GPT-5, Gemini-3, DeepSeek-R1), 11 open-source general models, and 4 domain-specific financial models.
• Metrics:
  • Offline: Accuracy (with specific tolerance intervals for sentiment and price prediction).
  • Online (PAA): Investment indicators including Total Return (TR), Profit Factor (PF), Sharpe Ratio (SR), and Maximum Drawdown (MD).
• Baselines: Commercial models were compared against traditional quantitative strategies (e.g., Moving Average Crossover, Equal-Weight Allocation, Momentum).

Key Contributions

1. Authentic Business Data-Driven Benchmark: BizFinBench.v2 is the first large-scale evaluation framework driven entirely by real-world business data from dual markets (China and U.S.), moving beyond synthetic or low-complexity datasets.
2. Innovative Dual-Track Evaluation Framework: The benchmark uniquely integrates "Core Business Capabilities" (offline) with "Online Performance" (real-time decision making), addressing the coverage deficiencies of traditional static assessments.
3. Expert-Informed Business Error Analysis: The paper provides a granular error analysis from the perspective of financial experts, identifying five specific business dilemmas: Financial Semantic Deviation, Long-term Business Logic Discontinuity, Multivariate Integrated Analysis Deviation, High-precision Computational Distortion, and Financial Time-Series Logical Disorder.

Experimental Results

• Overall Performance: Even the top-performing model, GPT-5, achieved only 61.5% average accuracy, falling significantly short of the practical business requirement threshold of 84.8% (established by human financial experts).
• Model Comparisons:
  • Proprietary Models: GPT-5, Gemini-3, and Doubao-Seed-1.6 ranked highest overall. However, performance varied widely across tasks; for instance, Claude-Sonnet-4 and Grok-4 struggled significantly with quantitative and reasoning tasks.
  • Open-Source Models: Qwen3-235B-A22B-Thinking-2507 led the open-source category (53.3% accuracy), outperforming specialized financial models.
  • Domain-Specific Models: Financial-specific models (e.g., Dianjin-R1) underperformed compared to general-purpose models of similar scale, attributed to narrow training data and lack of exposure to complex, volatile real-world scenarios.
• Online Task (PAA) Results:
  • DeepSeek-R1 demonstrated superior investment efficacy among commercial models, achieving a 47.34% total return and a Sharpe ratio of 1.25. However, it exhibited high risk with a 53% maximum drawdown.
  • Other commercial models (GPT-5, Gemini-3, etc.) generally underperformed, with some showing negative returns and drawdowns exceeding 10%.
  • Comparison with Quant Strategies: Most LLMs underperformed compared to traditional quantitative strategies (e.g., Equal-Weight and Momentum strategies). Only DeepSeek-R1 approached the performance of mid-tier quantitative strategies, while others lagged significantly in alpha generation and risk mitigation.
• CoT Impact: Contrary to expectations, Chain-of-Thought (CoT) prompting did not universally improve performance. For many models (e.g., Claude-Sonnet-4), CoT led to a significant performance decay, suggesting that extended reasoning chains can introduce logical fragmentation in complex financial contexts.

Significance and Claims

The paper posits that BizFinBench.v2 provides a substantiated foundation for advancing LLM deployment in the financial sector by shifting the evaluation paradigm from "test-taking proficiency" to "actual business capability."

• Realism over Simulation: By utilizing authentic user data, the benchmark reveals that current state-of-the-art models are not yet ready for high-stakes financial applications without significant optimization.
• Diagnostic Utility: The identified error categories (e.g., semantic deviation, computational distortion) offer targeted directions for model improvement, moving beyond generic accuracy metrics to specific business logic failures.
• Future Direction: The authors conclude that future assessments must be grounded in actual business scenarios. The release of the benchmark data, code, and the LLM investment simulation system aims to facilitate the development of more reliable, business-aligned financial AI.