Technical Summary: UniFinEval

Problem Statement

Despite the rapid advancement of Multimodal Large Language Models (MLLMs), existing evaluation benchmarks fail to adequately assess their capabilities within the high-stakes, high-information-density environment of the financial sector. Current limitations include:

• Modality Restriction: Many benchmarks (e.g., FinVQA, Fin-Fact) are limited to single-modality evaluations or static chart understanding, lacking the integration of text, images, and videos required for real-world analysis.
• Data Fidelity: Existing datasets often rely on simplified, truncated, or textbook-derived data, or utilize LLMs for annotation, which introduces potential bias and hallucinations. This fails to capture the noisy, unstructured, and dynamic nature of actual financial markets.
• Reasoning Depth: Current benchmarks often lack the rigorous multi-hop reasoning and cross-modal consistency checks necessary for complex tasks like risk sensing and asset allocation, leading to evaluation results that do not reflect true model boundaries in practical applications.

Methodology

To address these gaps, the authors propose UniFinEval, the first unified multimodal benchmark specifically designed for high-information-density financial environments.

Dataset Construction

• Human-Centric Curation: The dataset consists of 3,767 high-quality bilingual (Chinese and English) question-answer pairs. Crucially, all data was manually constructed by a team of ten senior financial experts (holding CFA/CPA certifications) with no involvement of LLMs in the generation or validation process to eliminate model-induced bias.
• Five Core Scenarios: The benchmark is grounded in real-world financial workflows, covering a progressive cognitive loop from perception to decision-making:
  1. Financial Statement Auditing (FSA): Verifying accuracy and consistency in high-density reports with complex layouts.
  2. Company Fundamental Reasoning (CFR): Analyzing intrinsic value through multi-source parameter extraction and multi-step calculations.
  3. Industry Trend Insights (ITI): Cross-enterprise and cross-period analysis using reports and macroeconomic data.
  4. Financial Risk Sensing (FRS): Identifying implicit risk signals by aligning dynamic video content with quantitative text data.
  5. Asset Allocation Analysis (AAA): Synthesizing insights from all prior stages to formulate strategies under complex constraints.
• Modality Integration: The benchmark supports full-modality inputs (text, images, videos) and complex cross-modal combinations (text-image, text-video, image-video, and text-image-video), featuring dedicated designs for cross-modal consistency checking and multi-hop reasoning.

Experimental Setup

• Models Evaluated: 10 mainstream MLLMs were tested, including 4 closed-source models (Gemini-3-pro-preview, GPT-5.1, Grok-4.1, Claude-Sonnet-4.5) and 6 open-source models (various Qwen3-VL, InternVL3.5, MiniCPM-V, Llama-3.2-Vision).
• Evaluation Settings: Models were assessed under Zero-Shot and Zero-Shot Chain-of-Thought (CoT) settings.
• Metrics: Accuracy was the core metric. To handle complex financial reasoning, Qwen-Max was utilized to standardize output extraction, with a manual inspection error rate of less than 1%.

Key Results

• Overall Performance: Gemini-3-pro-preview achieved the best overall performance with an average accuracy of 73.8% in the Zero-Shot setting, followed by Qwen3-VL-235B-Thinking (66.9%).
• Expert Gap: Despite high scores on perception tasks, all models exhibit a substantial gap compared to human financial experts (who scored 97.5% in FSA and 85.2% in AAA). The performance gap widens significantly as task complexity increases from perception to decision-making.
• Task-Specific Challenges:
  • Models perform relatively well in FSA and CFR (perception and basic reasoning) but struggle significantly in ITI (73.3% for top model vs. 90.1% for experts) and FRS (video-based tasks), indicating deficiencies in temporal modeling and cross-modal semantic alignment.
  • In AAA (Asset Allocation), the top model achieved only 61.1% accuracy, highlighting a failure to maintain long-term logical consistency under high information density.
• CoT Impact: The introduction of CoT reasoning yielded only marginal improvements for most models, suggesting that current MLLMs are already capable of adapting to explicit reasoning mechanisms without significant performance gains from the setting itself.

Error Analysis

A systematic analysis of incorrect predictions revealed five primary error categories:

1. Perception: Difficulties in interpreting financial images and data.
2. Hallucination: Inconsistent financial reasoning and fabrication of facts.
3. Knowledge: Deficiencies in domain-specific financial knowledge.
4. Cross-modal: Failures in integrating and aligning data across text, image, and video.
5. Computation: Errors in financial numerical analysis.
   The analysis indicates that current models are particularly weak in cross-modal data alignment and fine-grained multi-hop reasoning required for high-density financial environments.

Significance and Contributions

The paper claims the following contributions:

1. Unified Benchmark: UniFinEval is the first multimodal unified financial evaluation benchmark that deeply integrates text, images, and videos, providing a standardized paradigm for assessing MLLMs in the financial domain.
2. Rigorous Reasoning Design: It introduces multi-hop reasoning questions from a unique multi-source information fusion perspective, posing a rigorous challenge to model performance that reflects real-world business logic.
3. Practical Error Analysis: The work provides a targeted error analysis that identifies specific failure modes in high-information-density scenarios, offering practical directions for improving MLLM robustness in financial applications.

The authors conclude that while MLLMs show promise in information extraction, they currently lack the robustness, precision, and decision-making capabilities required for real-world financial operations, necessitating further development in cross-modal alignment and complex reasoning.