Technical Summary: FinDocMRE

Problem Statement

While Large Multimodal Models (LMMs) have demonstrated significant proficiency in general visual tasks, their deployment in specialized financial contexts remains limited. Existing benchmarks predominantly focus on isolated chart comprehension or natural language processing (NLP) tasks, failing to capture the complexity of real-world financial analysis. Professional financial reasoning requires the integration of interwoven text, dense data tables, and visual charts across multi-page documents, often necessitating cross-page reasoning. Current datasets lack the scale and diversity to evaluate document-level multimodal reasoning, and existing generation methods often introduce text bias, yielding coarse-grained queries that lack the numerical precision required for expert-level evaluation.

Methodology

The authors introduce FinDocMRE (Financial Document-level Multimodal Reasoning Evaluation), a benchmark constructed via a semi-automated pipeline designed to balance scalability with high annotation quality. The construction process involves three primary stages:

1. Data Preparation: The dataset is derived from 2,878 bilingual (Chinese and English) financial PDF reports, including annual reports, newsletters, and industry research. A "Chart-Centric Extraction" pipeline utilizes PyMuPDF and pdfplumber to parse embedded images and text. A Joint-Rule Filtering Mechanism is applied to ensure quality:

   • Geometric Filtering: Discards layout artifacts based on aspect ratios and resolution.
   • Semantic Filtering: Uses image similarity and OCR to remove non-informative elements like logos.
   • Textual Indexing: Retains only charts explicitly referenced in the document narrative to ensure analytical relevance.

2. Visual-Centric Generation: To mitigate text bias, the authors employ a strategy where aggregated chart contexts (excluding surrounding text) are input into an LLM (Gemini-2.5-Pro). The model is prompted to generate diverse question-answer pairs and reasoning traces, prioritizing cross-chart synthesis. The generation adheres to a strict taxonomy covering 12 financial domains (e.g., Valuation, Risk Management, ESG) and 5 reasoning categories (Quantitative Calculation, Pattern Identification, Comparative Analysis, Logical Deduction, Comprehensive Synthesis).

3. Expert Verification: A Human-In-The-Loop (HITL) protocol is implemented where three senior financial researchers verify each sample. Experts audit the visual grounding (chart IDs), logical consistency, and factual accuracy. Samples are discarded if any expert identifies flaws, resulting in a final retention rate of approximately 55%. This ensures the dataset contains only high-quality, expert-verified samples.

Key Contributions

• FinDocMRE Benchmark: The first large-scale benchmark specifically addressing multi-image, document-level financial multimodal reasoning. It comprises 12,207 samples derived from 2,878 reports, supporting five distinct question types (Single Choice, Multiple Choice, Numerical Precise, Numerical Approximate, Open-Ended) and covering 12 financial domains.
• Semi-Automated Pipeline: A novel construction methodology integrating Visual-Centric Generation with Expert Verification. This approach minimizes textual noise and hallucination risks, ensuring the dataset tests genuine visual reasoning rather than text memorization.
• Comprehensive Evaluation: The paper presents a rigorous evaluation of 11 representative LMMs (including proprietary models like Gemini-3.1 Pro and GPT-5, and open-source models like Qwen3-VL) against human expert baselines. The study employs multidimensional analyses, including ablation studies on visual context, model scale, and prompt robustness.

Results

Extensive experiments reveal that current LMMs face significant challenges in document-level financial reasoning:

• Performance Ceiling: No model surpasses an overall score of 65, with the best-performing model (Gemini-3.1 Pro) achieving 64.12. This leaves a substantial gap of approximately 15 points compared to the Human Expert baseline (79.41).
• Analyst-Calculator Dichotomy: Models exhibit proficiency in semantic narrative construction (Open-Ended tasks) but struggle with precise numerical estimation and cross-page visual grounding. For instance, while GPT-5.4 leads in semantic synthesis, its performance on numerical approximation tasks is significantly lower.
• Visual Grounding Bottleneck: Performance degrades as visual dependency increases. Models show a marked decline in accuracy when required to synthesize information from multiple images (e.g., dropping from ~61% on single-image tasks to ~57% on tasks requiring ≥3 images).
• Scale and Architecture: Scaling laws persist within model generations (e.g., Qwen3-VL-235B outperforms the 30B variant), but architectural evolution and multimodal alignment appear more critical than raw parameter count for financial reasoning.
• Context Sensitivity: Providing cropped images significantly improves performance compared to full PDF pages, indicating that locating dispersed visual evidence in extensive documents remains a major challenge. Conversely, adding bounding box hints often degrades performance due to visual occlusion.

Significance

The paper positions FinDocMRE as a rigorous standard to guide the evolution of financial LMMs. The authors argue that the current limitations highlight a need to shift from end-to-end generation paradigms toward agentic architectures capable of long-context grounding and tool-assisted calculation. By identifying the "visual grounding bottleneck" and the divergence between qualitative synthesis and quantitative precision, FinDocMRE provides a necessary foundation for developing trustworthy financial AI capable of expert-level document analysis. The benchmark serves not only as an evaluation metric but as a diagnostic tool to isolate specific deficiencies in visual grounding and logical reasoning within complex document environments.