When Tables Go Crazy: Evaluating Multimodal Models on French Financial Documents

1. Problem Statement

Vision-Language Models (VLMs) have demonstrated strong capabilities in general document understanding, yet their reliability in specialized, non-English domains remains underexplored. This gap is critical in the financial sector, where documents (e.g., prospectuses, KIDs, PRIIPs) are:

• Multimodal: They combine dense legal text, complex numerical tables, and visual charts.
• High-Stakes: Extraction errors can lead to significant regulatory and financial consequences.
• Language-Specific: Most existing benchmarks focus on English, leaving French financial document understanding largely untested.
• Interactive: Real-world financial analysis often requires multi-turn dialogue, where errors can propagate.

Current benchmarks are often too small, English-centric, or lack multimodal depth (specifically charts and conversational reasoning), making it difficult to assess if state-of-the-art models are ready for deployment in these scenarios.

2. Methodology: MULTIMODAL FINANCE EVAL

To address these gaps, the authors introduced MULTIMODAL FINANCE EVAL, the first multimodal benchmark for French financial document understanding.

Dataset Construction

• Source Material: 1,204 expert-validated questions derived from real investment prospectuses, Key Information Documents (KIDs), and PRIIPs from French asset management companies (spanning 15 years).
• Question Generation: A semi-automatic pipeline using LLMs (GPT-4o, Gemini-2.0) to generate candidate questions, followed by rigorous manual revision by a French financial domain expert.
  • 75% of generated questions were reformulated or removed to ensure quality, grounding, and diversity.
• Task Categories: The dataset covers six distinct tasks:
  1. Text Question: Extraction from pure text.
  2. Table Comprehension: Reasoning over structured tabular data.
  3. Chart Interpretation: Analyzing graphical financial representations.
  4. Special Cases: Nuanced terminology or implicit reasoning.
  5. Conversational (Gold Context): Multi-turn dialogue with correct previous answers (Oracle).
  6. Conversational (Model Context): Multi-turn dialogue where the model must rely on its own previous (potentially incorrect) answers.
• Question Types: Open-ended, Yes/No, True/False, and Multiple Choice.

Experimental Setup

• Models Evaluated: Six open-weight, state-of-the-art VLMs ranging from 8B to 124B parameters:
  • Qwen: Qwen3-VL-8B, Qwen3-VL-32B
  • Gemma: Gemma-3-12B, Gemma-3-27B
  • Pixtral: Pixtral-12B, Pixtral-Large-124B
• Evaluation Protocol: An LLM-as-Judge approach was used to evaluate open-ended responses. Three independent judge models (Llama-3.3-70B, Qwen3-32B, Gemma-3-27B) assessed answers, with a majority vote determining correctness to ensure robustness.

3. Key Results

The evaluation revealed a stark contrast between performance on well-defined extraction tasks and complex reasoning tasks.

• Text and Table Performance (Strong):
  • Models achieved high accuracy on text extraction (89–90%) and table comprehension (52–86%).
  • This indicates that current VLMs are effective at locating explicit information and structured data when the task is single-step.
• Chart Interpretation (Weak):
  • Performance dropped significantly for chart-based questions, ranging from 34% to 62%.
  • Even the best-performing model (Qwen3-VL-32B) only reached ~62%, while others struggled below 50%. Models struggle to infer trends and proportions from graphical elements compared to text or tables.
• Multi-Turn Conversational Failure (Critical):
  • Gold Context (Oracle): When provided with correct previous answers, models performed well (63–86%).
  • Model Context (Self-Generated): When models had to build on their own previous turns, accuracy collapsed to ~46–59% across all models, regardless of parameter size.
  • Scaling Law Breakdown: Increasing model size (from 8B to 124B) did not mitigate this failure. The error propagation effect negated the benefits of scaling.

4. Key Contributions

1. First French Multimodal Financial Benchmark: Introduction of MULTIMODAL FINANCE EVAL, a dataset of 1,204 expert-validated questions covering text, tables, charts, and multi-turn dialogue.
2. Systematic VLM Evaluation: A comprehensive evaluation of six SOTA models, highlighting that while text/table extraction is mature, chart interpretation and interactive reasoning remain major bottlenecks.
3. Discovery of Error Propagation: Empirical evidence that in multi-turn financial dialogue, early mistakes lead to a "cascading failure" where accuracy converges to ~50% regardless of model scale, suggesting that current architectures lack robust mechanisms for error correction in interactive settings.

5. Significance and Implications

• Reliability Concerns: The results suggest that while VLMs are suitable for static, well-scoped information extraction (e.g., "What is the entry fee?"), they are currently too brittle for interactive financial advisory where a user might ask follow-up questions based on previous answers.
• Beyond Scaling: The failure of larger models to solve the conversational error propagation problem indicates that simply increasing parameter count is insufficient. Future progress requires architectural changes focused on uncertainty modeling, error detection, and robust multi-step reasoning.
• Benchmark Utility: The dataset serves as a critical testbed for the financial AI community to measure progress in high-stakes, multimodal, and non-English environments, moving beyond simple text-based QA.

In conclusion, the paper argues that the path to reliable financial AI requires moving beyond single-step extraction capabilities to address the fragility of multimodal reasoning and the accumulation of errors in conversational contexts.