FCMBench: The First Large-scale Financial Credit Multimodal Benchmark for Real-world Applications

FCMBench introduces the first large-scale, privacy-compliant multimodal benchmark for real-world financial credit applications, featuring 5,198 synthetic images and 13,806 VQA samples to rigorously evaluate 28 state-of-the-art vision-language models on perception, reasoning, and robustness tasks, revealing significant performance gaps and the need for further domain-specific advancements.

The Gist

Imagine you are a loan officer at a bank. Every day, people apply for loans, and you have to check their "proof of life" documents: their ID cards, bank statements, tax returns, and property deeds. Your job is to look at these papers, make sure they aren't fake, check if the numbers add up, and decide if the person is trustworthy.

For a long time, computers (AI) have been getting better at reading text and looking at pictures. But nobody knew if they were actually good at this specific job because:

1. Privacy: Banks can't share real customer documents with researchers (it's like showing your diary to a stranger).
2. Complexity: It's not just about reading the words; it's about connecting the dots. Does the income on the tax form match the bank statement? Is the ID card expired?

Enter FCMBench. Think of this paper as the introduction of the "Ultimate Driver's License Test" for AI, but instead of driving a car, the AI is trying to approve a loan.

Here is the breakdown in simple terms:

1. The Problem: The "Privacy Wall"

Previously, researchers trying to build AI for finance were stuck. They had to use fake, low-quality data or public documents that didn't look like real bank paperwork. It was like trying to teach a pilot to fly a plane using a toy airplane. They didn't know if the AI would crash when faced with a real, messy, blurry photo of a bank statement.

2. The Solution: The "Fake Real" World

The team behind FCMBench (from Qifu Technology and Fudan University) built a massive, secret laboratory.

• The Actors: They created 26 different types of "fake" documents (IDs, deeds, tax forms) using computer code.
• The Actors' Lives: They invented 5,000+ fictional people with fake names, fake jobs, and fake bank accounts.
• The Twist: They didn't just take a screenshot. They printed these fake documents out, held them up to a camera, and took photos in real life. They even took photos with bad lighting, blurry focus, and crooked angles.
• Why? This creates a "Goldilocks" dataset: it looks and feels exactly like a real loan application, but because the people and numbers are made up, there is zero risk of leaking real private data.

3. The Exam: What is the AI Being Tested On?

The benchmark puts the AI through two main types of tests, mimicking a real loan officer's day:

• The "Eagle Eye" Test (Perception):
  • Can you see it? Is the photo too blurry? Is the light reflecting off the plastic ID card so you can't read it?
  • What is it? Is this a driver's license or a marriage certificate?
  • Can you read it? Extract the specific numbers (like the ID number or the salary).
• The "Detective" Test (Reasoning):
  • Does it make sense? If the ID says the person is 20, but the tax form says they've been working for 30 years, the AI needs to catch that lie.
  • Do the pieces fit? Does the address on the utility bill match the address on the bank statement?
  • Is it valid? Did the document expire yesterday?

4. The "Stress Test" (Robustness)

Real life is messy. People take photos of their documents with their phones while rushing to the bank. The lighting is bad, the photo is crooked, or the paper is crumpled.
FCMBench tests the AI with 10 different types of "messiness" (like a photo taken from a weird angle or a photo of a photo on a computer screen).

• The Result: Even the smartest AIs (like Google's Gemini or Kimi) started to stumble when the photos were messy. They got confused, just like a human would if squinting at a blurry receipt.

5. The Results: Who Won?

The researchers tested 28 different AI models (the "students").

• The Top Student: Gemini 3 Pro (a commercial model) got the highest score, but even it only got about 65% correct.
• The Open-Source Star: Kimi-K2.5 was the best of the free/open models, scoring around 60%.
• The Reality Check: The average score was only 45%. This means the test is hard. It proves that current AI is not yet ready to fully replace human loan officers without supervision. They are good at reading, but they still struggle with the "detective work" and messy photos.

Why Does This Matter?

Think of FCMBench as a standardized fitness test for AI in the financial world.

• Before: Banks were guessing if their AI was good.
• Now: They have a ruler to measure exactly how well an AI can handle a real loan application.
• The Future: By making this test public (open-source), the paper invites scientists and companies to work together to build AI that is not just "smart," but reliable, safe, and ready for the real world.

In a nutshell: This paper built a safe, realistic training ground to see if AI can actually do the boring but critical job of checking loan documents, and it found that while AI is getting better, it still has a long way to go before it can be trusted to handle the messiness of real life on its own.

Technical Summary

1. Problem Statement

Financial credit review is a critical workflow where loan approval decisions rely on verifying diverse borrower-uploaded documents (e.g., ID cards, bank statements, tax returns). While Multimodal Large Language Models (MLLMs) show promise in automating these tasks, their deployment is hindered by a lack of high-quality, domain-specific benchmarks.

• Gap in Existing Benchmarks: General benchmarks (e.g., MME, MMMU) lack financial specificity. Document understanding benchmarks (e.g., OCRBench, WildDoc) focus on OCR or charts but ignore credit-specific workflows. Existing financial benchmarks (e.g., CFBenchmark-MM) prioritize financial knowledge analysis over the practical, decision-oriented credit review process and often suffer from privacy constraints or reliance on fragmented open-source data.
• Real-World Challenges: Credit review involves strict privacy requirements, complex cross-document reasoning, and significant robustness challenges (e.g., poor lighting, blurry scans, cropped images) that current models struggle to handle.

2. Methodology: FCMBench Construction

The authors introduce FCMBench-V1.1, a privacy-compliant, large-scale multimodal benchmark designed to mirror real-world credit review workflows.

A. Data Generation & Privacy Compliance

To address privacy concerns while ensuring realism, the team employed a "Synthetic-to-Physical" workflow:

1. Template Synthesis: Created high-fidelity synthetic templates for 26 certificate types (17 Chinese, 9 English) using fictional identities with diverse demographics (age, income, marital status).
2. Physical Fabrication: Printed these templates on physical media (cards or A4 paper) to ensure realistic textures, dimensions, and material properties.
3. On-Site Photography: A team of 11 participants captured images under ecologically valid conditions.
4. Privacy Measures: All personal data (names, logos, seals) are fictionalized. No real-world images are used. The dataset includes 5,198 images and 13,806 paired VQA samples.

B. Task Taxonomy

The benchmark evaluates models across three dimensions, aligned with the credit review lifecycle:

1. Perception Tasks (3 types):
   • Image Quality Evaluation (IQE): Detecting defects like blur, glare, or out-of-focus issues.
   • Document Type Recognition (DTR): Identifying document categories and checking for completeness.
   • Key Information Extraction (KIE): Extracting specific fields (e.g., ID numbers, income amounts) for profile construction.
2. Reasoning Tasks (4 types):
   • Consistency Checking (CC): Cross-validating information across multiple documents (e.g., matching ID with Household Register).
   • Validity Checking (VC): Verifying compliance with policy rules (e.g., expiration dates, format validity).
   • Numerical Calculation (NC): Aggregating and transforming numeric data (e.g., calculating total income from bank statements).
   • Rationality Review (RR): Assessing economic plausibility (e.g., does the stated income match the tax certificate?).
3. Robustness Challenges (10 types):
   • Models are tested under realistic interferences including off-axis viewpoints, uneven illumination, specular reflections, out-of-focus blur, small regions of interest (ROIs), secondary captures (screenshots), cluttered backgrounds, watermarks, cropped images, and multi-document images.

C. Evaluation Metrics

• Exact-Match Oriented: Due to the atomic nature of financial data (e.g., an incorrect ID digit is a failure), the benchmark uses strict exact-match evaluation rather than semantic equivalence.
• Normalization: Outputs are normalized into key-value sets to handle diverse formats (lists, tables, strings).
• Scoring: Precision, Recall, and F1 scores are calculated at instance, subtask, and task levels. Robustness is measured as a relative F1 ratio compared to "Normal Captures."

3. Key Contributions

1. First Large-Scale Credit Benchmark: FCMBench is the first multimodal benchmark explicitly tailored for financial credit, featuring 26 certificate types and a fully synthetic-to-physical data generation pipeline to ensure privacy and uniqueness.
2. Application-Oriented Evaluation System: It moves beyond simple OCR or classification to evaluate end-to-end credit workflows, integrating perception, complex reasoning, and robustness stress testing.
3. Academia-Industry Bridge: By open-sourcing the dataset and code, it provides a standardized metric for financial institutions to compare AI models and offers researchers a high-quality, privacy-safe dataset for domain-specific studies.

4. Experimental Results

The authors evaluated 28 State-of-the-Art (SOTA) Vision-Language Models (released 2025–2026) from 14 organizations, including commercial giants (Gemini, GPT, Claude) and open-source leaders (Qwen, Kimi, Llama).

• Overall Performance:

  • The benchmark is non-trivial, with a mean F1 score of 44.8 ± 10.3 across all models.
  • Top Performers: Gemini 3 Pro achieved the highest commercial score (65.16), while Kimi-K2.5 led the open-source baselines (60.58).
  • Scaling Laws: Performance correlates positively with model size for both Dense and Mixture-of-Experts (MoE) architectures.

• Task-Specific Insights:

  • Perception vs. Reasoning: Models generally excel at Document Type Recognition (DTR) but struggle significantly with Key Information Extraction (KIE) and complex Reasoning tasks. Even top models score below 50 on KIE, indicating a bottleneck in precise data extraction.
  • Reasoning Variance: There is high variance in reasoning tasks (CC, VC, NC, RR), suggesting that while models can "read" text, they differ greatly in their ability to integrate evidence and apply domain constraints.

• Robustness Findings:

  • All models suffer performance degradation under robustness challenges.
  • Even top-performing models are vulnerable to specific artifacts (e.g., severe blur, multi-doc compositions).
  • The gap between "Normal Captures" and "Robustness Challenges" highlights that current models are not yet robust enough for uncurated real-world user uploads without mitigation strategies.

5. Significance and Future Work

• Significance: FCMBench establishes a rigorous standard for evaluating AI in the high-stakes financial credit sector. It shifts the focus from general multimodal capabilities to domain-specific reliability, privacy compliance, and robustness.
• Implications: The results suggest that before deploying VLMs in credit workflows, developers must prioritize robustness testing and implement quality gates (e.g., re-capture prompts) to handle real-world image noise.
• Future Directions: The authors plan to expand the benchmark to include digital formats (screenshots, PDFs), non-image modalities (audio/video), and further diversify certificate categories to match evolving credit risk workflows.

In conclusion, FCMBench provides a critical tool for advancing "Credit AI," moving the field from single-point optimization toward collaborative, reliable, and real-world applicable systems.