FIND: Toward Multimodal Financial Reasoning and Question Answering for Indic Languages

Technical Summary: FIND: Toward Multimodal Financial Reasoning and Question Answering for Indic Languages

Problem Statement
Financial decision-making in multilingual contexts, particularly within India, requires accurate numerical reasoning grounded in diverse modalities (text, tables, charts). However, existing benchmarks largely overlook this high-stakes challenge, focusing primarily on English and lacking coverage of Indic languages, multimodal financial data, and controlled reasoning complexity. Current Vision-Language Models (VLMs) often struggle with hybrid reasoning, exhibiting errors in numerical accuracy, inconsistencies in multimodal grounding, and an over-reliance on textual cues while underutilizing visual financial signals. Furthermore, existing datasets like FinQA and TAT-QA are predominantly English-centric and often limited to shallow arithmetic or tabular extraction, failing to evaluate the complex, domain-aware reasoning required for realistic financial scenarios in regional languages.

Methodology
To address these gaps, the authors introduce FinVQA, a large-scale multilingual benchmark, and FIND, a framework designed for reliable multimodal financial reasoning.

• FinVQA Dataset Curation:

  • Source: Data is curated from authoritative Indian educational sources, including National Council of Educational Research and Training (NCERT) textbooks (Classes 9–12) and ICMAI–CMA professional materials.
  • Scale and Scope: The dataset comprises 18,900 samples across 14 financial domains (e.g., Advance Accounting, Financial Mathematics, Taxation). It covers six languages: English, Hindi, Bengali, Marathi, Gujarati, and Tamil.
  • Structure: Samples are organized into three difficulty levels (easy, moderate, hard) and four question formats: multiple-choice, fill-in-the-blank, table matching, and true/false.
  • Multilingual Adaptation: English instances were translated into five Indic languages using GPT-4O, validated via sentence embedding cosine similarity, and refined by native speakers. For multimodal samples, OCR (PaddleOCR) was used to extract text from images, which was then translated and re-rendered into the original bounding boxes using specific Indic fonts to preserve layout.
  • Annotation: A pipeline involving six annotators (four junior, two senior domain experts) ensured quality. Senior experts annotated seed samples, and inter-annotator agreement (Fleiss' Kappa) reached 0.78.

• The FIND Framework:
  The authors propose a three-stage experimental pipeline to evaluate and improve reasoning:

  1. Zero-Shot Inference: Evaluating pretrained VLMs without task-specific adaptation to establish baselines.
  2. Constrained Decoding: Implementing inference-time constraints (using VLLM) to restrict output to valid answer option tokens (e.g., A/B/C/D). This mitigates formatting instability and verbosity while preserving internal reasoning. Language-specific token constraints are applied (e.g., single-token for English/Hindi, up to three for Bengali/Tamil).
  3. Supervised Fine-Tuning (SFT): Fine-tuning VLMs on the dataset using answer-only supervision. To ensure parameter efficiency, the authors employ Low-Rank Adaptation (LoRA) with rank $r=8$, updating only the query, key, and value projections in self- and cross-attention layers. The model is trained to generate the correct answer sequence conditioned on the input.

Key Contributions

1. FinVQA Benchmark: The introduction of the first large-scale, multilingual benchmark for financial visual question answering in Indic contexts, featuring 18,900 samples across 14 domains and six languages.
2. FIND Framework: A constraint-aware framework integrating supervised fine-tuning with structured decoding to enhance numerical faithfulness, multimodal alignment, and structured decision-making.
3. Comprehensive Evaluation: A rigorous evaluation of state-of-the-art VLMs (Qwen2.5-VL, Qwen3-VL, Gemma-3) across varying parameter scales (3B to 32B) and training regimes (zero-shot, constrained decoding, SFT).

Experimental Results

• Performance Hierarchy: Model performance is strongly governed by scale and training regime. Large-scale models (e.g., Qwen3-VL-32B, Gemma-3-27B) significantly outperform smaller counterparts.
• Impact of SFT: Supervised Fine-Tuning exerts the strongest influence on performance, yielding substantial gains over zero-shot baselines, particularly for Indic languages. For instance, SFT improved Marathi accuracy for Qwen3-VL-4B from 26.2% (zero-shot) to 47.8%.
• Role of Constrained Decoding: While constrained decoding aids structural validity, it provides modest gains compared to SFT. However, combining SFT with constrained decoding yields the most robust results.
• Language Trends: Hindi and Bengali generally achieve the highest accuracies, reflecting better resource availability. Tamil remains the most challenging due to typological distance and script complexity, though SFT still provides consistent improvements.
• Human Evaluation: Human assessments reveal that while formatting compliance is achievable at lower compute scales, high-fidelity financial reasoning (domain understanding, problem interpretation, mathematical correctness) is strictly capacity-dependent. Large models (27B–32B) approach near-ceiling performance (0.9–1.0 scores), whereas smaller models (3B–4B) struggle with logical coherence and visual grounding.
• Error Analysis: Smaller models frequently fail in visual grounding and logical consistency. Intermediate models (7B–12B) show improved parsing but suffer from brittle mathematical execution. Only large-scale models (27B–32B) demonstrate robust, end-to-end execution of visual-mathematical tasks.

Significance and Claims
The paper posits that reliable, high-stakes financial reasoning in multilingual settings requires a combination of large-scale model capacity, supervised alignment with domain-specific data, and strict output constraints. The authors claim that FIND and FinVQA establish a rigorous evaluation paradigm for multimodal financial reasoning, highlighting that current models often exhibit "fluency without correctness." They emphasize that while constraint-aware decoding improves format compliance, it does not guarantee reasoning accuracy, underscoring the necessity of human oversight and the limitations of current models in prescriptive financial decision-making. The work serves as a foundational step toward bridging the gap in financial literacy tools for Indic language speakers, though the authors acknowledge limitations regarding sample diversity (derived from translated sources) and the exclusion of closed-source baselines due to resource constraints.