💬 NLP

BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension

This paper introduces BaFCo, a benchmark dataset comprising 200 complex Bangla government forms with fine-grained annotations, to evaluate and highlight the current limitations of Multimodal Large Language Models in Bangla document layout analysis and key information extraction.

Original authors: Abu Tyeb Azad, Ishita Sur Apan, Fahim Ahmed, Sumaiya Karim Katha, Ezharuddin Jubaer, Armun Alam, Pranjal Kumar Nandi, Amin Ahsan Ali, Aman Chadha, Md Mofijul Islam, AKM Mahbubur Rahman

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abu Tyeb Azad, Ishita Sur Apan, Fahim Ahmed, Sumaiya Karim Katha, Ezharuddin Jubaer, Armun Alam, Pranjal Kumar Nandi, Amin Ahsan Ali, Aman Chadha, Md Mofijul Islam, AKM Mahbubur Rahman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of government forms from Bangladesh. These aren't just simple receipts; they are complex, multi-page documents filled with handwritten notes, official stamps, checkboxes, and dense tables. For a human, reading these is a challenge. For a computer, it's like trying to solve a puzzle while wearing blindfolds.

This paper introduces BaFCo, a new "training gym" designed to teach Artificial Intelligence (AI) how to read these specific Bangla forms.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Language Gap"

Think of AI models (like the smart brains behind ChatGPT or Gemini) as students who have studied hard in English, French, and Spanish. They are experts at reading documents in those languages. However, Bangla is like a language they barely know. Even though 284 million people speak it, there are very few "textbooks" (datasets) for AI to learn from.

Because of this, when these AI students try to read a Bangla government form, they get confused. They might miss a signature, misread a number, or fail to understand which box belongs to which question.

2. The Solution: The "BaFCo" Training Manual

The authors created BaFCo to fix this. Think of it as a specialized textbook written specifically for teaching AI about Bangla forms.

The Content: They gathered 200 real, complex government forms from sectors like banking, agriculture, and land management.
The Detail: They didn't just scan the pages; they meticulously labeled every single part. Imagine a teacher drawing a box around every single word, signature, and table cell, and then writing a note explaining what that box is.
- They defined 26 specific types of things to look for (like "Header," "Signature," "Checkbox," or "Table Row").
- They also created a 5-category "easy mode" (like grouping all "text" together) to see if the AI performs better with simpler instructions.
The Quality: They used human experts to check the work, ensuring the "textbook" is accurate. They even graded the forms by difficulty: Easy (simple lists), Medium (some tables), and Hard (messy, multi-page documents with stamps and handwriting).

3. The Test: The "AI Olympics"

The authors took the world's top AI models (the "flagship" models from companies like Google, OpenAI, Anthropic, and others) and put them through a test using the BaFCo textbook. They asked the AI two main questions:

Task A: Document Layout Analysis (DLA) - "Where is it?"
- The Analogy: Imagine the AI is a security guard looking at a crowded room. The task is to point a laser pointer at specific people (e.g., "Point to the person with the red hat").
- The Result: The AI struggled. Even the smartest models had a hard time pinpointing exactly where a specific box or signature was on the page. It's like the AI could see the room but kept tripping over its own feet trying to point at the right spot.
- Key Finding: The AI got much better at this task when the instructions were simpler (the "Easy Mode" with 5 categories) rather than the complex "Hard Mode" with 26 categories.
Task B: Key Information Extraction (KIE) - "What does it say?"
- The Analogy: Now the AI is a secretary. The boss asks, "What is the name written in the top right corner?" The AI just needs to read the text and type it out.
- The Result: The AI was much better at this. It could read the words and extract the answers quite accurately.
- Key Finding: Interestingly, the AI's performance depended heavily on the language. Some models were great at reading Bangla forms but worse at English ones, while others were the opposite.

4. The Surprising Discoveries

The paper found a few things that might seem counter-intuitive:

Thinking Harder Doesn't Always Help: The researchers tried asking the AI to "think step-by-step" (a technique called Chain-of-Thought) or to use "high reasoning" modes. Surprisingly, this didn't always make the AI better at finding the boxes on the page. Sometimes, it made them slightly worse. It seems that for finding where things are, the AI relies more on "seeing" patterns than "thinking" through logic.
Language Matters for Reading, Not for Finding: When the AI was just trying to find a box (DLA), it didn't matter much if the form was in Bangla or English. But when the AI had to read and extract the text (KIE), the language made a huge difference.
The "Coarse" Shortcut: The AI performed significantly better when the task was simplified. If you tell the AI "Find all the text," it does a good job. If you tell it "Find the specific 'Signature' field vs. the 'Date' field vs. the 'Name' field," it gets confused.

5. The Conclusion

The paper concludes that while AI is getting smarter, it still has a long way to go to understand complex, low-resource languages like Bangla, especially when it comes to the precise layout of forms.

BaFCo is now available for other researchers to use. It's like opening the doors of the training gym to everyone, hoping that by practicing on these specific Bangla forms, AI will eventually become as good at reading them as a human expert.

Where to find the data: The authors have made the dataset and code public on Hugging Face, so anyone can download the "textbook" and try to train their own AI.

Technical Summary: BaFCo Benchmark for Complex Bangla Form Comprehension

Problem Statement
Document comprehension, specifically Document Layout Analysis (DLA) and Key Information Extraction (KIE), is critical for real-world applications in banking, education, and public administration. While Multimodal Large Language Models (MLLMs) have advanced these tasks, their adoption in low-resource languages like Bangla is hindered by a scarcity of high-quality annotated data. Despite Bangla being the world's 7th most spoken language, existing benchmarks (e.g., FUNSD, XFUND) focus on English or other high-resource languages, and current Bangla datasets (e.g., BaDLAD) lack the fine-grained form-specific annotations required for complex government document processing.

Methodology
To address this gap, the authors introduce BaFCo, a benchmark dataset and evaluation framework for Bangla form comprehension.

Data Collection and Curation: The dataset consists of 200 multi-page Bangladeshi government forms sourced from diverse sectors (agriculture, education, banking, land management, etc.). The authors prioritized "quality over quantity," selecting forms with complex layouts, including multi-column structures, nested fields, tables, and handwritten or stamped regions.
Annotation Schema:
- Fine-Grained Entities: A taxonomy of 26 semantic entity types was defined, covering structural components (headers, sections), functional fields (key-value pairs, checkboxes, signatures), and auxiliary elements.
- Coarse Entities: For comparative analysis, these 26 types were mapped to 5 coarse categories.
- Relationships: The dataset includes 8,771 annotated relationships linking related fields (e.g., key-to-value, key-to-key).
- Difficulty Levels: Forms are categorized into Easy, Medium, and Hard based on layout density, visual clutter, and semantic coupling.
- KIE Subset: A specific subset of 156 forms (1,926 key-value pairs) was curated for Key Information Extraction tasks.
Annotation Process: 17 trained annotators, proficient in Bangla, labeled the data using Label Studio. The process involved independent annotation followed by expert review, achieving a Cohen's $\kappa$ of 0.974.
Evaluation Framework: The authors evaluated five flagship MLLMs (GPT-5.2, Gemini 3 Pro, Claude Opus 4.6, Qwen 3.6 Plus, Kimi K2.5) using zero-shot and Chain-of-Thought (CoT) prompting. Experiments were conducted under "Low" and "High" reasoning effort settings.
- DLA Metrics: Precision, Recall, F1, and mean Average Precision (mAP) using Intersection-over-Union (IoU) thresholds of 0.3 and 0.5.
- KIE Metrics: Precision, Recall, F1, and Normalized Edit Similarity (NES).

Key Results

Model Performance (DLA): Current MLLMs struggle with fine-grained Bangla form localization. Gemini 3 Pro achieved the best performance across most configurations, with an mAP of 0.1177 on the granular entity set and 0.2646 on the coarse set (at IoU@0.3, High Reasoning, Zero-Shot). Open-source models (Kimi K2.5, Qwen 3.6 Plus) trailed significantly, though Kimi K2.5 showed notable improvement with increased reasoning effort.
Model Performance (KIE): Models performed substantially better on KIE than on DLA. Gemini 3 Pro led on Bangla forms (F1: 0.848), while GPT-5.2 led on English forms (F1: 0.847). Open-source model Qwen 3.6 Plus demonstrated competitive performance, narrowing the gap with proprietary models.
Granularity Impact: Performance improved markedly when moving from granular (26 types) to coarse (5 types) entity sets, indicating that distinguishing fine-grained entity types is a primary challenge for current models.
Reasoning and Prompting: Increasing reasoning effort and using CoT prompting yielded mixed or marginal results. CoT often performed comparably to zero-shot prompting, and higher reasoning effort did not consistently improve geometric precision (Avg. IoU).
Language Effect: For DLA, the performance difference between Bangla and English forms was minimal ( $\Delta$ mAP $\le$ 0.02), suggesting layout detection is largely language-agnostic. However, for KIE, language differences were more pronounced, with models generally performing better on English, except for Gemini 3 Pro which favored Bangla.

Significance and Contributions
The paper claims three primary contributions:

BaFCo Dataset: The introduction of the first high-quality, fine-grained benchmark for Bangla government forms, featuring 26 entity types and annotated relationships, validated through expert review.
Benchmarking Gap: Establishing the first benchmark for DLA and KIE on Bangla forms, addressing a critical void in low-resource document understanding research.
Empirical Analysis: Providing a systematic evaluation of off-the-shelf flagship MLLMs, revealing that while these models are effective at semantic extraction (KIE), they face significant limitations in precise geometric localization (DLA) of fine-grained form elements.

The authors position BaFCo not as a general-purpose corpus, but as a specialized, high-quality resource to stimulate research into Bangla document layout analysis and form comprehension, highlighting the specific challenges low-resource languages face in adopting advanced multimodal AI.