💬 NLP

Khondo: A Multimodal Benchmark for Document Packet Splitting of Bangla Forms

The paper introduces Khondo, the first vision-native, bilingual benchmark for splitting and reordering concatenated Bangla government forms, revealing that while multimodal large language models can effectively cluster pages by document, they struggle significantly with reconstructing the original page order, particularly in low-resource languages.

Original authors: Abu Tyeb Azad, Fahim Ahmed, Ishita Sur Apan, Ezharuddin Jubaer, Sumaiya Karim Katha, Armun Alam, Amin Ahsan Ali, Aman Chadha, Md Mofijul Islam, AKM Mahbubur Rahman

Published 2026-07-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Abu Tyeb Azad, Fahim Ahmed, Ishita Sur Apan, Ezharuddin Jubaer, Sumaiya Karim Katha, Armun Alam, Amin Ahsan Ali, Aman Chadha, Md Mofijul Islam, AKM Mahbubur Rahman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian in a chaotic, magical library where books don't just sit on shelves; they sometimes get glued together into giant, messy scrolls, and then someone shakes the table, mixing the pages of different stories until they are completely jumbled. Your job is to look at this tangled mess, figure out which pages belong to which story, and then re-stack them in the correct order so the story makes sense again. This is the real-world problem of "document packet splitting." In the digital world, governments and offices often scan hundreds of papers into one big file. Sometimes, the scanner gets confused, or a worker accidentally shuffles the pages, mixing a birth certificate with a tax form or a license with a medical report. For computers, this is a nightmare. While we have built smart AI that can read text, teaching it to look at a picture of a page, understand the visual clues, and then untangle a shuffled pile of documents is a much harder challenge, especially when the documents are written in languages that computers don't see very often.

This is where a new project called Khondo (which means "split" or "segment" in Bangla) comes in. The researchers behind this project created a special test set, like a giant puzzle box, specifically designed to see how well modern AI can untangle these messy document piles. They didn't just use English; they used real government forms from Bangladesh, which are often written in Bangla, English, or a mix of both. They took thousands of pages, glued them together into fake "packets," and then deliberately shuffled the pages in five different ways: from keeping them in order, to mixing them up within a single document, to completely scrambling pages from different documents together. They then asked the world's most advanced AI models to look at the pictures of these pages and try to sort them out.

The results were a mix of "not bad" and "oh no." The AI models were surprisingly good at the first part of the job: they could look at a jumbled pile and correctly guess which pages belonged together. It was like the AI could tell, "Hey, these four pages are all about farming, and these three are about taxes," even when the pages were mixed up. However, the AI hit a massive wall when it came to the second part: putting the pages back in the right order. When the pages were shuffled, the AI struggled to figure out which page came first, second, or last. It was as if the AI could identify the characters in a story but couldn't remember the plot sequence.

The researchers dug deeper to find out why this was happening. They tried two main tricks. First, they gave the AI very specific instructions, telling it, "Hey, these pages are mixed up; please fix the order!" This helped a lot, but it didn't solve the problem completely. The AI still made mistakes, suggesting that the difficulty wasn't just about the AI not listening, but about the task itself being genuinely hard. Second, they tested the AI with English-only packets and Bangla-only packets. They found that the AI was much better at ordering the English pages than the Bangla ones. It seems that reading the visual clues in the Bangla script was an extra layer of difficulty that slowed the AI down, even though it could still group the pages correctly.

In short, this paper shows that while AI is getting great at recognizing what a document is about, it is still struggling to act like a human librarian who can look at a shuffled stack of papers and instantly know the correct reading order, especially when the text is in a language like Bangla. The researchers have now made this "messy library" puzzle available for everyone to use, hoping that by studying these failures, future AI will learn to untangle the knots and restore order to our digital documents.

Technical Summary: Khondo – A Multimodal Benchmark for Document Packet Splitting of Bangla Forms

1. Problem Definition

In government and administrative workflows, documents are frequently processed as "packets"—single files containing multiple concatenated documents (forms and supporting materials) that may appear out of their original order. Recovering the constituent documents involves three tasks: clustering (grouping pages by source document), typing (classifying document domains), and ordering (reconstructing the original page sequence).

While prior work has addressed document splitting, existing benchmarks focus primarily on English business documents and rely on OCR-extracted text. This leaves a critical gap for low-resource languages like Bangla, where Optical Character Recognition (OCR) is often unreliable due to script complexity and legacy fonts. Furthermore, current research predominantly treats page stream segmentation as a boundary-detection problem assuming sequential arrival, rather than a reconstruction problem where pages are shuffled or interleaved.

2. Methodology

2.1 The Khondo Dataset

The authors introduce Khondo (Bangla for "split"), the first multimodal benchmark for document packet splitting on Bangladeshi government forms.

Data Source: 423 authentic forms collected from the national e-forms portal across 14 administrative domains (e.g., agriculture, finance, legal, registration).
Bilingual Nature: The dataset is inherently bilingual (Bangla–English), reflecting real-world administrative workflows. At the page level, it comprises 67% Bangla, 26% English, and 5% mixed pages.
Construction Pipeline:
1. Collection & Filtering: Manual review to discard non-forms and low-quality scans.
2. Partitioning: Forms are split into disjoint train/validation/test sets (55:20:25) stratified by domain to prevent content leakage.
3. Packet Assembly: Packets are generated by sampling forms to reach a target length of 5–20 pages.
4. Concatenation Schemes: Five variants are created to isolate different difficulties:
  - MonoSeq/PolySeq: Sequential concatenation (ordered).
  - MonoRand: Pages shuffled within each document block.
  - PolyInt: Round-robin interleaving of pages from different documents.
  - PolyRand: Global shuffling of all pages across the entire packet.
Scale: 1,950 packets totaling 28,513 pages.

2.2 Task Formalization

The task is defined as recovering a partition $\mathcal{D}$ , a domain map $t$ , and page permutations $\sigma_i$ for each document $i$ from a packet of page images $P = \langle p_1, \dots, p_n \rangle$ . Unlike previous works, Khondo operates vision-native, processing raw page images without relying on OCR text.

2.3 Evaluation Protocol

Models Evaluated:
- Zero-Shot: Five flagship Multimodal Large Language Models (MLLMs): Gemini-3.5-Flash, GPT-5.4, Qwen 3.6 Plus, GLM-4.6V, and Kimi K2.5.
- Fine-Tuned: Three small open-weight models (Qwen3-VL-4B, InternVL3.5-4B, Gemma 4 E2B) adapted via QLoRA.
Metrics:
- Clustering ( $S_{clu}$ ): Mean of V-measure and Rand Index.
- Ordering ( $S_{ord}$ ): Mean Kendall's $\tau_a$ over predicted sub-documents.
- Typing ($PgAcc$): Page-level domain classification accuracy.
- Overall Score ( $S_{pkt}$ ): Equal weighted average of clustering and ordering.
Prompting: Experiments compare an Order-Aware (OAw) prompt (explicitly instructing re-sequencing) against an Order-Agnostic (OAg) prompt.

3. Key Results

3.1 Zero-Shot Performance

Clustering vs. Ordering: MLLMs demonstrate robust clustering capabilities but struggle significantly with page ordering when pages are shuffled.
- On sequential variants, ordering scores are high ( $S_{ord} \approx 0.95$ ), but this is trivial as pages are already ordered.
- On shuffled variants (MonoRand, PolyRand), ordering scores drop drastically (e.g., Gemini-3.5-Flash drops from 0.95 to 0.68; weaker models drop to ~0.05).
- Clustering remains relatively stable ( $S_{clu} \approx 0.55–0.85$ ) even under shuffling.
Model Comparison: Gemini-3.5-Flash and Qwen 3.6 Plus outperform other models, particularly in restoring page order on shuffled packets.

3.2 Fine-Tuning Results

Fine-tuning small open-source models improves performance over their baselines but does not eliminate the bottleneck.
While clustering improves significantly (reaching $S_{clu} \approx 0.95$ on PolySeq), ordering on shuffled variants remains poor ( $S_{ord} \approx 0.32$ for the best model, Qwen3-VL-4B).
Scaling model size (from 4B to 32B) yields monotonic improvements on shuffled variants but does not close the gap to flagship zero-shot models.

3.3 Controlled Analyses

Prompt Sensitivity: Explicit instructions to reorder pages (OAw) significantly improve ordering scores on shuffled variants (gains of 0.43–0.67) compared to OAg prompts, which often simply preserve input order. However, a substantial deficit remains even with explicit instructions, suggesting the difficulty is intrinsic to the task, not just a prompting artifact.
Linguistic Effects: When tested on monolingual packets, models order English pages significantly better than Bangla pages ( $\Delta S_{ord} = 0.13–0.36$ on shuffled variants). Clustering is less affected by language. This indicates that the ordering bottleneck is partly linguistic, exacerbated by the low-resource nature of Bangla document understanding.

4. Contributions

Khondo Benchmark: The first multimodal benchmark for document packet splitting on low-resource, Bangla-dominant government forms, featuring 1,950 packets across 5 concatenation schemes and 14 domains.
Empirical Findings: Demonstration that while MLLMs can cluster pages effectively, page-order reconstruction is the primary bottleneck, especially under shuffling and for low-resource languages.
Fine-Tuning Insights: Evidence that while small MLLMs can learn the task, a significant performance gap persists on shuffled variants compared to flagship models.
Factor Isolation: Controlled studies revealing that explicit prompting aids ordering but is insufficient, and that language (Bangla vs. English) is a secondary but consistent factor degrading ordering performance.

5. Significance and Claims

The paper positions Khondo as establishing page-order reconstruction as a key open problem in vision-based, low-resource document understanding. The authors claim that:

Current MLLMs treat packet splitting as a "solved" problem for sequential text but fail when applied to vision-native, shuffled, low-resource documents.
The difficulty in ordering is intrinsic to the task (requiring reasoning over visual continuity and logical flow) rather than merely a failure of prompt engineering.
The performance gap between English and Bangla highlights a specific challenge for low-resource languages in multimodal document processing, where the lack of reliable OCR and linguistic resources compounds the difficulty of inferring document structure.

The work concludes that while fine-tuning and larger models offer incremental gains, solving the ordering bottleneck for shuffled, low-resource document packets remains a significant challenge for the field.