💬 NLP

MUDIDI: A Two-Stage Framework for Multilingual Dictionary Digitization with Language Models

This paper introduces MUDIDI, a two-stage framework leveraging Vision Language Models and Large Language Models to effectively digitize multilingual dictionaries into machine-readable formats, supported by a new annotated dataset and benchmarks demonstrating the superior performance of LLMs in handling complex scripts and layouts.

Original authors: David Setiawan, Temuulen Khishigsuren, Milind Agarwal, Pagnarith Pit, Aso Mahmudi, Ekaterina Vylomova

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: David Setiawan, Temuulen Khishigsuren, Milind Agarwal, Pagnarith Pit, Aso Mahmudi, Ekaterina Vylomova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of old, dusty dictionaries written in hundreds of different languages. Some are written in English, but many are in rare, endangered languages with unique scripts that look like ancient runes or intricate calligraphy. These books are treasures for linguists and the communities that speak these languages, but right now, they are stuck in "scan mode." They are just pictures of pages. You can't search them, you can't count the words, and you can't easily use them to teach a language or build a translation app.

The problem is that these dictionaries are messy. They have weird layouts, tiny abbreviations, and symbols that standard computer scanners (like the ones you use to scan a receipt) get completely confused by.

The authors of this paper, MUDIDI, built a new "digital translator" to solve this. Think of it as a two-step assembly line that turns a picture of a dictionary page into a clean, organized digital database.

The Two-Stage Assembly Line

Stage 1: The "Super-Scanner" (Reading the Page)
Imagine a very careful librarian who looks at a chaotic page of text and tries to type it out exactly as it appears, preserving the bold words, the italics, and the order of the columns.

The Challenge: Standard scanners often miss letters or mix up the order of columns.
The Solution: The authors tested various "AI librarians" (specifically Large Language Models and Vision Language Models). They found that the smartest AI models (like Gemini) are like super-human librarians. They can read complex scripts (like ancient cuneiform or Arabic-based scripts) and keep the formatting perfect, far better than traditional scanning software.
The Trick: Sometimes, giving the AI a "cheat sheet" (a list of valid letters for that specific language) helps it read better, but not always. Sometimes, just letting the AI look at the picture is enough.

Stage 2: The "Organizer" (Sorting the Entries)
Once the text is typed out, it's still just a wall of words. It needs to be sorted into neat little boxes. A dictionary entry isn't just a word; it has a definition, a part of speech (noun/verb), examples, and cross-references.

The Challenge: The AI needs to look at the wall of text and say, "Okay, this bold word is the Headword, this italicized part is an Example, and this symbol means Cross-Reference."
The Solution: The AI is given a specific rulebook (called the MDF schema, which is like a standard filing system for dictionaries). The AI learns to cut the text into individual entries and tag every piece of information correctly.
The Boost: The authors found that if they gave the AI the dictionary's introduction page (which explains how the book is organized) along with the rulebook, the AI became much better at sorting the data. It's like giving a new employee the employee handbook before asking them to organize the filing cabinet.

What They Discovered

Smart AI beats old scanners: The new "general-purpose" AI models are much better at reading these tricky, old dictionaries than the specialized scanning software we've used for decades. They handle weird fonts and layouts with ease.
Context is King: If you want the AI to sort the dictionary entries correctly, you have to give it the "context." Showing the AI the dictionary's introduction page helps it understand the rules of that specific book, leading to much fewer mistakes.
It works for the hard stuff: This system works on a huge variety of languages, from common ones like Greek and Japanese to endangered ones like Chukchi and Syriac.

The Result: A Digital Treasure Chest

The authors didn't just build the tool; they also built a test kit. They gathered 30 different dictionaries, had human experts check the work, and created a dataset to prove their system works.

In short: They created a way to take a picture of a dusty, complex dictionary page and turn it into a clean, searchable, machine-readable file. This allows communities and researchers to finally unlock the knowledge trapped in these old books, helping to preserve languages that might otherwise disappear.

The Catch: While the AI is amazing, it's not perfect. For very rare scripts or extremely messy layouts, human experts still need to double-check the work. But this new method makes that human job much faster and easier than before.

Technical Summary: MUDIDI – A Two-Stage Framework for Multilingual Dictionary Digitization

Problem Statement
Multilingual dictionaries are critical resources for language documentation, education, and revitalization, particularly for low-resource and endangered languages. However, the vast majority of dictionaries published in the 19th and 20th centuries remain inaccessible in machine-readable formats, existing only as scanned images. Digitizing these resources presents unique challenges: they often feature diverse and complex writing systems (including historical scripts and diacritics), dense multi-column layouts, extensive abbreviations, cross-references, and specific lexicographic conventions. While recent Vision-Language Models (VLMs) and Large Language Models (LLMs) have shown promise in document understanding, their ability to faithfully transcribe diverse scripts and correctly interpret complex dictionary structures has not been systematically evaluated.

Methodology: The MUDIDI Framework
The authors introduce MUDIDI, a two-stage framework designed to decouple faithful page transcription from lexicographic parsing. This decomposition allows for independent evaluation and improvement of character recovery versus structural interpretation.

Stage 1: Faithful Page Transcription (Vanilla OCR)
The goal is to produce a Unicode text transcription of the dictionary page that preserves typographic markup (bold <b> and italic <i> tags) and adheres to a specific reading order (column-major for standard layouts, row-wise for entries spanning columns).
- Data: The framework utilizes a dataset of 30 public-domain dictionaries covering diverse language families, writing systems (e.g., Cuneiform, Syriac, Devanagari, Cyrillic, Latin-based), and formats.
- Evaluation: Models are benchmarked on Grapheme Character Error Rate (GCER), Word Error Rate (WER), TextEdit (normalized edit distance), Markup F1 (preservation of bold/italic), and ReadOrderEdit.
- Ablation Studies: The authors test the impact of providing a source-language alphabet list and an OCR-assisted hint (raw output from Mathpix) to the models.
Stage 2: Lexicographic Parsing
The goal is to segment the transcribed text into ordered dictionary entries and map them to SIL's Multi-Dictionary Formatter (MDF) schema. This involves identifying headwords, parts of speech, definitions, examples, and cross-references.
- Pipeline: The process uses a two-pass approach. Pass 1 infers dictionary-specific parse rules (MDF markers, abbreviations, structure) from introduction pages and a sample entry. Pass 2 applies these rules to generate the final MDF-formatted output.
- Inputs: Models receive the gold-standard transcript from Stage 1, the page image, and optionally the dictionary's introduction pages and official SIL MDF guidelines.
- Evaluation: Metrics include Entry Accuracy (matching dictionary blocks), MDF Field F1 (correct field assignment), and ReadOrderEdit for entry sequences.

Key Contributions

Two-Stage Evaluation Framework: The paper proposes a novel decomposition of dictionary digitization into transcription and parsing, enabling granular diagnosis of model failures (e.g., distinguishing between character recognition errors and structural interpretation errors).
Multilingual Benchmark Dataset: The authors release the first dataset for dictionary digitization, consisting of human-annotated gold standards for three pages from each of ~30 dictionaries. This dataset covers a wide range of scripts, language families, and typographic conventions.
Systematic Evaluation of LLMs and VLMs: This is the first study to evaluate the performance of general-purpose LLMs and specialized VLMs on both diverse script recognition and complex dictionary entry segmentation.

Results

Stage 1 (Transcription): General-purpose LLMs, particularly Gemini 3.1 Pro and Gemini 3 Flash, substantially outperform specialized document VLMs (e.g., MinerU, PaddleOCR-VL) and conventional OCR systems (e.g., Mathpix) across most writing systems. LLMs demonstrated superior capability in preserving markup and reading order.
- Alphabet Lists: Providing an alphabet list yielded modest improvements for some models (e.g., Gemini) but degraded performance for others (e.g., GLM-OCR).
- OCR Hints: Providing preliminary OCR text as a hint generally had a slight negative effect or negligible impact on general-purpose LLMs, sometimes anchoring them to systematic OCR errors.
- Challenges: Scripts with limited digital presence, such as Cuneiform (Assyrian) and certain Arabic-based orthographies (Syriac, Circassian), remained the most challenging, exhibiting higher error rates.
Stage 2 (Parsing): General-purpose LLMs achieved high entry detection accuracy (often $\ge$ 0.99). Performance in MDF field assignment varied by model and input configuration.
- Auxiliary Information: Including the dictionary's introduction pages improved MDF field assignment scores by 3–4 F1 points for Gemini and GPT models.
- Guidelines: Providing official SIL MDF guidelines further improved performance, particularly when dictionary introductions were absent.
- Parse Rules: Replacing inferred parse rules with human-validated gold rules significantly improved MDF field assignment F1 scores (by ~6 points on average), indicating that errors in Pass 1 rule inference propagate to Pass 2.

Significance and Claims
The paper claims that MUDIDI provides a practical pathway for converting legacy lexicographic resources into structured, machine-readable data. The authors demonstrate that general-purpose LLMs, when guided by appropriate context (introductions and guidelines), can outperform specialized OCR engines in the complex task of dictionary digitization.

The work emphasizes that while current models show superior performance, they remain sensitive to under-digitized scripts. The authors advocate for a workflow that leverages the flexibility of LLMs to incorporate custom, dictionary-specific post-processing rules to address systematic errors, rather than relying solely on re-transcription. Ultimately, the framework aims to support linguists, archivists, and speaker communities in unlocking decades of linguistic knowledge for language revitalization and research. The paper concludes with practical recommendations for practitioners, including model selection (favoring Gemini 3 Flash for cost-effective transcription) and the strategic use of auxiliary information to maximize parsing quality.

The Two-Stage Assembly Line

What They Discovered

The Result: A Digital Treasure Chest

More like this