💬 NLP

SumTablets: A Transliteration Dataset of Sumerian Tablets

This paper introduces SumTablets, a comprehensive dataset pairing 91,606 Sumerian cuneiform tablets with their Oracc transliterations to enable modern NLP applications, demonstrating that fine-tuned autoregressive language models can achieve high accuracy (97.55 chrF) in automating the transliteration process.

Original authors: Cole Simmons, Richard Diehl Martinez, Dan Jurafsky

Published 2026-02-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Cole Simmons, Richard Diehl Martinez, Dan Jurafsky

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of ancient clay tablets, written in a language that hasn't been spoken for 4,000 years. These tablets are covered in wedge-shaped marks called cuneiform. To a modern eye, they look like abstract art or a secret code.

For centuries, experts called Assyriologists have had to sit down, squint at photos of these clay tablets, and manually translate the wedge marks into the Latin alphabet we use today. This process is called transliteration. It's slow, tedious, and requires years of specialized training.

This paper introduces a new tool called SumTablets that aims to change the game. Here is the story of how they did it, explained simply.

1. The Problem: A Missing Link

Think of the existing digital archives of Sumerian texts as two separate piles of books:

Pile A: High-quality photos or digital images of the clay tablets (the "pictures").
Pile B: The experts' handwritten translations of those pictures into English-like text (the "dictionary").

The problem? These two piles were never glued together. There was no single dataset that showed, "This specific picture of a wedge mark equals this specific letter." Without this connection, computers couldn't learn how to translate the pictures themselves. It was like trying to teach a robot to read a map without ever showing it the map and the street names side-by-side.

2. The Solution: SumTablets (The "Rosetta Stone" for AI)

The authors created SumTablets, a massive dataset that acts as a giant bridge between the pictures and the words.

The Scale: They gathered 91,606 ancient tablets.
The Content: They paired over 6.9 million individual wedge marks (glyphs) with their corresponding translations.
The Magic: They didn't just copy-paste; they cleaned up the data. They turned the messy notes experts use (like "this part is broken" or "this part is missing") into special computer codes. This makes the data "clean" enough for a computer to learn from.

Think of SumTablets as a giant flashcard deck for an AI. On one side is the ancient symbol; on the other is the modern reading.

3. How They Taught the Computer (The "Teacher" and the "Student")

Once they had the flashcards, they tried two different ways to teach a computer to do the translation.

Method A: The "Dictionary Guess" (The Baseline)

Imagine you are playing a game where you have to guess a word based on a picture. You have a dictionary that says, "This picture could mean Cat, Bat, or Rat."

The computer looks at the picture.
It checks the dictionary.
It picks the most common guess based on how often that word appears in history.
Result: It got about 61% of the words right. It's like a student who memorized a dictionary but doesn't understand the context of the sentence.

Method B: The "Super-Reader" (The Neural Model)

This is where the paper gets exciting. They used a Transformer model (the same type of AI brain behind tools like ChatGPT).

The Challenge: These AI models usually speak 100+ modern languages (like French, Chinese, and Spanish). Sumerian is a "language isolate"—it has no living relatives. It's like trying to teach a polyglot to speak a language from a completely different planet.
The Trick: The researchers didn't teach the AI from scratch. They took a model that already knew how to understand grammar and sentence structure in many languages, and they "fine-tuned" it using their SumTablets flashcards.
The Result: The AI learned to look at a sequence of wedge marks and predict the translation with 97.5% accuracy.

4. Why This Matters

Think of the current state of Assyriology as a world where only a handful of people can read ancient maps, and they have to draw every single road by hand.

Before SumTablets: An expert spends 3 hours manually translating one tablet.
With SumTablets: The AI can generate a draft translation in seconds. The expert then just acts as a proofreader, checking the AI's work for errors rather than doing the whole job from scratch.

This doesn't replace the experts; it gives them a superpower. It frees them from the boring, repetitive work so they can focus on the interesting stuff: understanding the history, politics, and stories of the ancient world.

The Bottom Line

The authors built the first massive, clean dataset connecting ancient Sumerian pictures to their modern translations. They proved that even for a dead, unique language with very few speakers, modern AI can learn to read it almost perfectly.

It's like giving the world's best archaeologists a pair of glasses that instantly turns ancient clay scribbles into readable text, letting them spend more time exploring history and less time deciphering code.

1. Problem Statement

The paper addresses a critical bottleneck in the field of Assyriology and Natural Language Processing (NLP): the lack of a comprehensive, accessible dataset pairing Sumerian cuneiform glyphs (visual/Unicode representations) with their transliterations (scholarly Latin-script interpretations).

Context: While large repositories of Sumerian transliterations exist (e.g., Oracc, CDLI, ETCSL), they are typically stored as text strings without parallel digital representations of the original cuneiform glyphs.
Challenge: Sumerian glyphs are polyvalent, meaning a single glyph can have multiple valid readings depending on context (e.g., a glyph might mean "mouth," "to speak," or "nose"). Deciphering requires complex contextual analysis.
Gap: The absence of paired glyph-transliteration data has prevented the application of modern sequence-to-sequence NLP models to automate or assist in the transliteration process.

2. Methodology

A. Dataset Construction (SumTablets)

The authors created SumTablets, a dataset containing 91,606 Sumerian tablets totaling 6,970,407 glyphs. The construction pipeline involved:

Source Aggregation: Data was sourced from the Electronic Pennsylvania Sumerian Dictionary (ePSD2) and the Oracc Sign List (OSL), which aggregate data from projects like Oracc, CDLI, and ETCSL.
Preprocessing & Standardization:
- Cleaning: Embedded ASCII annotations (e.g., editor's notes on missing text) were removed to prevent bias. Text in square brackets (broken segments) was replaced with a special token (...), while partially visible text was retained.
- Structural Preservation: Formatting information (surfaces, newlines, columns, rulings) was converted into special tokens (e.g., <SURFACE>, <COLUMN>, \n) to maintain the tablet's physical structure in the sequence.
Glyph Mapping:
- Transliterations were split into morphemes (readings).
- Using ePSD2 and OSL dictionaries, each reading was mapped back to its source Unicode glyph name (e.g., reading "ka" maps to glyph KA).
- Glyph names were converted to Unicode characters.
- Success Rates: 99.93% of readings were mapped to glyph names; 99.96% of those were mapped to Unicode. Unmappable items were replaced with <UNK>.
Metadata & Splitting: The dataset includes metadata for time periods (10 categories) and genres (14 categories). The data was stratified by period to ensure balanced training/validation/test splits (90%/5%/5%).

B. Baseline Models

The authors implemented and compared two approaches to the transliteration task (defined as a sequence-to-sequence conversion from Glyphs $\to$ Transliteration):

Dictionary Baseline (Weighted Sampling):
- For each input glyph, the model samples a reading based on the frequency of that reading in the ePSD2/OSL dictionaries.
- This serves as a simple, non-neural lower bound.
Neural Baseline (Fine-tuned Transformer):
- Architecture: An Encoder-Decoder model based on XLM-R (a multilingual transformer pre-trained on 100+ languages).
- Tokenization: Since XLM-R lacks Sumerian support, the authors retrained the SentencePiece tokenizer twice:
  - Glyph Tokenizer: Trained on Unicode glyphs (Vocab size: 632).
  - Transliteration Tokenizer: Trained on Latin transliterations (Vocab size: 1024).
- Training Strategy:
  - Stage 1: Masked Language Modeling (MLM) on the encoder using only glyphs to learn glyph representations.
  - Stage 2: Freeze encoder, train the decoder to predict transliterations from glyph embeddings.
  - Stage 3: Unfreeze encoder and jointly train the full encoder-decoder model.
- Handling Long Sequences: Tablets exceeding the 128-token limit were segmented by newlines to preserve structural integrity.

3. Key Contributions

SumTablets Dataset: The first large-scale, open-access (CC BY 4.0) dataset of paired Sumerian Unicode glyphs and transliterations, released via Hugging Face.
Standardization: A rigorous preprocessing pipeline that converts heterogeneous Oracc data into a uniform format suitable for deep learning, preserving structural nuances (surfaces, breaks) via special tokens.
First NLP Transliteration Model: The first demonstration of an automatic Sumerian transliteration model using transformer-based architectures.
Benchmarking: Established a new state-of-the-art benchmark for the task, providing a baseline for future research in low-resource, language-isolate NLP.

4. Results

The models were evaluated using the character-level F-score (chrF) with $\beta=2$ and $n=6$ .

Dictionary Baseline: Achieved a chrF score of 61.22.
Neural Baseline (XLM-R): Achieved a chrF score of 97.54 on the held-out test set.
Performance by Genre/Period:
- The model performed best on Administrative texts (98.14 chrF), likely due to their formulaic nature and high frequency in the training data.
- Performance dropped on Literary (90.67) and Liturgy (77.68) texts, which have more complex vocabulary and fewer training examples.
- The model demonstrated robustness across different historical periods (e.g., Ur III: 98.46, Old Babylonian: 90.38).

5. Significance and Future Impact

Accelerating Assyriology: The high accuracy (97.54 chrF) suggests that AI can generate preliminary transliterations for scholars, allowing them to focus on verification and error correction rather than manual transcription from scratch. This is crucial given the scarcity of Sumerian specialists relative to the volume of extant texts.
Low-Resource NLP: The success of adapting a multilingual model (XLM-R) to a language isolate (Sumerian) demonstrates the potential of cross-lingual transfer learning for other under-resourced ancient languages.
Pipeline Development: This work serves as a foundational step toward a complete Sumerian translation pipeline (Glyph $\to$ Transliteration $\to$ Translation).
Open Science: By releasing the dataset and code, the authors lower the barrier to entry for computational linguists and Assyriologists to collaborate on digitizing ancient history.

Limitations Noted

Data Imbalance: The dataset is heavily skewed toward administrative texts; the model struggles slightly with rare genres.
Unicode Gaps: Some complex compound glyphs are not yet in Unicode and are currently mapped to <UNK>.
Orthographic Variation: Unicode standardization flattens historical variations in glyph shapes, which may obscure meaningful linguistic evolution.
Name Prediction: The model occasionally struggles with phonetic variations in proper names, a known challenge in transliteration.