💬 NLP

The Latin Substrate: How Language Models Represent and Mediate Script Choice

This paper reveals that large language models mediate script choice through shared latent representations and language-agnostic mechanisms, exhibiting a structural asymmetry where non-Latin scripts are controlled by specific late-layer components while Latin script emerges as a privileged, diffuse default substrate.

Original authors: Daniil Gurgurov, Alan Saji, Katharina Trinley, Josef van Genabith, Simon Ostermann

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniil Gurgurov, Alan Saji, Katharina Trinley, Josef van Genabith, Simon Ostermann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a large language model (LLM) as a massive, multilingual translator who has read almost everything written in the world. But here's the twist: this translator has a favorite "home base" language for thinking: Latin script (the A, B, C alphabet used in English, French, Spanish, etc.).

This paper investigates how this translator handles languages that use different writing systems, like Hindi (which uses Devanagari), Arabic, or Russian (Cyrillic). The researchers discovered that the model doesn't just switch between these scripts like flipping a light switch. Instead, it follows a very specific, one-way street.

Here is the breakdown of their findings using simple analogies:

1. The "Hidden Middleman" (Latent Romanization)

When you ask the model to translate a word from Malayalam (a language from India) into Devanagari, you might expect it to go straight from "Malayalam" to "Devanagari."

However, the researchers looked inside the model's "brain" layer by layer and found something surprising. Before the model writes the final Devanagari letters, it briefly thinks in Latin letters.

The Analogy: Imagine you are trying to explain a complex idea to a friend who speaks a different language. You don't just switch languages instantly; you first think of the concept in your own language, maybe even write it down in your own script to make sure you have the meaning right, and then translate it to your friend's script.
The Finding: The model does this automatically. It converts the input into a "Latin-ized" version in its middle layers before finally outputting the target script. It treats Latin as a universal bridge.

2. The "One-Way Street" (Directional Asymmetry)

The paper found a major imbalance in how the model handles these scripts.

Going to Latin (Non-Latin → Latin): This is easy and direct. The model has a specific, compact "gate" or switch that turns on to produce Latin text. It's like a dedicated highway exit.
Going to Non-Latin (Latin → Non-Latin): This is messy and diffuse. To produce Hindi, Arabic, or Russian, the model doesn't use one single switch. Instead, it relies on hundreds of tiny, weak signals scattered all over the network working together.
The Analogy: Think of the model's brain as a city.
- Latin Script is the City Center. It's huge, open, and everywhere. Getting to the city center is easy because you can arrive from any direction.
- Non-Latin Scripts are Specialized Neighborhoods. To get there, you need a specific, narrow tunnel (a "gate"). If you try to leave the city center to go to a neighborhood, you have to navigate a complex, winding path that relies on many different roads. If you block just one road, you might still get there, but if you block the specific tunnel, you can't get in at all.

3. The "Steering Wheel" (Controlling the Script)

The researchers tried to "steer" the model using math. They calculated a simple vector (a direction) that represents "Switch to Latin" or "Switch to Native Script."

The Result: They found they could push the model to output Latin or a specific non-Latin script just by nudging its internal math in the right direction.
The Catch: The "nudge" worked perfectly to turn any non-Latin text into Latin. But turning Latin text into a specific non-Latin script was harder and less stable. It's like having a steering wheel that turns the car left (to Latin) very smoothly, but turning it right (to a specific non-Latin script) requires a much stronger, more precise push.

4. The "Universal Gatekeepers" (Mechanistic Localization)

The most exciting part is that the researchers found the exact "switches" (specific attention heads) responsible for this.

The Finding: They found a tiny group of about 5 specific "workers" (neurons/heads) deep inside the model that act as the gatekeepers for non-Latin scripts.
The Magic: These workers are language-agnostic. If you take the "gatekeepers" trained on Hindi and Arabic and use them on Russian, Japanese, or Greek, they still work! They don't care which non-Latin language it is; they just know how to switch the model from "Latin mode" to "Not-Latin mode."
The Analogy: Imagine a security guard at a club. This guard doesn't care if you are wearing a sari, a kimono, or a kilt. They just check your ID to see if you are allowed into the "Non-Latin VIP section." Once you are in, the guard lets you in regardless of your specific outfit.

Summary

The paper concludes that Large Language Models have a privileged bias toward Latin script.

Latin is the model's default, comfortable "home" where information is stored and processed broadly.
Non-Latin scripts are accessed through a specialized, compact "gate" that sits on top of this Latin foundation.

The model doesn't treat all scripts as equals; it treats Latin as the universal substrate (the foundation) and builds everything else on top of it using specific, localized mechanisms.

Technical Summary: The Latin Substrate in Large Language Models

Problem Statement

Many languages are written in multiple scripts (e.g., Hindi in Devanagari and Latin, Serbian in Cyrillic and Latin). While Multilingual Large Language Models (LLMs) can generate equivalent linguistic content across these distinct orthographic systems, the internal mechanisms mediating script variation remain poorly understood. Prior work suggests that representations are partially shared across scripts, with phenomena like "latent romanization" observed where intermediate layers transiently represent information in Latin form. However, it is unclear how LLMs route information between scripts, whether they rely on shared latent representations, and if specific architectural components are responsible for script selection.

Methodology

The authors employ a multi-faceted approach combining representational analysis, linear steering, and mechanistic localization across four open-source instruction-tuned LLMs (Llama-3.1-8B/70B and Aya-Expanse-8B/32B), with specific focus on Hindi and Arabic.

Per-layer Output Analysis (Logit Lens): Using the logit lens technique, the authors decode intermediate layer activations to observe the next-token distribution during transliteration tasks. They quantify the "latent fraction"—the proportion of timesteps where romanized subwords appear in the top-1 prediction—to detect intermediate Latin representations.
Representational Analysis:
- Separability: They measure script separability using linear probes, cosine similarity, and Fisher discriminant ratios on mean-pooled hidden states of native vs. romanized sentences.
- Steering: They construct linear steering vectors ( $v_{nat2lat}$ and $v_{lat2nat}$ ) based on the difference in mean hidden representations between script classes. These vectors are applied to the residual stream to test if script identity can be flipped while preserving semantics.
- Generalization: Steering vectors derived from Hindi and Arabic are applied to unseen languages (e.g., Russian, Japanese, Thai) to test for language-agnostic script axes.
Mechanistic Analysis (Activation Patching): The authors use counterfactual activation patching to localize specific attention heads that causally mediate script choice. By swapping activations between native and Latin runs, they measure the change in the log-odds of producing native-script vs. Latin-script tokens. They validate these findings through targeted interventions (zeroing or patching specific head subsets) on held-out prompts.

Key Results

1. Latent Romanization

During transliteration tasks, the models consistently exhibit latent romanization. In middle-to-upper layers (e.g., layers 20–40 in Llama-3.1-8B), the model's internal predictions transiently align with romanized subwords before committing to the final target script. This suggests that Latin script acts as a shared intermediate representation bridge during script transformation.

2. Representational Asymmetry and Steering

Separability: Script information becomes increasingly separable across layers, with distinct clusters for different scripts emerging primarily in the upper third of the network.
Linear Steering: A simple linear steering direction can flip a model's output script while largely preserving semantic content.
Directional Asymmetry: The steering effect is asymmetric. Converting non-Latin to Latin ($lat2nat$) is highly reliable and requires lower intervention strength. Conversely, converting Latin to non-Latin ($nat2lat$) is less stable and often results in outputs drifting toward various non-Latin scripts rather than a single target. This implies that Latin occupies a "diffuse" region in the representation space, while non-Latin scripts form tighter, more distinct clusters.

3. Mechanistic Localization of a "Script-Mediation Gate"

Compact Non-Latin Gate: Causal analysis reveals that non-Latin script generation is mediated by a small, compact set of late-layer attention heads (e.g., L30H20, L30H23, L26H7 in Llama-3.1-8B). Patching as few as five of these heads is sufficient to flip the output script from Latin to a native script (e.g., Devanagari) with high success rates.
Diffuse Latin Generation: In contrast, Latin-script generation does not rely on a specific, localized circuit. Instead, it emerges from diffuse, low-magnitude contributions across many heads. Removing the "non-Latin gate" causes the model to default to Latin output for a portion of prompts, suggesting Latin is the model's fallback substrate.
Language Agnosticism: The identified heads transfer across unrelated languages and writing systems. Heads identified using Hindi or Arabic successfully redirect output scripts in Russian (Cyrillic), Marathi (Devanagari), Urdu (Perso-Arabic), Greek, and Japanese, indicating a language-agnostic mechanism for script routing.

Significance and Claims

The paper claims to provide the first unified analysis of how LLMs internally represent and mediate multiple scripts. The central finding is that Latin script functions as a privileged substrate for surface-language representations in LLMs.

Privileged Substrate: The model organizes script variation around shared latent representations aligned with Latin. Non-Latin outputs are produced via a dedicated, identifiable, and compact routing mechanism (the "gate"), whereas Latin outputs emerge from the network's default, diffuse processing pathways.
Mechanistic Insight: The work identifies a specific, transferable set of attention heads that causally control script choice, demonstrating that script routing is implemented by language-agnostic components rather than entirely separate processing pathways for each script.
Implications for Understanding: These findings suggest that the dominance of Latin script in pretraining data has shaped the internal geometry of LLMs, creating an asymmetric architecture where Latin is the "ground state" and non-Latin scripts require active, localized gating to be generated.

The authors note limitations, including the focus on specific open-source models and languages with established romanization conventions, acknowledging that models trained on different data distributions or closed models might exhibit qualitatively different mechanisms.