💬 NLP

Rashid: A Cipher-Based Framework for Exploring In-Context Language Learning

The paper introduces Rashid, a framework that reversibly ciphers high-resource languages to create controllable, resource-rich "unseen" languages, enabling scalable and rigorous evaluation of in-context language learning (ICLL) strategies and overcoming the data and tool limitations typically associated with low-resource languages.

Original authors: Niyati Bafna, Ryan Soh-Eun Shim, Barbara Plank, David Yarowsky, Hale Sirin

Published 2026-03-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Niyati Bafna, Ryan Soh-Eun Shim, Barbara Plank, David Yarowsky, Hale Sirin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a super-smart robot how to speak a language it has never heard before, like a rare dialect spoken by a small tribe in the Amazon. The problem? You don't have a dictionary, a grammar book, or any native speakers to help you. The robot is flying blind, and it's impossible to know if it's learning correctly because you can't understand the output either.

This is the big headache researchers face with In-Context Language Learning (ICLL): trying to teach AI new languages without the usual tools.

Enter Rashid, a clever new framework introduced in this paper. The name is inspired by the legendary Caliph Harun al-Rashid, who famously disguised himself as a commoner to walk the streets of Baghdad and understand the people's problems. Similarly, this framework "disguises" a well-known language to trick the AI into thinking it's learning a brand-new one, allowing researchers to study the learning process in a controlled, safe environment.

Here is how it works, broken down with simple analogies:

1. The "Magic Scrambler" (The Cipher)

Instead of trying to find a rare, low-resource language to test on, the researchers took 10 common, high-resource languages (like French, German, Hindi, and Turkish) and ran them through a "Magic Scrambler."

The Analogy: Imagine you have a perfect English sentence: "The cat sat on the mat."
The Scramble: You apply a secret code where every letter is swapped for another (e.g., 'T' becomes 'X', 'h' becomes 'q'). The sentence becomes: "Xq qat qat on qe mat."
The Result: To the AI, this looks like a completely alien language. But to the researchers, it's still French or Hindi underneath. Because they know the secret code, they can instantly "unscramble" the AI's answer to check if it's right.

This is the genius part: They created "fake" unknown languages that actually have perfect dictionaries and grammar books available. This lets them test AI strategies without the usual resource shortages.

2. The Experiment: Teaching the Robot

Once the language is scrambled, the researchers tried to teach the AI to translate it using different "study aids," just like a human student would:

The Dictionary: Giving the AI a list of word-for-word translations.
The Grammar Book: Giving it rules about sentence structure (like "verbs go at the end").
The Examples: Showing it sample sentences of the language being used.

What they found:

Reading is easier than writing: The AI got pretty good at understanding the scrambled language (translating it to English) when given a dictionary and grammar rules. It's like reading a foreign menu; you can guess the ingredients.
Writing is hard: When asked to generate text in the scrambled language (translating English to the new language), the AI struggled. Even with a dictionary, it couldn't figure out how to put the words together correctly. It was like trying to write a poem in a language you only know the words for, but not the rhythm.

3. The "Pivot" Trick (The Bridge)

The researchers discovered a clever workaround for the writing problem. Instead of going straight from English to the "Scrambled Language," they went through a related language first.

The Analogy: Imagine you want to translate a story from English to a rare dialect of Hindi. You don't know the dialect. But you do know standard Hindi.
The Strategy: You translate English -> Standard Hindi -> Scrambled Hindi.
The Result: Because Standard Hindi and the Scrambled version share the same "skeleton" (grammar and word order), the AI did a much better job. It was like using a bridge to cross a river instead of trying to swim across it in the dark.

4. Why This Matters

This framework is like a flight simulator for language learning.

Real Life: Testing AI on real, rare languages is expensive, slow, and risky. If the AI fails, you might not even know why because you don't speak the language.
The Rashid Simulator: Researchers can crash the plane (the AI) a thousand times, check the black box (the output), and fix the engine (the strategy) instantly because they can "unscramble" the crash.

The Bottom Line

The paper concludes that while current AI methods are getting better at reading new languages, they are still terrible at speaking them. However, by using this "scrambled language" sandbox, researchers can now rapidly test new ideas—like using related languages as a bridge—to figure out how to finally teach AI to speak the world's thousands of languages fluently.

It's a way to solve the "low-resource" problem by pretending the resources don't exist, so we can learn how to build them for real.

1. Problem Statement

The field of In-Context Language Learning (ICLL) aims to enable Large Language Models (LLMs) to process unseen or extremely low-resource languages (LRLs) by providing linguistic resources (lexicons, grammar books, exemplars) within the prompt. However, progress in this field is severely hindered by resource scarcity:

Lack of Evaluation: True LRLs often lack high-quality evaluation tools, human expertise, and benchmark data, making it difficult to assess model performance reliably.
Confounding Factors: It is difficult to distinguish whether an LLM's performance is due to genuine language learning or residual knowledge from pre-training on related languages.
Experimental Limitations: Researchers cannot easily conduct large-scale experiments or iterate on strategies because creating resources for true LRLs is expensive and time-consuming.

2. Methodology: The Rashid Framework

The authors introduce Rashid, a framework designed to create a "sandbox" for systematic ICLL experimentation. Instead of using real low-resource languages, Rashid constructs artificially unseen languages by applying reversible character-level ciphers to high-resource languages (HRLs).

Cipher Construction: The framework uses a monoalphabetic substitution cipher ( $\chi_L$ $χ_{L}$ ) applied at the character level.
- Reversibility: The cipher is bijective, allowing model outputs to be deciphered back into the original HRL for evaluation using standard NLP tools.
- Obscuring: The cipher is complex enough (with a keyspace of $\approx |C|!$ ) that LLMs cannot decipher it via reasoning or cryptanalysis, forcing them to rely on in-context learning.
- Preservation: Crucially, the cipher preserves all underlying linguistic properties (morphology, syntax, semantics, word order) of the source language, only altering the surface form.
Experimental Setup:
- Languages: 10 mid-to-high resource languages (German, French, Spanish, Hindi, etc.) covering diverse typologies (agglutinative, fusional, isolating) and scripts (Latin, Devanagari, Telugu).
- Models: GPT-5.1 (primary), Llama-3.1-8B, and Qwen-2.5-7B.
- Tasks: Machine Translation (MT) is the primary task (WMT24++), with secondary evaluation on downstream tasks (MMLU-ProX, XNLI, XStoryCloze).
- Strategies Tested: The authors incrementally add resources to the prompt:
  1. Lexicons (L): Word-to-word mappings.
  2. Exemplars (LE): In-context examples (few-shot).
  3. Lemmatization (Lem): Morphological lemmas.
  4. Morphology (M): POS tags and grammatical features.
  5. Syntax (S): High-level syntactic profiles (e.g., SOV vs. SVO).
  6. New Strategies: Near-perfect lexicon coverage (via Google Translate proxy) and cascading through a related High-Resource Language (RHRL).

3. Key Contributions

The Rashid Framework: A novel method to simulate "truly unseen" languages while retaining access to the full suite of NLP tools and resources available for HRLs. This allows for rigorous, large-scale, and reproducible experimentation.
Comprehensive Assessment of Current ICLL: The first systematic evaluation of current ICLL methods using State-of-the-Art (SOTA) metrics (xCOMET, GEMBA-MQM) and manual analysis, moving beyond simple string-matching metrics like BLEU.
Identification of Limitations: The paper establishes that current ICLL methods are effective for comprehension (translation to English) but largely ineffective for generation (translation from English), with outputs often being unusable.
New Strategic Directions:
- Lexicon Coverage: Demonstrated that near-perfect lexicon coverage significantly improves comprehension but does not solve generation issues.
- Cascading via Related Languages: Showed that translating English $\to$ Related HRL $\to$ Ciphered Target Language significantly improves syntactic coherence and idiomatic translation compared to direct translation.

4. Key Results

Comprehension vs. Generation Gap:
- $\to$ English (Comprehension): Performance improves incrementally with added resources. The best strategy (Lexicon + Lemmas + Morphology + Syntax) yields the highest scores but still lags significantly behind the "topline" (un-ciphered) performance.
- English $\to$ (Generation): Current strategies show minimal improvement over a simple word-for-word baseline. Models struggle to generate grammatically correct and semantically coherent text in the unseen language.
Metric Analysis:
- String-based metrics (chrF) often overestimate quality for generation tasks by matching nonsensical word-for-word translations.
- Semantic metrics (xCOMET) and error-based metrics (GEMBA-MQM) reveal that outputs are frequently unusable, with high rates of critical errors.
Qualitative Findings:
- Exemplars help reduce topic drift and improve specific word usage.
- Morphological info aids in comprehension but is inconsistently applied in generation.
- Syntax guides help reorder simple phrases but fail in complex sentences.
New Strategies Performance:
- Near-Perfect Lexicon ( $L_{cov}$ ): Improved comprehension scores significantly (e.g., +10 xCOMET points for German) but failed to fix generation quality.
- Cascading ( $CL_{cov}$ ): Translating via a related language (e.g., English $\to$ French $\to$ Ciphered Spanish) resulted in unanimous wins in manual evaluations for word order and idiomaticity, outperforming direct translation strategies.

5. Significance and Implications

Supplementary Sandbox: Rashid does not replace studies on real LRLs but provides a necessary, low-cost, and controlled environment to test hypotheses that are currently impossible to validate due to resource constraints.
Guiding Resource Creation: The findings suggest that for improving ICLL generation, researchers should prioritize parallel resources between related languages rather than just English-centric resources.
Task-Oriented Insights: The study extends beyond MT, showing that cascaded approaches (Translate $\to$ Solve Task) outperform direct task solving on ciphered languages, highlighting the need for task-oriented exploration in ICLL.
Ethical Consideration: The authors emphasize that while the framework is powerful, insights must be validated on real languages to avoid erasing language-specific phenomena unique to marginalized communities.

In conclusion, Rashid provides a rigorous methodology to deconstruct the challenges of in-context language learning, revealing that while LLMs can learn to understand unseen languages with sufficient context, generating them remains a significant hurdle requiring new strategies like cross-lingual cascading and better morphological modeling.