💬 NLP

LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval

The paper introduces LEMUR, a large-scale multilingual corpus of EU environmental legislation, and demonstrates that fine-tuning multilingual embedding models on this high-fidelity data significantly improves legal information retrieval accuracy, especially for low-resource and cross-lingual scenarios.

Original authors: Narges Baba Ahmadi, Jan Strich, Martin Semmann, Chris Biemann

Published 2026-02-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Narges Baba Ahmadi, Jan Strich, Martin Semmann, Chris Biemann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Lost in Translation" Legal Library

Imagine you are in a massive, ancient library filled with millions of law books. These books aren't just written in plain text; they are often old, dusty PDFs with complex tables, tiny footnotes, and weird layouts. To make matters worse, the library is international—the books are written in 25 different languages.

Now, imagine you need to find a specific rule about environmental protection. You ask a robot librarian (an AI) to find it for you. But there are two big problems:

The Robot is a Generalist: The robot was trained to read Wikipedia and news articles, not complex legal jargon. It sees a legal term and thinks it means something common, like how a doctor might mistake a medical term for a casual word.
The "Blurry Vision" Problem: When the robot tries to read the old PDF books, the text comes out garbled. It’s like trying to read a document through a foggy window.

Because of this, the robot often "hallucinates"—it confidently gives you the wrong answer or points you to a book that has nothing to do with your question.

The Solution: Project LEMUR

A team of researchers created LEMUR to fix this. Think of LEMUR as a "Legal Bootcamp" for AI librarians. Here is how they did it:

1. Building the Ultimate Textbook (The Corpus)

Instead of letting the AI learn from messy internet scraps, the researchers went straight to the source: the official EU environmental laws. They gathered nearly 25,000 official documents in 25 different languages. This is like giving the student a perfectly curated set of textbooks instead of a pile of random magazines.

2. Cleaning the Glasses (The LCS Score)

To fix the "blurry vision" problem, they created a tool called the Lexical Content Score (LCS). Think of this as a quality-control inspector. Every time they converted a PDF into digital text, the inspector compared it to the original to make sure no words were lost or scrambled. They used a high-tech tool (olmOCR) to make sure even the complicated tables stayed organized, ensuring the AI wasn't reading "gibberish."

3. Specialized Training (Fine-Tuning)

The researchers took three "smart" AI models and put them through intensive legal training.

The Monolingual Drill: They taught them to be experts in one language at a time (like training a translator to be a master of just French law).
The Bilingual Buddy System: They tried teaching them two languages at once (like English and Latvian) to see if knowing one helped the AI understand the other better.

The Results: A Smarter Librarian

The results were impressive! Here is what they found:

The "Expert" Boost: Once the AI went through the "Legal Bootcamp," it became much better at finding the right document. Even if you only gave it a tiny hint (like a short title), it could find the massive, complex law it belonged to.
The "Language Bridge": This was the coolest part. They discovered that if you train the AI to understand law in English, it actually gets better at understanding law in other languages, even if it hasn't studied those specific languages deeply.
- Analogy: It’s like teaching a musician how to read music in English. Once they understand the logic of how music works, they can pick up a piece of music written in Italian or German much faster. The AI learned the "logic" of law, which helped it jump across language barriers.
Helping the Underdogs: The AI's improvement was most dramatic for "low-resource" languages (languages that don't have as much data on the internet). The training gave these languages a massive boost, bringing them up to speed with major languages like English or German.

Summary

In short, the researchers built a high-quality, multilingual "legal library" and used it to turn general-purpose AI into specialized legal experts. This makes it much more likely that when a lawyer or a citizen asks a question, the AI will find the exact law they need, rather than just guessing.

Technical Summary: LEMUR – A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval

1. Problem Statement

The deployment of Large Language Models (LLMs) in legal practice is currently hindered by two primary issues: hallucinations and a lack of grounding in authoritative sources. While Retrieval-Augmented Generation (RAG) is the standard solution to mitigate these risks, its effectiveness depends heavily on the quality of the retrieval component (the embedding model).

Existing challenges include:

Domain Gap: General-purpose embedding models are often untrained on the complex, archaic, and polysemous terminology of legal text.
Language Gap: Most specialized legal models are monolingual, whereas legal work (especially in the EU) is inherently multilingual.
Data Gap: Existing multilingual legal datasets are typically designed for pretraining or classification, not for semantic retrieval.
Extraction Gap: Legal documents are often stored as PDFs, where imperfect text extraction (due to tables and multi-column layouts) introduces noise that degrades retrieval performance.

2. Methodology

The authors propose a comprehensive pipeline to bridge these gaps, moving from raw PDF collection to fine-tuned retrieval models.

A. Data Construction (LEMUR Corpus):

Source: 24,953 official EUR-Lex PDF documents covering EU environmental legislation across 25 languages.
PDF-to-Text Conversion: To combat the "extraction gap," the authors used olmOCR to convert PDFs into structured JSONL (Markdown) format, which preserves table structures better than standard HTML conversion.
Quality Validation (LCS): They introduced the Lexical Content Score (LCS), a metric using cosine similarity between bag-of-words vectors of the converted text and the authoritative HTML version, to quantify conversion fidelity.
Query-Document Pairing: They utilized a "metadata-to-document" approach. The introductory metadata of a legal act (type, date, subject) serves as the query, while the substantive legislative text serves as the document. This mimics realistic legal search behavior.

B. Model Fine-Tuning:
The authors fine-tuned three state-of-the-art (SOTA) multilingual embedding models: E5-Multilingual, Qwen3-0.6B, and Qwen3-4B.

Monolingual Contrastive Fine-Tuning: Used a Multiple Negatives Ranking (MNR) loss to align metadata queries with their corresponding legal texts within a single language.
Bilingual Multi-Positive Fine-Tuning: Extended the objective to include multiple language versions of the same act as "positives" for a single query, encouraging cross-lingual semantic alignment.

C. Evaluation Framework:

Metrics: Top- $k$ accuracy (Acc@1, Acc@3, Acc@5).
Settings: Monolingual (training and testing in the same language), Bilingual (training on high-resource + low-resource pairs), and Cross-Lingual (training on one language and testing on an unseen target language).

3. Key Contributions

LEMUR Corpus: A large-scale, high-quality multilingual corpus of EU environmental law specifically designed for semantic retrieval.
Lexical Content Score (LCS): A systematic method for measuring the reliability of PDF-to-text conversion in multilingual legal contexts.
Empirical Framework: A robust methodology for fine-tuning and evaluating multilingual legal embeddings across varying resource levels (high vs. low resource).

4. Results

Consistent Improvement: Monolingual fine-tuning significantly improved Top- $k$ retrieval accuracy across all tested languages (English, German, French, Latvian, and Maltese) compared to base models.
Low-Resource Gains: While baseline performance was lower for low-resource languages (Latvian, Maltese), fine-tuning brought their accuracy levels up to be comparable to high-resource languages.
Bilingual Dynamics: Bilingual training (e.g., English + Latvian) generally improved performance for the low-resource language without degrading the high-resource language, though it did not always provide an additive boost to the high-resource language itself.
Cross-Lingual Transfer: Most importantly, fine-tuning on one language improved retrieval performance on unseen languages. This suggests that the models learn language-independent, content-level legal representations rather than just language-specific patterns.

5. Significance

This work provides a blueprint for building reliable, multilingual RAG systems for the legal domain. By demonstrating that domain-specific fine-tuning enables cross-lingual transfer, the authors show that even if a developer lacks massive amounts of data for a specific language (e.g., Maltese), they can leverage high-resource legal data (e.g., English) to build effective retrieval tools for that language. This is a critical step toward democratizing access to legal information across the diverse linguistic landscape of the European Union.