💬 NLP

DistilledGemma: Balanced Efficiency-Accuracy for Person-Place Relation Extraction from Multilingual Historical Articles

The paper presents DistilledGemma, a three-stage knowledge distillation pipeline that leverages a 26B Gemma 4 teacher model to train a compact 2.3B student model for multilingual person-place relation extraction in historical articles, achieving top-tier efficiency-accuracy rankings in the HIPE-2026 shared task.

Original authors: Youssef Aboelwafa, Ahmed Samir, Nagwa Elmakky, Marwan Torki

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Youssef Aboelwafa, Ahmed Samir, Nagwa Elmakky, Marwan Torki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, dusty library of old newspapers from the 1800s written in English, German, and French. These papers are full of stories about people and places, but they are messy: the ink is faded, the text is smudged (like a bad photocopy), and the sentences are tricky to understand.

Your goal is to answer two simple questions for every person and place mentioned in these articles:

The "At" Question: Did this person ever visit this place? (Maybe they lived there years ago, or maybe they just passed through).
The "IsAt" Question: Is this person currently at this place right now, in the story being told?

This is the challenge the paper tackles. The authors, a team from Alexandria University, built a system called DistilledGemma to solve this. Here is how they did it, explained simply:

The Problem: The "Brain" vs. The "Backpack"

To read these old, messy newspapers well, you need a very smart "brain" (a huge computer model). The authors started with a giant brain called Gemma 26B. It's incredibly smart and can figure out the tricky connections between people and places, but it's also huge, heavy, and expensive to run. It's like trying to carry a full-sized library in your backpack just to read a single page.

The team wanted a solution that was smart enough to get the job done but small enough to fit in a regular backpack (a standard computer) so it could be used easily by historians and researchers.

The Solution: A Three-Step "Apprentice" Plan

Instead of just using the giant brain or trying to make a tiny brain guess on its own, they used a clever three-step teaching method called Knowledge Distillation. Think of it like a master chef training a junior chef.

Step 1: Finding the Best Master Chef
First, they tested eight different "brains" (AI models) to see which one was the best at reading these old newspapers. They found that the giant Gemma 26B was the smartest. They decided this would be the "Teacher."

Step 2: The Teacher Does the Homework
The Teacher (the giant model) read through thousands of the old newspaper articles. But instead of just giving a simple "Yes" or "No" answer, the Teacher was asked to write down its thought process.

Example: "I see the word 'Paris' and 'Napoleon'. The text says he was there in 1800, so the answer is 'Yes', but the text doesn't say he is there today, so the second answer is 'No'."
This created a massive set of "silver standard" homework answers that included the reasoning, not just the final grade.

Step 3: The Student Learns from the Notes
Now, they took a much smaller, lighter brain called Gemma 2.3B (the "Student"). Instead of just showing the Student the correct answers, they showed them the Teacher's notes and reasoning.
The Student studied these notes, learning how the Teacher thought. This is like a student learning not just the math answers, but the logic behind them. The result? The Student became very good at the task, but it was 11 times smaller than the Teacher.

The Safety Net: The Rulebook

Even with a smart Student, the AI can sometimes get confused, especially because the old newspapers have so many errors (like typos from bad scanning). To fix this, the team added a "Rulebook" (post-processing).

If the AI says a person is "Currently at" a place, the Rulebook forces it to also say they were "At" that place at some point. (You can't be there now if you've never been there before).
It also checks if the person and place actually make sense together based on the text.

The Results: Small but Mighty

When they tested their system in a big competition (HIPE-2026):

Accuracy: The small Student model performed almost as well as the giant Teacher (about 88% as good).
Ranking: They came in 3rd place for overall accuracy and 2nd place for the "balanced" score (which rewards being both accurate and efficient).
Efficiency: The final system is so small it can run on a single, standard computer graphics card, whereas the "Teacher" would need a massive, expensive supercomputer.

In a Nutshell

The paper shows that you don't need a supercomputer to understand history. By having a super-smart AI teach a smaller, cheaper AI how to think (not just what to answer), you can get nearly the same results with a fraction of the cost and effort. It's like teaching a child to be a historian by letting them read the notes of a professor, rather than hiring the professor to do all the work.

Technical Summary: DistilledGemma for Person-Place Relation Extraction

Problem Definition

The paper addresses the challenge of Person-Place Relation Extraction from multilingual historical newspaper articles, specifically within the context of the HIPE-2026 shared task. The goal is to classify the relationship between extracted person and place entities in English, German, and French texts. The task involves predicting two specific relations:

at: Whether the person has ever been associated with the place (labels: TRUE, PROBABLE, FALSE).
isAt: Whether the person is at the place within the immediate temporal horizon of the article (labels: TRUE, FALSE).

The task presents significant hurdles, including:

Noisy OCR Text: Historical documents contain frequent optical character recognition errors that degrade entity mention quality.
Class Imbalance: The negative labels dominate the dataset (approx. 55% for at and 78% for isAt), biasing models toward false predictions.
Multilingual Reasoning: Models must perform consistently across three typologically diverse languages with varying data availability.
Implicit Evidence: Many connections are implied rather than explicit, requiring world knowledge and contextual inference.

While Large Language Models (LLMs) show promise, deploying models with sufficient capacity (e.g., 26B+ parameters) for processing millions of historical documents is computationally impractical. The core research question is whether the relation extraction capabilities of a large, fine-tuned LLM can be distilled into a smaller model suitable for practical deployment without catastrophic performance loss.

Methodology

The authors propose DistilledGemma, a three-stage knowledge distillation pipeline designed to balance classification accuracy with computational efficiency.

Stage 1: Model Selection and Prompt Engineering

The authors systematically evaluated eight LLM configurations across four families (Gemma, Qwen, Mistral, HY-MT) using various prompt engineering strategies.

Prompt Design: Five prompt variants were tested, with a Chain-of-Thought (CoT) template proving most effective. This prompt frames the model as a historian, explicitly defines the at and isAt relations, enforces logical constraints (e.g., if isAt is TRUE, at must be TRUE), and requires concise, evidence-grounded JSON outputs.
Selection: The Gemma 4 26B A4B model was selected as the "teacher" due to its superior zero-shot macro-averaged recall (0.7255) and its alignment with the target student family.

Stage 2: Teacher Fine-Tuning and Silver Data Generation

The selected teacher model was fine-tuned using QLoRA (Quantized Low-Rank Adaptation) on the HIPE training data.

Configuration: LoRA rank $r=32$ , $\alpha=64$ , applied to all attention and MLP projections. Training ran for 3 epochs with a batch size of 32 (via gradient accumulation).
Data Generation: The fine-tuned teacher generated silver-standard training data for the entire corpus. Crucially, these outputs included not just the predicted labels but also Chain-of-Thought explanations, capturing the reasoning process behind each decision.

Stage 3: Student Distillation

A compact Gemma 4 E2B model (approx. 2.3B parameters) served as the "student."

Training Strategy: The student was trained on the teacher's generated silver data using QLoRA ( $r=16, \alpha=32$ ) for 6 epochs.
Distillation Approach: This was a response-level distillation where the student learned to mimic the teacher's structured reasoning patterns and label outputs, rather than training solely on gold labels.
Deployment: The LoRA adapters were merged into the base model for inference, resulting in a deployable 2.3B parameter model.

Post-Processing

To address class imbalance and ensure semantic consistency, a rule-based post-processing step was applied to all outputs. This enforced:

Entity-type compatibility.
Relation-direction constraints (person-to-place).
Contextual trigger patterns.
Logical constraints (specifically, isAt=TRUE implies at=TRUE).

Key Results

The system, submitted by team "WHEREAMI," achieved the following results in the official HIPE-2026 evaluation:

Standard Accuracy Profile: Ranked 3rd with a mean profile score of 0.688.
Binary Accuracy Profile: Ranked 2nd with a mean profile score of 0.8156.
Balanced Efficiency-Accuracy Profile: Ranked 2nd on both standard and binary test sets.

Distillation Effectiveness:

The distilled 2.3B student model recovered approximately 88% of the 26B teacher's macro-averaged recall performance.
This was achieved with an 11x reduction in deployed parameter count.
The distilled student outperformed several larger zero-shot models (e.g., Qwen3.5 9B and Gemma 4 E4B) in macro-averaged recall, validating that distillation can bridge the performance gap between small and large models.

Efficiency:

The 2.3B student model can be served on a single consumer GPU (<8 GB VRAM), processing ~50 pairs per second.
In contrast, the 26B teacher requires high-memory hardware and was used only offline for data generation.

Significance and Claims

The paper claims that DistilledGemma provides a practical and scalable solution for historical document processing. Its primary significance lies in demonstrating that:

Knowledge Distillation is Viable for Historical NLP: It is possible to transfer complex reasoning capabilities from a large, fine-tuned LLM to a compact model without significant performance degradation.
Reasoning Transfer Matters: Training on the teacher's Chain-of-Thought traces (response-level distillation) allows the student to learn structured inference patterns, which is more effective than training on labels alone, particularly for tasks requiring nuanced evidence assessment.
Efficiency-Accuracy Trade-off: The approach achieves competitive performance in multilingual relation extraction without the excessive computational costs associated with deploying 26B+ parameter models, making it suitable for large-scale digital humanities pipelines.

The authors conclude that while large models dominate in zero-shot settings, a carefully engineered distillation pipeline offers a superior balance for real-world deployment scenarios where computational resources are constrained.