💬 NLP

EURO-5K: When Does Domain Pretraining Matter? Benchmarking Transformers for EU Reporting Obligation Extraction

This paper introduces EURO-5K, a specialized dataset for extracting EU reporting obligations, and demonstrates that while legal pretraining offers marginal gains for fully fine-tuned models, it significantly boosts the performance of parameter-efficient tuning and accelerates learning with limited data, ultimately validating both discriminative and generative approaches for regulatory compliance automation.

Original authors: Marios Koniaris, Vasileios Kotronis, Eugenia Giannini, Panayiotis Tsanakas

Published 2026-06-03✓ Author reviewed ⓘ

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Marios Koniaris, Vasileios Kotronis, Eugenia Giannini, Panayiotis Tsanakas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the European Union as a massive library containing 180,000 different rulebooks (laws and regulations) written in a very formal, complex language. Inside these books, there are three main types of instructions, not just two:

Behavioral Rules: "You must do this action" (e.g., "Treat the water to make it safe").
Disclosure Rules: "You must make this information public" (e.g., "Publish your carbon emissions on a public website").
Reporting Rules: "You must send a report about this action to the government" (e.g., "Tell the Commission how much water you treated").

The problem is that these three types of rules often look exactly the same on the page. They all use words like "shall" and "must." Finding the specific "Reporting Rules" manually is like trying to find a specific needle in a haystack the size of a mountain, but with the added difficulty that the "Behavioral" and "Disclosure" needles look almost identical to the "Reporting" ones. Doing this by hand takes forever, costs a fortune, and requires a lawyer to read every single sentence to distinguish between telling the government, telling the public, and just doing the work.

This paper introduces a project called EURO-5K to build a "smart robot" that can automatically find these reporting needles. Here is how they did it, explained simply:

1. The Training Data: A Rigorous Scientific Method

The researchers didn't just "clean up" a messy pile of text; they built a new, gold-standard dataset from scratch.

The Analogy: Imagine you are training a team of detectives. Instead of giving them a stack of old, confusing case files, you create a brand-new training manual. You define exactly what a "Reporting Obligation" looks like using five strict criteria. You then have humans and AI work together to label examples, followed by a dual-blind review (where two experts check each other's work without knowing who did what) to ensure they agree.
The Result: This rigorous process created EURO-5K, a dataset of 5,253 perfectly labeled examples. The team measured their agreement scientifically (a score called kappa of 0.613), proving this wasn't just a quick cleanup, but a high-quality, standalone scientific contribution. They taught the robots to distinguish reporting rules from both behavioral rules and disclosure rules, using "tricky" examples to prevent the robots from cheating with easy keywords.

2. The Contenders: Two Types of Robots

They tested two different kinds of AI "brains" to see which one was better at finding these rules:

The "Highlighter" (Discriminative/BERT): This robot reads a sentence and highlights the specific words that make it a reporting rule. It's like a student underlining the answer in a textbook.
The "Writer" (Generative/LLM): This robot reads the sentence and writes out the answer from scratch. If it sees a reporting rule, it copies the sentence; if not, it says "None." It's like a student writing the answer on a blank sheet of paper.

They tested these robots in two ways:

Full Training (Fine-tuning): Teaching the robot everything from scratch using the new legal data.
Efficient Training (QLoRA/LoRA): Teaching the robot using a "shortcut" method that updates only a tiny fraction of its brain (like putting a new appendix in a book rather than rewriting the whole thing). This saves massive amounts of computer power.

3. The Big Questions & Results

Q: Do we need a robot that was already trained on legal books, or will a generic robot work?

The Finding: Surprisingly, a generic robot trained on general text performed almost exactly as well as a robot specifically trained on legal text.
The Twist: This isn't just a lucky guess. The researchers used advanced statistical tests (Welch's t-tests and bootstrap resampling) to prove that the difference between the "legal specialist" and the "generalist" is statistically non-significant.
The Analogy: It's like finding that a general mechanic is just as good at fixing a specific type of car engine as a specialist mechanic, if you give them the right manual and enough time to practice. The "legal pre-training" didn't give a real advantage; the data and training method matter more.

Q: Which robot type is better: The Highlighter or The Writer?

The Finding: They are essentially tied. Both the "Highlighter" and the "Writer" achieved similar high scores (an F1-score of about 0.891). Note: F1-score is a metric that balances how many rules the robot found correctly against how many it missed, which is crucial when the data is tricky.
The Real "Twist": The "shortcut" training (efficient methods) did not beat the full training. In fact, for both robot types, Full Training significantly outperformed Efficient Training (p<0.01).
The Competitive Edge: However, there was a fascinating crossover. A "Writer" (Generative) model using the efficient shortcut training slightly edged out the best "Highlighter" (Legal-BERT) that used full training. While this difference was small and not statistically significant (p=0.082), it shows that a smart, efficient generative model can compete with a heavy, fully-trained discriminative model.

Q: How much data do we need?

The Finding: The robots learned very fast at the beginning, but after about 3,000 examples, they stopped getting much better.
The Analogy: It's like learning to ride a bike. You wobble a lot at first, but once you get the hang of it (around 3,000 miles of practice), adding more miles doesn't make you a much better rider. This proves their dataset of 5,000 examples was "just right"—not too small, not wastefully large.

Q: Do the robots actually understand the law, or are they just guessing?

The Finding: The researchers tested the robots on new laws they had never seen before (including financial laws).
The Result: The robots were very good at saying "No" to rules that weren't reporting rules (like rules about public safety or behavior). They didn't get confused. They acted like specialized detectives, not general guessers.

4. Why This Matters: The Real-World Impact

This isn't just about better AI; it's about saving billions of euros and reducing red tape for businesses.

The Stakes: A real-world example cited in the paper is the 2025 EU Omnibus simplification package. By identifying overlapping reporting obligations across three sustainability frameworks, the EU was able to remove about 80% of companies from unnecessary reporting scopes. This is projected to save roughly EUR 4.4 billion per year.
The Goal: With the EU having on the order of 180,000 legal acts, manually finding these overlaps is impossible at scale. This paper provides the first open dataset, trained models, and a deployment-ready tool to automate this analysis. This directly supports the European Commission's target of cutting regulatory burdens by 25%.

5. The "Magic" Tool

The team didn't just stop at the research. They built a public website where anyone can paste a chunk of EU law, and the robot will:

Find the reporting rules.
Show you why it found them (highlighting the specific words like "notify" or "Commission").
Export the results into a structured format that computers can use to build databases.

Summary

The paper concludes that we don't need expensive, specialized legal AI to solve this problem. A standard, well-trained AI, using smart and rigorous training methods, can do the job just as well. They have proven that we can automate the tedious task of finding "who needs to report what" in EU laws, saving time and money. By providing the data, the models, and the tools, they have given policymakers and businesses the power to tackle regulatory complexity at scale, directly supporting the EU's mission to reduce burdens and foster growth.

Technical Summary: EURO-5K and Benchmarking Transformers for EU Reporting Obligation Extraction

Problem Definition

The extraction of reporting obligations from European Union (EU) legislation is a critical task for assessing and reducing regulatory burdens. However, distinguishing specific reporting requirements (data transmission to authorities) from structurally similar behavioral obligations (conduct requirements) or disclosure obligations (public transparency) requires specialized legal understanding. Current Natural Language Processing (NLP) methods lack specialized datasets with clear guidelines and comparative evaluations of extraction paradigms, particularly regarding the efficacy of domain adaptation and parameter-efficient training strategies for this specific task.

Methodology

Dataset Curation: EURO-5K

The authors curated EURO-5K, a corpus of 5,253 sentence-level examples derived from 136 EU legislative acts. The dataset was constructed from the raw Annotation of Reporting Obligations in EU Legislation Dataset (AROLD), which underwent a rigorous multi-stage curation process to address structural noise, multi-sentence segmentation issues, and misclassifications.

Composition: 1,751 positive examples (reporting obligations) and 3,502 negative examples.
Hard Negatives: 532 negative examples (10.3%) were specifically selected to represent challenging boundary cases, such as behavioral requirements and procedural coordination, to prevent superficial pattern learning.
Annotation Protocol: A definition of reporting obligations was operationalized using a five-criteria annotation framework to distinguish them from behavioral and disclosure obligations, requiring mandatory language, a reporting action, and a target regulatory authority. The validation pipeline combined rule-based filtering, LLM-assisted review, and dual-blind human validation, achieving an inter-annotator agreement of kappa = 0.613. This rigorous methodology establishes EURO-5K as a distinct, high-quality contribution to legal NLP.

Experimental Design

The study compares two extraction paradigms across generic and legal-domain transformers:

Discriminative Token Classification: Using BERT-base and Legal-BERT.
Generative Span Extraction: Using Llama-3.1-8B, Mistral-7B, and Saul-7B (a Mistral variant with legal continued pre-training).

Training Strategies:

Full Fine-Tuning (FFT): Updating all parameters.
Parameter-Efficient Tuning: LoRA for BERT models and QLoRA (4-bit quantization + LoRA) for LLMs.
Baselines: Rule-based Regex/Keyword matching, Dependency Parsing, and Few-Shot Prompting (without parameter updates).

Evaluation Framework:

Metrics: Precision, Recall, and F1-score based on exact span matches.
Statistical Validation: Welch's t-test for multi-seed BERT comparisons and bootstrap resampling (1,000 iterations) for LLMs to estimate confidence intervals.
Cross-Dataset Evaluation: Tested on an external EU regulatory corpus (Brandsma et al., 2025) to assess specificity (rejecting non-reporting statements) and on a financial reporting corpus (Chuor, 2025) to assess zero-shot sensitivity.
Explainability: LIME for BERT and attention weight analysis for LLMs.

Key Results

Model Performance

Parity of Paradigms: Both discriminative (BERT) and generative (LLM) approaches achieved comparable high performance. The best generative model (Llama-3.1-8B with QLoRA) achieved an F1 of 0.891, slightly outperforming the best discriminative model (Legal-BERT with FFT at 0.883), though the difference was not statistically significant ( $p=0.082$ ).
Domain Adaptation: Legal pretraining offered only marginal gains. Legal-BERT outperformed generic BERT by 1.8 F1 points in full fine-tuning, but this difference was not statistically significant ( $p=0.307$ ). Similarly, for generative models, the legally pre-trained Saul-7B performed negligibly better than the generic Mistral-7B (0.3-point gap).
Training Strategy: Full fine-tuning significantly outperformed parameter-efficient methods (LoRA/QLoRA) in terms of F1-score ( $p<0.01$ ), confirming an accuracy-efficiency trade-off. However, parameter-efficient methods still achieved strong results (e.g., Legal-BERT LoRA: 0.791 F1).
Baselines: Supervised fine-tuning provided substantial gains over baselines. Few-shot prompting (0.762 F1) and dependency parsing (0.727 F1) were competitive but inferior to fine-tuned models.

Data Efficiency and Learning Curves

Convergence: Learning curve analysis demonstrated that all models converge around 3,000 samples, with diminishing returns thereafter, validating the sufficiency of the EURO-5K dataset size.
Early Learning: Legal pretraining (specifically Saul-7B) accelerated early learning in low-data regimes (e.g., achieving nearly half its full performance with only 10 samples), but this advantage disappeared as data volume increased.

Generalization and Specificity

Specialized Learning: Cross-dataset evaluation confirmed that models act as specialized reporting obligation extractors rather than generic regulatory classifiers. On an external corpus of general regulatory statements, models correctly rejected the majority of non-reporting obligations (low recall of 12–17%), demonstrating high specificity.
Zero-Shot Sensitivity: On an out-of-domain financial reporting corpus, models achieved high zero-shot recall (88.7%–90.3%), indicating that the semantic structure of reporting obligations was learned rather than mere memorization of the training distribution.

Explainability

Models consistently emphasized institutional actors (e.g., "Commission," "Member States") and regulatory frameworks.
Crucially, models evaluated semantic context rather than relying solely on keywords. For instance, they correctly distinguished between "shall notify" (reporting) and "shall make public" (disclosure) within the same sentence, assigning negative weights to disclosure terms.

Significance and Contributions

The paper claims the following contributions:

EURO-5K Dataset: The release of the largest annotated corpus for reporting obligation extraction, featuring a principled five-criteria annotation framework, an LLM-assisted plus dual-blind human validation pipeline (kappa = 0.613), and challenging hard negatives.
Paradigm Comparison: The first systematic comparison of discriminative and generative paradigms for this task, revealing that generative models can match or exceed discriminative performance when properly optimized.
Domain Adaptation Insights: Evidence that systematic hyperparameter optimization allows generic models to approach the performance of domain-adapted models, suggesting that legal pretraining offers modest, non-significant benefits for this specific task when resources are optimized.
Parameter Efficiency: Demonstration of the accuracy-efficiency trade-offs between full fine-tuning and parameter-efficient methods (LoRA/QLoRA) in a legal context.
Policy Significance and Practical Deployment: The authors contextualize their work within the 2025 EU Omnibus simplification package, which identified overlapping reporting obligations across three sustainability frameworks, removed ~80% of companies from reporting scope, and is projected to save ~EUR 4.4 billion annually. Given the EU's ~180,000 legal acts, the release of the open EURO-5K dataset, trained models, and a deployment-ready tool (featuring an interactive web interface with explainability visualizations and an RDF export tool compliant with the EU's Reporting Requirements Metadata Vocabulary) enables automating such obligation analysis at scale. This directly supports the European Commission's target of a 25% regulatory-burden reduction.

The authors conclude that while domain pretraining offers minor accelerations in low-data regimes, the choice of model scale and training strategy (full vs. efficient) is more critical than the domain-specific initialization for achieving state-of-the-art extraction performance.