💻 computer science

Classifier or Prompt: A Case Study on Legal Requirements Traceability

This paper evaluates automated solutions for legal requirements traceability, demonstrating that a specialized prompt-engineered LLM (RICE_LRT) and a semantic similarity classifier (Kashif) significantly outperform existing baseline methods in identifying trace links between technical requirements and legal regulations like the GDPR.

Original authors: Romina Etezadi, Sallam Abualhaija, Chetan Arora, Lionel Briand

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Romina Etezadi, Sallam Abualhaija, Chetan Arora, Lionel Briand

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Lost in Translation" Game

Imagine you are building a new app, like a ride-sharing service called WeMobilize. You have a team of engineers who write the "rules" for how the app should work (e.g., "Users must be able to delete their accounts").

Now, imagine there is a massive, complex rulebook written by the government called GDPR (a law about protecting people's private data). This rulebook is written in "Legalese"—a very formal, specific, and often vague language.

The Challenge:
Your engineers need to make sure every single rule in their app matches a rule in the government's book. If they miss one, the company could get fined millions of dollars.

The Problem: The engineers speak "Tech," and the law speaks "Law." They don't use the same words.
- Engineer says: "I want to stop getting emails."
- Law says: "The data subject has the right to withdraw consent."
The Old Way: Humans have to read thousands of lines of code and thousands of lines of law, trying to guess which ones match. It's like trying to find a specific needle in a haystack while wearing blindfolded. It takes forever and is prone to mistakes.

The Solution: Two Digital Assistants

The authors of this paper built two different "AI assistants" to help humans find these matches automatically. They tested them to see which one was better.

Assistant #1: Kashif (The "Super-Translator")

How it works:
Think of Kashif as a highly trained librarian who has read millions of books. It uses a technology called "Sentence Transformers."

Instead of just matching words (like "delete" matching "delete"), Kashif understands the meaning of the sentence.
It takes an engineer's requirement and a law, turns them into "mathematical shapes" (vectors), and sees how close those shapes are to each other.
The Analogy: If you ask a normal computer, "Do 'apple' and 'fruit' match?" it might say "No" because the letters are different. Kashif understands that an apple is a fruit, so it says, "Yes, they are very close."

The Result:
Kashif was great at the "easy" test (using a dataset called HIPAA). It found about 63% of the correct matches, beating all the old methods by a huge margin.

The Catch: When the test got harder (using the complex GDPR rules), Kashif got confused. It only found 15% of the matches. It was like a librarian who is great at organizing fiction books but gets lost when you ask them about complex legal contracts.

Assistant #2: Rice LRT (The "Expert Consultant")

How it works:
This assistant uses a Large Language Model (LLM), like the advanced AI behind tools such as ChatGPT (specifically GPT-4o).

Instead of just calculating math, the researchers gave the AI a very specific "job description" (a prompt).
They told the AI: "You are a legal expert. Here are 5 examples of how to match a rule to a law. Now, look at this new rule and tell me which laws it breaks, and explain why."
The Analogy: If Kashif is a librarian who sorts books by color, Rice LRT is a senior lawyer you hire for a consultation. You give them a few examples of your past cases, and they use their reasoning skills to figure out the new case.

The Result:
Rice LRT was a superstar. On the hard GDPR test, it found 84% of the correct matches.

It didn't just say "Yes/No." It gave a rationale (a reason).
- AI says: "This requirement matches the 'Right to Access' law because the user needs a key file to unlock the database, which is a form of authentication."
This helped humans trust the AI and fix the few mistakes it made.

The Showdown: Who Won?

The researchers ran a race between these two assistants and several older, slower methods.

The Old Guard (The Baselines): These were the "dumb" computers from the past (like simple word-counting machines). They performed terribly, often finding less than 20% of the matches. They couldn't understand the difference between "Tech" and "Law."
Kashif (The Classifier): It was the runner-up. It was fast and good at simple tasks, but it struggled when the rules got too complex or the vocabulary was too different.
Rice LRT (The Prompted LLM): It won the race. By using a smart "prompt" (a set of instructions), it understood the nuance of the law better than anything else.

Why This Matters to You

You might think, "I'm not a lawyer or a coder, why do I care?"

Safety & Privacy: This research helps companies build apps that actually protect your data. If the AI can't find the link between your app's features and the privacy laws, your data might be leaked, and the company might get fined.
Speed: Instead of a team of lawyers spending months checking an app, an AI can do the heavy lifting in minutes, leaving the humans to just double-check the final list.
The Future: The paper shows that while "smart math" (Kashif) is good, "smart reasoning" (Rice LRT) is even better for complex legal problems. It proves that we need to teach AI not just to calculate, but to think like a legal expert.

The Bottom Line

The paper concludes that to keep software safe and legal, we can't just use old-school computer tricks. We need to use advanced AI that understands the spirit of the law, not just the words. The "Rice LRT" method, which acts like a smart consultant, is currently the best tool we have for this job.

1. Problem Statement

The paper addresses Legal Requirements Traceability (LRT), the critical task of linking software requirements to specific legal provisions (e.g., GDPR, HIPAA) to ensure regulatory compliance.

The Challenge: Manual traceability is infeasible for complex systems with hundreds of requirements due to time constraints and the need for legal expertise.
Key Obstacles:
1. Language Discrepancy: Legal texts use generic, domain-specific terminology, while software requirements use technical language. There is often minimal lexical overlap, making simple keyword matching ineffective.
2. Data Scarcity: Annotated datasets linking requirements to regulations are rare due to privacy and confidentiality concerns, limiting the training of supervised models.
3. Limitations of Existing Methods: Traditional Information Retrieval (IR) and standard Machine Learning (ML) models rely heavily on lexical overlap and fail to capture deep semantic relationships or indirect logical connections required in legal contexts.

2. Methodology

The authors propose and evaluate two distinct automated approaches based on recent Natural Language Processing (NLP) technologies:

A. Kashif (Classifier Approach)

Core Technology: Sentence Transformers (ST) and Cosine Similarity.
Mechanism:
1. Fine-tuning: A pre-trained Sentence Transformer model is fine-tuned on a labeled dataset of requirement-provision pairs. The model learns to generate dense vector embeddings where semantically similar pairs (linked) are closer in the vector space than dissimilar pairs.
2. Prediction: The system computes cosine similarity between a requirement and all candidate provisions.
3. Thresholding: Trace links are predicted if the similarity score exceeds a threshold ( $\theta$ ). The authors tested four threshold strategies: Constant (0.5), Dynamic (based on negative examples), Maximum Delta (largest gap in similarity), and Tuned (optimized on training data).
Goal: To determine if a fine-tuned semantic similarity model can outperform traditional classifiers.

B. Rice LRT (Prompt-Based Approach)

Core Technology: Large Language Models (LLMs), specifically GPT-4o, utilizing the RICE prompt engineering framework (Role, Instruction, Context, Constraints, Examples).
Mechanism:
1. Prompt Design: The prompt includes:
  - Context: Defines the task as a requirements analyst mapping to GDPR.
  - Examples (Few-Shot): Provides 5 examples of requirements with their trace links and the rationale for the link.
  - Instruction: Explicitly guides the LLM to consider indirect links, prioritize recall, and reason through domain-specific terminology gaps.
  - Output Indicator: Requests a list of regulatory codes and the rationale for each.
2. Execution: The LLM is queried for each requirement to generate trace links and explanations.
Goal: To leverage the reasoning capabilities and pre-trained knowledge of LLMs to bridge the semantic gap without extensive fine-tuning.

3. Experimental Setup

The study evaluates these approaches against seven baselines (LSI, LDA, GloVe, TraceBERT, RoBERTa, LLaMa, and a probabilistic baseline 'B') across two datasets:

HIPAA Dataset: A standard, publicly available benchmark with 1,891 requirements and 10 provisions. Used for initial model selection and baseline comparison.
GDPR Dataset (New): A newly created, more complex dataset containing 310 requirements across four domains (cybersecurity, digital services, etc.) traced to 26 GDPR provisions. This dataset tests generalization to unseen, complex legal contexts.

4. Key Results

RQ1: Model Selection (Kashif)

Among 38 pre-trained Sentence Transformer models, ST29 (paraphrase-multilingual-mpnet-base-v2) achieved the highest Area Under the Curve (AUC) of 0.859 in a zero-shot setting.

RQ2: Performance on HIPAA Benchmark

Kashif (Constant Threshold 0.5) achieved an F2 score of ~63%, outperforming the best baseline (RoBERTa) by 21 percentage points and the best existing literature baseline by 34 percentage points.
It also achieved the highest Mean Average Precision (MAP) of 81.4% and AUC of 0.93, demonstrating superior ranking and discrimination capabilities.
Observation: Traditional IR methods (LSI, LDA) and standard BERT-based models performed poorly, highlighting the difficulty of the LRT task for non-specialized models.

RQ3: Performance on Unseen GDPR Data (Kashif)

When applied to the complex, unseen GDPR dataset without re-fine-tuning, Kashif's performance dropped significantly (Recall ~15%, F2 ~13.5%).
This indicates that while fine-tuning helps, the classifier struggles to generalize across different regulatory domains and complex requirement types without specific domain exposure.

RQ4: Performance of Rice LRT (LLM Approach)

Rice LRT significantly outperformed both Kashif and all baseline prompts on the GDPR dataset.
Metrics: Achieved an 84% Recall and an F2 score of 61%.
Efficiency: It reduced the manual effort for analysts by requiring them to vet only 9.5% of the total provisions while capturing 84% of true links.
Comparison: Rice LRT improved the F2 score by 34 percentage points over the best baseline prompt and 69 percentage points in accuracy over Kashif on the GDPR dataset.
Qualitative Benefit: Unlike classifiers, Rice LRT provided rationales for its predictions, aiding human analysts in validation.

5. Key Contributions

Novel Approaches: Introduced Kashif (a fine-tuned Sentence Transformer classifier) and Rice LRT (a structured, few-shot prompting framework for LLMs) specifically tailored for Legal Requirements Traceability.
Benchmarking: Conducted a comprehensive evaluation against seven state-of-the-art baselines, demonstrating that generic traceability models fail in legal contexts and that specialized solutions are necessary.
New Dataset: Created and released a new, complex dataset linking software requirements to 26 GDPR provisions across diverse domains, addressing the scarcity of annotated LRT data.
Empirical Evidence: Proved that while fine-tuned classifiers (Kashif) work well on familiar domains, engineered prompts with LLMs (Rice LRT) offer superior generalization and reasoning capabilities for complex, unseen legal regulations.

6. Significance and Conclusion

The paper concludes that Legal Requirements Traceability cannot be adequately addressed by generic techniques proposed in existing literature. The semantic gap between legal and technical language requires models capable of deep reasoning.

For Practitioners: The study suggests that Prompt Engineering with LLMs (specifically Rice LRT) is currently the most effective approach for LRT in complex, multi-domain scenarios, offering high recall and explainable rationales.
For Researchers: It highlights the limitations of transfer learning across different regulations (HIPAA $\to$ GDPR) and emphasizes the need for domain-specific prompt design and potentially hybrid approaches (combining retrieval with generation).
Impact: The proposed solutions can significantly reduce the time and cost of regulatory compliance audits, allowing organizations to deploy software with higher confidence in their legal adherence.