💬 NLP

Parameter Efficient Fine Tuning Llama 3.1 for Answering Arabic Legal Questions: A Case Study on Jordanian Laws

This study demonstrates that parameter-efficient fine-tuning with LoRA and 4-bit quantization using the Unsloth framework significantly enhances the Llama-3.1 model's ability to answer Arabic legal questions based on Jordanian laws, achieving improved accuracy and resource efficiency.

Original authors: Mohammed Fasha, Bassam Hammo, Bilal Sowan, Husam Barham, Esam Nsour

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohammed Fasha, Bassam Hammo, Bilal Sowan, Husam Barham, Esam Nsour

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, well-read librarian named Llama. This librarian has read almost everything in the world and knows how to speak many languages, including Arabic. However, if you ask this librarian a very specific question about Jordanian law (like "What happens if I get arrested for a specific crime?"), they might give you a vague, generic answer. They know the language, but they haven't studied the specific rulebook of Jordan's legal system.

This paper is about teaching that librarian to become a specialist.

The Problem: The Generalist vs. The Specialist

The authors noticed that while big AI models (like Llama 3.1) are great at general conversation, they struggle with complex, specific tasks like answering legal questions in Arabic. It's like asking a general doctor to perform heart surgery; they know medicine, but they need specific training for that specific job. Also, training a massive AI model usually requires a supercomputer the size of a building, which is expensive and hard to access.

The Solution: "Training Wheels" for the AI

The researchers used a clever trick called Parameter Efficient Fine-Tuning (PEFT).

The Analogy: Imagine the Llama model is a giant, heavy truck. To teach it a new route (Jordanian laws), you don't need to rebuild the whole engine or the chassis. Instead, you just attach a small, lightweight GPS module (called a LoRA adapter) to the dashboard.
The Result: This small module teaches the truck exactly where to go without needing to upgrade the entire vehicle. This makes the training process fast, cheap, and possible on a standard cloud computer (like a Google Colab Pro).

They also used 4-bit quantization, which is like compressing a high-definition movie into a smaller file size so it fits on your phone without losing too much quality. This saved even more memory.

The Training Data: Building a Custom Textbook

To teach the AI, the authors didn't just throw random books at it. They:

Collected 18 specific Jordanian laws (covering things like civil law, labor, crimes, and courts).
Extracted 3,578 specific articles from these laws.
Created a "Flashcard" Deck: They used another AI to turn these dry legal articles into 6,000 Question-and-Answer pairs.
- Example: Instead of just reading the law, the AI learned: "Question: When can a request for reconsideration be accepted? Answer: Before entering the trial."

The Experiment: The Showdown

The researchers tested two versions of the AI:

The Base Model: The "raw" Llama 3.1 (the general librarian).
The Instruct Model: A version of Llama 3.1 that was already taught to follow instructions well.

They then took both of these and applied their "GPS module" (fine-tuning) using the Jordanian legal flashcards.

The Results: Who Won?

The paper compared the answers using a scoring system (BLEU and ROUGE) that measures how close the AI's answer is to a human-written "correct" answer.

The Winners: Both fine-tuned models scored significantly higher than the untrained versions. They learned the specific legal jargon and rules much better.
The Surprise: The "raw" Llama 3.1, after training, actually performed slightly better than the "Instruct" version.
- Why? The authors suggest the "Instruct" model might have had some "pre-conceived notions" from its previous training that got in the way. The "raw" model was like a blank slate, so it could absorb the Jordanian legal rules perfectly without any old habits interfering.

What Did the AI Actually Do Better?

The paper shows that before training, the AI would give long, vague, or slightly wrong answers. After training:

Precision: If the law said a request must be made before a trial starts, the trained AI said exactly that. The untrained AI might have given a generic explanation about "court procedures."
Context: When asked about urgent measures (like detention), the trained AI knew exactly which specific legal steps applied, whereas the untrained one was confused.

The Bottom Line

This paper proves that you don't need a supercomputer to make a massive AI expert in a specific field. By using efficient "training wheels" (PEFT) and a good set of "flashcards" (the 6,000 legal Q&A pairs), you can turn a general AI into a specialist in Jordanian Law.

The authors conclude that this method works well and sets the stage for building tools that can help people understand legal documents, though they note this specific study focused only on the training part, not on building a full search engine yet.

Technical Summary: Parameter Efficient Fine Tuning Llama 3.1 for Answering Arabic Legal Questions

Problem Statement
The paper addresses the challenge of applying Large Language Models (LLMs) to domain-specific Arabic legal queries, particularly within the context of Jordanian law. While models like Llama-3.1 show general adaptability, they face significant hurdles in the legal domain due to the complexity of Arabic linguistics, the scarcity of annotated datasets, and the need for precise reasoning rather than general knowledge retrieval. Existing Arabic NLP systems often lack the specialized training required to interpret legal terminology and procedural nuances accurately. Furthermore, standard fine-tuning approaches are often computationally prohibitive, limiting their accessibility for researchers working with resource-constrained environments.

Methodology
The study employs a structured pipeline to adapt Llama-3.1 models for Jordanian legal question-answering (QA):

Data Curation: The authors collected and curated articles from 18 specific Jordanian laws, including the Civil Law, Penal Code, Labor Law, and Electronic Crimes Law. From these texts, 3,578 legal articles were extracted.
Dataset Generation: Using the OpenAI API (GPT-3.5 Turbo), the team generated a custom dataset of approximately 6,000 structured (Question, Answer, Context) triples based on the extracted legal articles. This dataset was split into 80% for training and 20% for testing.
Model Selection and Quantization: Two variants of the Llama-3.1-8B model were selected: the base model and the instruction-tuned variant. Both were quantized to 4-bit precision (bnb-4bit) to reduce memory overhead.
Parameter-Efficient Fine-Tuning (PEFT): The models were fine-tuned using Low-Rank Adaptation (LoRA) adapters. The configuration included a rank ( $r$ ) of 16, a LoRA alpha of 16, and no dropout. Training was conducted on a Google Colab Pro environment utilizing an NVIDIA A100-SXM4-40GB GPU.
Training Parameters: The models were trained for one epoch with a learning rate of $2 \times 10^{-4}$ , a batch size of 2 (utilizing gradient accumulation), and FP16/BF16 precision. The Unsloth framework was leveraged to accelerate the training process.

Key Results
The performance of the fine-tuned models was evaluated against their respective base models using BLEU and ROUGE metrics (ROUGE-1, ROUGE-2, and ROUGE-L).

Quantitative Improvements: Both fine-tuned models significantly outperformed their base counterparts.
- The base llama3.1 achieved a BLEU score of 0.058, while fine_tuned_llama3.1 reached 0.290.
- The base llama3.1_instruct scored 0.128 on BLEU, while fine_tuned_llama3.1_instruct scored 0.270.
- Similar improvements were observed across all ROUGE metrics, indicating that the fine-tuned models generated responses much closer to the human-written ground truth.
Model Comparison: Interestingly, the fine-tuned base model (fine_tuned_llama3.1) slightly outperformed the fine-tuned instruction model (fine_tuned_llama3.1_instruct) in BLEU and ROUGE scores. The authors suggest this may be due to the base model's neutral initialization allowing for more specific adaptation to the legal task, whereas the instruction-tuned model may have carried prior biases conflicting with the specific dataset patterns.
Qualitative Analysis: Human evaluation of random samples revealed that fine-tuned models provided concise, procedurally accurate answers aligned with legal ground truths. In contrast, base models often produced verbose, generalized, or procedurally incorrect responses. For instance, in questions regarding procedural timing or urgent measures, the fine-tuned models correctly identified specific legal conditions (e.g., reporting crimes before discovery), whereas base models failed to capture these nuances.

Key Contributions

Dataset Creation: The creation and public release of a curated dataset of 6,000 Jordanian legal QA pairs, derived from 18 specific laws.
Efficient Adaptation Strategy: Demonstration of a resource-efficient workflow using 4-bit quantization, LoRA, and the Unsloth framework to fine-tune Llama-3.1 on a single consumer-grade GPU (via Colab), making domain adaptation accessible.
Empirical Evidence: Providing quantitative and qualitative evidence that fine-tuning significantly enhances LLM performance in the Arabic legal domain, specifically for Jordanian law.

Significance and Claims
The paper claims that this work serves as a foundational step for building offline Retrieval-Augmented Generation (RAG) systems for Arabic legal applications. By demonstrating that fine-tuned LLMs can effectively understand and answer specific legal questions, the study underscores the potential of adapting large language models for specialized, low-resource domains. The authors emphasize that their approach offers a replicable methodology for advancing Arabic NLP in legal contexts, paving the way for practical tools such as legal research assistants and automated document analysis. The study concludes that while fine-tuning is effective, future work should focus on integrating these models into full RAG pipelines and expanding datasets to include diverse Arabic dialects and unstructured legal documents.