💬 NLP

Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh

This paper demonstrates that fine-tuning small bilingual language models on context-injected legal examples significantly improves their ability to correctly apply retrieved Bangladeshi statutory provisions and maintain language consistency, though these gains vary by model size and do not uniformly benefit larger models.

Original authors: Moniruzzaman Mahadi, Abrar Mohammed Tanzim Alam, Sayma Siddika Monalisa, Mir Mohammad Asif Abdullah, Swakkhar Shatabda, Md Adnan Arefeen

Published 2026-07-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Moniruzzaman Mahadi, Abrar Mohammed Tanzim Alam, Sayma Siddika Monalisa, Mir Mohammad Asif Abdullah, Swakkhar Shatabda, Md Adnan Arefeen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers are like incredibly fast, super-smart students who have read almost every book in the library. These students are called "Large Language Models." They are great at chatting, writing stories, and solving puzzles, but they have a tricky habit: sometimes they make things up or forget the specific rules of a game they are playing. In the field of Artificial Intelligence, researchers are constantly trying to teach these digital students how to stop guessing and start following the rules exactly. One popular way to do this is "retrieval," which is like handing the student a specific page from a textbook right before they take a test, hoping they will read it and use that information to answer correctly. But here is the big question: if you hand a small, less powerful student that textbook page, will they actually read it and use it, or will they just ignore it and keep guessing? This is the puzzle scientists in Bangladesh are trying to solve, especially because legal systems there rely on specific laws written in both English and Bengali, and getting the answer wrong can have serious real-world consequences.

The researchers behind this study decided to test a clever idea: instead of just hoping the student reads the book when they get to the test, what if we train them with the book open in front of them from the very beginning? They created a special training set of 2,165 legal questions and answers, where every single question came with the exact law needed to solve it, written right next to it. They then took a small, open-source computer brain called Qwen3.5 and gave it three different sizes: a tiny 0.8-billion-parameter version, a medium 2-billion version, and a slightly larger 4-billion version. They taught these models using their "context-injected" method, where the law was always part of the lesson.

After the training, they put the models to the test using real exam questions from the Bangladesh Bar Council, the organization that tests lawyers. They gave the models the same questions again, but this time, they didn't just let the models guess; they also tried giving them the relevant laws using two different search methods (BM25 and FAISS) to see if the models could actually use the information provided. The results were a mix of great success and surprising confusion. For the smallest model (0.8B), the training worked like magic. Before training, it barely got any questions right when laws were provided, scoring just 2 out of 100. After training, it jumped to 34 out of 100. The medium model (2B) also got significantly better, showing that it learned to actually read the laws it was given.

However, the story changed for the largest model (4B). While it got slightly better at answering in the correct language, it actually got worse at answering the English questions when the laws were provided, suggesting that for this size, the training might have confused it rather than helped. Another huge win for the training was fixing a silly mistake: before training, when asked a question in Bengali, the models often ignored the language and answered in English about 44% to 53% of the time. After training, this mistake almost vanished, dropping to less than 1%. The study suggests that for smaller models, teaching them to use the laws they are given is a game-changer, but for larger models, it's not a guaranteed fix, and simply having the right information isn't enough if the model doesn't know how to use it.

Technical Summary: Context-Injected Fine-Tuning for Legal QA in Bangladesh

Problem Statement

Small language models (SLMs) often fail to utilize governing statutory provisions provided in their context, even when those provisions are explicitly present in the prompt. This issue is particularly acute in low-resource languages like Bengali, where legal terminology combines colonial-era and Persian-influenced terms with standard Bengali. While Retrieval-Augmented Generation (RAG) aims to ground model outputs in retrieved evidence, existing benchmarks often assume retrieval quality is the primary bottleneck. This paper investigates whether the bottleneck lies instead in the model's ability to use supplied context. Specifically, it asks: Does supervised fine-tuning (SFT) on legal QA pairs where the relevant statutory text is already injected into the training prompt improve a small model's ability to utilize retrieved context at inference time?

Methodology

Dataset Construction

The authors curated a bilingual (Bangla and English) dataset of 2,165 legal QA records derived from six primary Bangladeshi acts and three schedules that dominate the Bangladesh Bar Council examination syllabus:

Acts: Penal Code (1860), Code of Criminal Procedure (1898), Evidence Act (1872), Code of Civil Procedure (1908), Specific Relief Act (1877), and Limitation Act (1908).
Structure: The corpus includes single-hop extraction, advanced selection (distractor-based), and bar-exam style synthetic MCQs.
Context Injection: A critical design choice was "Context-Injected Fine-Tuning." Every training example was formatted as a retrieval-augmented prompt, containing a "Retrieved Sections" block filled deterministically with the governing provision and distractors. This forces the model to learn to ground answers in supplied text rather than parametric memory.

Models and Training

Base Models: Qwen3.5 series at 0.8B, 2B, and 4B parameters.
Fine-Tuning: LoRA (Low-Rank Adaptation) was applied to all seven linear projections (r=32, $\alpha$ =64). Approximately 1% of parameters were trainable.
Configuration: Training used AdamW 8-bit, a learning rate of $2 \times 10^{-4}$ with cosine decay, 2 epochs, and 2,048-token sequences. The training prompt template was preserved exactly for inference to ensure consistency.

Evaluation Protocol

Benchmark: The 2022 and 2023 Bangladesh Bar Council MCQ examinations (original Bangla and machine-translated English).
Conditions: Models were tested under three retrieval conditions:
1. None: No retrieved context.
2. BM25: Top-5 sections retrieved via sparse retrieval.
3. FAISS: Top-5 sections retrieved via dense retrieval (multilingual-MiniLM).
Scoring: Evaluation used strict consistency (all three fixed-seed runs must be correct), majority vote, and mean accuracy. Statistical significance was assessed using paired McNemar tests and randomization tests with Holm correction.

Key Results

1. Fine-Tuning Improves Context Utilization in Small Models

The study demonstrates that fine-tuning on context-injected data significantly improves performance for smaller models (0.8B and 2B) when retrieval is present.

0.8B Model: The strict-consistency score for the 2022 English exam with FAISS retrieval jumped from 2 to 34 (out of 100) after fine-tuning. The overall mean strict accuracy increased from 1.8 to 15.9.
2B Model: Showed consistent gains across 10 of 12 test cells, with mean strict accuracy rising from 10.0 to 17.8.
Retrieval Synergy: While retrieval alone provided negligible or negative gains for base models (e.g., -1.0 points for 0.8B), the fine-tuned counterparts leveraged the same retrieved text to achieve significant gains (+8.1 points for 0.8B).

2. Diminishing Returns at Larger Scales (4B)

The benefits of context-injected fine-tuning did not scale linearly.

4B Model: Fine-tuning resulted in a net regression in overall strict-consistency scores (-2.1 points). While Bangla performance improved slightly (15.7 to 17.3), several English conditions regressed. The base 4B model already utilized retrieval effectively (+6.6 gain), leaving less room for improvement via fine-tuning.

3. Elimination of Language Drift

A significant finding was the correction of language adherence. Base models frequently answered Bangla exam questions in English (44.0%–53.2% of outputs).

Impact: Fine-tuning reduced this "language drift" to below 1% (0.2%–0.7%) across all scales, with statistically significant differences ( $p < .001$ ).

4. Performance Gap Between Languages

Across all scales and conditions, models performed significantly better in English than in Bangla. Even the informal ceiling provided by GPT OSS 120B showed a disparity (72 vs. 42 on specific tasks), highlighting the challenges remaining for low-resource legal NLP in Bengali.

Significance and Claims

The paper claims that retrieval quality is not the only bottleneck in legal RAG systems. Even when the correct statutory provision is supplied, small bilingual models may fail to use it or may answer in the wrong language.

Context Utilization: The study establishes that supervised fine-tuning on examples containing the relevant law is a necessary step to teach small models how to ground their answers in supplied context.
Scale Dependency: The efficacy of this approach is highly dependent on model size. It yields substantial gains for models under 4B parameters but offers no detectable net benefit (and potential regression) for 4B models in this specific domain.
Language Control: Fine-tuning serves as a robust mechanism for enforcing language constraints in bilingual legal settings, preventing the model from defaulting to English when queried in Bengali.

The authors conclude that while language technology cannot fix systemic legal issues, it can lower the cost of accessing accurate legal information, provided the models are explicitly trained to remain grounded in the governing statute and adhere to the requested language. The dataset and code are publicly available to facilitate further research in low-resource legal QA.