💬 NLP

ImmigrationQA: A Source-Grounded Dataset and Small-Model Adaptation for U.S. Immigration Law

This paper introduces ImmigrationQA, a source-grounded dataset of over 17,000 question-answer pairs derived from U.S. immigration regulations, and demonstrates that fine-tuning a small 3B-parameter Llama 3.2 model on this data significantly improves its performance on procedural immigration queries compared to larger base models, all while maintaining a low computational cost.

Original authors: Nazarii Shportun

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nazarii Shportun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the U.S. immigration system as a massive, ever-changing library. Inside, there are thousands of books, rulebooks, and pamphlets written in very specific, complicated language. If you are trying to move to the U.S., you need to find the exact page that tells you which form to fill out, how much to pay, and when to send it. But the library is huge, the rules change often, and most people don't have a librarian (a lawyer) to help them find their way.

This paper is about building a smart, pocket-sized guidebook to help people navigate that library, but with a very important warning: it's a helper, not a lawyer.

Here is how the researchers built it, using simple analogies:

1. Gathering the Raw Materials (The "Library Crawl")

The researchers didn't just guess the answers. They went out and collected the actual official rulebooks.

The Sources: They grabbed 11 different types of documents, ranging from the heavy, official "Constitution-style" laws (like the USCIS Policy Manual) to community forums where people ask each other questions.
The Cleanup: They found over 10,000 documents. They cleaned them up, removed duplicates (like finding two copies of the same book), and chopped them into small, manageable "chunks" (like cutting a long novel into individual chapters).
The Result: They ended up with 18,000 little text snippets, all tagged with exactly where they came from.

2. Teaching the Robot (The "Question & Answer Factory")

Now they had a pile of text, but no questions. They needed to teach a computer how to ask and answer questions based only on what it read.

The Teacher: They used a very smart AI (Claude Sonnet) to act as a teacher. They gave the smart AI a chunk of text and said, "Read this and make up a question and a correct answer based only on this text."
The Safety Check: They set up a strict filter. If the smart AI made up an answer that wasn't actually in the text (a "hallucination"), they threw that pair away. They did this 22 times to ensure the data was honest.
The Dataset: This process created a massive flashcard deck of 17,058 questions and answers covering 13 different areas of immigration (like family visas, asylum, or travel documents).

3. Training the "Pocket Guide" (The "Small Student")

They didn't try to train a giant, expensive super-computer. Instead, they trained a small, lightweight model (Llama 3.2 3B). Think of this as training a smart high school student rather than a PhD professor.

The Method: They used a technique called LoRA. Imagine this as giving the student a set of "sticky notes" with the new immigration rules written on them, rather than forcing them to rewrite their entire brain. This is cheap and fast.
The Cost: The whole process cost about $29 in cloud computing fees. That's less than a dinner for two.

4. The Test Drive (The "Report Card")

They put the trained "pocket guide" to the test against a hidden set of 101 questions.

The Competitors:
1. The Untrained Student: The same small model before it learned anything (Score: 0.85/3).
2. The Big Professor: A much larger, untrained model (Llama 3 8B) (Score: 0.85/3).
3. The Expert: A top-tier AI (Claude Sonnet) that didn't study the specific cards but just used its general smarts (Score: 1.52/3).
4. Our Trained Student: The small model after studying the 17,000 flashcards (Score: 1.08/3).

The Results:

The trained student did 27% better than the untrained version of itself.
It got 16.8% of the answers "perfectly correct," compared to only 4% for the untrained version.
Where it shined: It got really good at "procedural" questions, like "What form do I use for a travel document?" or "How do I adjust my status?" These are like following a recipe.
Where it struggled: It was still weak on complex legal reasoning (like "Why did this specific court case rule this way?") or questions about very recent statistics. It also sometimes got confused on "humanitarian" or "asylum" cases, which require more nuance than a simple recipe.

The Big Warnings (The "Fine Print")

The authors are very clear about what this tool is NOT:

It is not a lawyer. It cannot give legal advice for your specific situation.
It is not up-to-the-minute. The library of rules they used was "frozen" at a specific date. If the government changes a fee or a form number tomorrow, this guide won't know.
It can make mistakes. In the test, it sometimes guessed a date or a number that sounded right but was wrong (like confusing the date a war ended with the date a specific law changed).

The Bottom Line

The researchers built a publicly available, low-cost toolkit that shows you can teach a small, cheap computer to understand complex immigration rules if you give it the right "flashcards" from official sources. It's a great step forward for making legal information more accessible, but it's a study aid, not a replacement for a qualified attorney.

All the code, the data, and the model are free for anyone to download and use, just like a public library book.

Technical Summary: ImmigrationQA and Small-Model Adaptation for U.S. Immigration Law

Problem Statement
U.S. immigration law constitutes a procedurally dense domain with high practical stakes for petitioners, particularly those lacking legal representation. The relevant information is scattered across thousands of pages of official policy (USCIS Policy Manual), federal regulations (8 CFR), case law (BIA precedents), and procedural guidance that change frequently. General-purpose Large Language Models (LLMs) struggle in this domain, frequently confusing form numbers, misstating deadlines, and conflating requirements across visa categories. The core difficulty identified is data scarcity: while official guidance is publicly available, it is sparse in the pretraining corpora of most LLMs relative to its procedural complexity, leading to hallucinations and factual errors.

Methodology
The paper addresses this gap through a two-part methodology: the construction of a source-grounded dataset and the parameter-efficient fine-tuning of a small language model.

Corpus Construction (ImmigrationQA):
- Sources: The dataset was assembled from 11 primary and secondary sources, including the USCIS Policy Manual, 8 CFR, BIA precedent decisions, and community forums (labeled as non-authoritative). This yielded 10,056 validated canonical documents and 18,308 text chunks.
- Generation: Structured Question-Answer (QA) pairs were generated from these chunks using Claude Sonnet 4.6 via Amazon Bedrock. Five generation modes were defined based on source type (e.g., faq, rule, form, precedent, statistics), each with distinct prompts.
- Validation: A rigorous filtering process rejected pairs where the answer had no lexical overlap with the source span (to mitigate hallucination) or was too short. This resulted in a final dataset of 17,058 QA pairs across 13 immigration subdomains. Each pair is annotated with metadata including answer type, subtopic, authority level, and a time-sensitivity flag.
Model Adaptation:
- Architecture: A Llama 3.2 3B Instruct model was fine-tuned using Low-Rank Adaptation (LoRA).
- Configuration: The training utilized a rank ( $r$ ) of 32 and alpha of 64, targeting all four attention projection matrices. Training was conducted on an AWS SageMaker ml.g5.2xlarge instance for 2 epochs with a learning rate of 5e-5.
- Cost Efficiency: The entire pipeline, including data generation and model training, cost approximately $29 in cloud compute.
Evaluation:
- Setup: The fine-tuned model was evaluated against a held-out split of 993 pairs (using a 101-example stratified sample for detailed scoring).
- Baselines: Performance was compared against the Llama 3 8B base model (zero-shot) and a zero-shot Claude Sonnet 4.6 baseline.
- Metric: An "LLM-as-judge" paradigm was employed, where Claude Sonnet 4.6 scored predictions on a 0–3 rubric based on factual correctness and completeness.

Key Results

Performance Gains: The fine-tuned 3B model achieved a mean score of 1.08/3.0, a 27% relative improvement over the Llama 3 8B base model (0.85/3.0). The fully correct rate (score of 3) increased from 4% to 16.8%.
Subdomain Variance: Improvements were concentrated in procedural subdomains such as travel documents (+1.00 mean score), adjustment of status, and nonimmigrant visas. In these areas, the fine-tuned model significantly outperformed the base model, which often scored near zero.
Limitations: The fine-tuned model performed worse than the 8B base model on complex subdomains requiring multi-step legal reasoning or specific regulatory citations, such as removal, humanitarian, admissibility, and asylum. It also struggled with time-sensitive statistics and numerical facts, often substituting plausible but incorrect values.
Baseline Comparison: While the fine-tuned model improved upon its base family, it did not surpass the zero-shot Claude Sonnet 4.6 baseline (1.52/3.0), reflecting the gap in parameter count and pretraining.

Significance and Claims
The paper claims its primary contribution is the ImmigrationQA dataset and the reproducible pipeline for constructing domain-specific, source-grounded QA data from official government sources at a commodity cloud cost. The work demonstrates that:

A high-quality, auditable corpus of over 10,000 immigration documents can be assembled and converted into structured training data for under $30.
Fine-tuning a small (3B parameter) model on this data yields significant improvements in procedural accuracy compared to larger, un-tuned base models, specifically for discrete, rule-based queries.
The approach provides a foundation for small-model legal adaptation, though the authors explicitly state the resulting model is not a substitute for legal counsel.

The authors emphasize that the system is limited by its static corpus (frozen at the crawl date) and its parametric nature (lacking retrieval augmentation), making it unsuitable for time-sensitive legal advice or complex, multi-step legal reasoning. The dataset, model, code, and prompts are publicly released to support further research in this domain.