Can Small Models Reason About Legal Documents? A Comparative Study

This study demonstrates that sub-10B parameter models, particularly a 3B Mixture-of-Experts architecture, can match or surpass frontier models like GPT-4o-mini on legal benchmarks when paired with appropriate prompting strategies, revealing that model architecture and training quality are more critical than raw parameter count while highlighting the task-dependent efficacy of different prompting methods and the sufficiency of dense retrieval for RAG.

The Gist

Imagine you are a law firm trying to build a team of digital assistants to help analyze contracts, find legal precedents, and predict court outcomes. You have two main options:

1. The "Super-Genius" Consultant: A massive, expensive, top-tier AI (like GPT-4o-mini) that costs a fortune to hire per hour, requires a secure cloud connection, and might eat your confidential client data.
2. The "Local Interns": Smaller, cheaper AI models that you can run on your own computers. They are faster and cheaper, but you worry they might not be smart enough to handle complex legal logic.

This paper is essentially a talent show where the researchers put nine different "Local Interns" (small AI models) up against the "Super-Genius" to see who can actually do the legal work.

Here is the breakdown of their findings, using some everyday analogies:

1. The "Small but Mighty" Surprise

The Finding: A specific small model called Qwen3-A3B (which only "thinks" with 3 billion parameters at a time, even though it has 30 billion total) performed just as well as the expensive Super-Genius.
The Analogy: Imagine a sprinter (the small model) running a race against a marathon runner (the large model). You'd expect the marathon runner to win because they have more stamina (parameters). But the sprinter won because they are built differently. The small model uses a "Mixture-of-Experts" architecture, which is like having a team of specialists in a room where only the right expert wakes up to answer a specific question. This made it incredibly efficient.
The Result: The small model matched the big one in accuracy and even beat it on a specific task: identifying the "holding" (the main legal rule) of a court case.

2. Bigger Isn't Always Better

The Finding: The largest model they tested, a 9-billion-parameter model called Nemotron, actually performed the worst of all.
The Analogy: It's like hiring a giant, bloated bureaucracy to solve a simple problem. The Nemotron model was so heavy and poorly trained that it got confused easily. Meanwhile, a tiny 3-billion model (Qwen3-A3B) was nimble and sharp.
The Lesson: It's not about how big the brain is; it's about how well it was trained and how its internal gears are arranged.

3. The "How You Ask" Matters More Than "Who You Ask"

The researchers tested five different ways of talking to the AI (Prompting Strategies). The results were like trying different keys on a lock:

• Chain-of-Thought (CoT): This asks the AI to "think step-by-step" out loud.
  • The Twist: It worked wonders for Contract NLI (checking if a contract clause makes sense), boosting scores significantly. But for CaseHOLD (picking the right answer from 5 choices), it was a disaster.
  • The Analogy: Asking a lawyer to "write a 5-page essay explaining their reasoning" before giving a simple "Yes/No" answer. For a contract review, the essay helps. For a multiple-choice quiz, the essay distracts the lawyer, and they forget to circle the right letter.
• Few-Shot Prompting: This is like giving the AI three examples of a solved problem before asking it to solve a new one.
  • The Winner: This was the most consistent strategy. It worked well across almost every task and model. It's like showing an intern a few past cases before asking them to draft a new one.
• Direct Prompting: Just asking the question. This was the baseline, but often lacked the "nudge" needed for complex tasks.

4. The "Search Engine" Myth

The researchers tested two ways to help the AI find information (Retrieval Augmented Generation or RAG):

1. BM25 (Sparse): Like a classic library card catalog (matching exact keywords).
2. Dense Retrieval: Like a modern Google search (understanding the meaning of words).

The Finding: There was almost no difference between the two.
The Analogy: It didn't matter if you gave the AI a keyword index or a semantic search engine. The bottleneck wasn't finding the right book in the library; it was the AI's ability to read and understand the book once it found it. The AI was the weak link, not the librarian.

5. The Cost of the Experiment

The Finding: They ran 405 different experiments (testing 9 models, 5 strategies, 3 different legal tests, and 3 random variations).
The Analogy: Usually, running this many tests would require a warehouse full of expensive graphics cards (GPUs) costing thousands of dollars. Instead, they used cloud APIs and spent a total of $62.
The Takeaway: You don't need a supercomputer to do serious scientific research on AI anymore. You just need a credit card and a good experimental design.

Summary: What Should You Do?

If you are a legal professional or a developer looking to build legal AI tools, the paper suggests:

1. Don't just buy the biggest model. Look for efficient, well-trained small models (like the Qwen3-A3B). They are cheaper, faster, and just as smart for legal tasks.
2. Don't force "Step-by-Step" thinking on everything. If you need a multiple-choice answer, just ask for the answer. If you need a contract analysis, ask for the reasoning.
3. Use "Examples" (Few-Shot). Show the AI what you want before asking it to do it.
4. Simple Search is Fine. You don't need complex AI search engines to find legal text; a good keyword search is often enough.

The Bottom Line: Small, smart models can do the heavy lifting in legal tech without breaking the bank or compromising privacy, provided you know how to talk to them correctly.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) have demonstrated success in legal tasks, deploying frontier models (e.g., GPT-4) raises significant practical barriers:

• Cost: API costs scale linearly with query volume.
• Privacy: Proprietary models pose data privacy risks for sensitive legal documents.
• Latency: Production systems often require smaller, locally deployable models.

Despite the emergence of sub-10B parameter models (including Mixture-of-Experts architectures), it remains unclear if they can match the reasoning capabilities of commercial frontier models on specialized legal tasks. Furthermore, the interaction between model architecture, size, prompting strategies, and retrieval methods in the legal domain is not well understood.

2. Methodology

The authors conducted a rigorous comparative study involving 405 experiments across three legal benchmarks, nine models, and five prompting strategies.

Models Evaluated

• Open-Weight (7 models): Spanning 3B to 9B parameters, including dense models (Llama 3.2/3.1, Gemma, Mistral, Nemotron) and a Mixture-of-Experts (MoE) model (Qwen3-A3B, activating only 3B of 30B parameters).
• Commercial APIs (2 models): GPT-4o-mini and Claude 3.5 Haiku, serving as baselines.
• Constraint: All models were accessed via cloud APIs with a uniform 8,192-token context window to ensure fair comparison without dedicated GPU infrastructure.

Datasets (Legal Benchmarks)

1. CONTRACTNLI: Natural Language Inference on contract clauses (Entailment, Contradiction, Not Mentioned).
2. CASEHOLD: Legal holding identification via 5-way multiple-choice selection from court opinions.
3. ECTHR (Task A): Multi-label classification predicting violated articles of the European Convention on Human Rights (11 possible labels).

Prompting Strategies

1. Direct (Zero-shot): Standard instruction.
2. Chain-of-Thought (CoT): Explicit step-by-step reasoning scaffold.
3. Few-shot (3-shot): In-context learning with three labeled examples.
4. BM25 RAG: Retrieval-Augmented Generation using sparse (BM25) retrieval of top-3 passages.
5. Dense RAG: RAG using dense retrieval (sentence-transformers).

Experimental Design

• Scale: 9 models × 5 strategies × 3 datasets × 3 random seeds.
• Evaluation: Stratified sampling (500 samples for ContractNLI/CASEHOLD; 225 for ECTHR).
• Metrics: Accuracy (exact match), Macro F1, and parse error rates.
• Cost: Total experiment cost was $62, demonstrating accessibility.

3. Key Contributions & Findings

A. Architecture Efficiency > Parameter Count

• MoE Superiority: Qwen3-A3B (3B active parameters) matched GPT-4o-mini in mean accuracy (46.5% vs. 47.2%) and surpassed it on the CASEHOLD benchmark (71.2% vs. 67.9%) using few-shot prompting.
• Size Paradox: The largest model tested, Nemotron-9B (9B parameters), performed the worst overall (17.7% mean accuracy), underperforming models one-third its size. This indicates that training quality and architecture (MoE) matter more than raw parameter count for legal reasoning.

B. Task-Dependent Prompting Effects

• Chain-of-Thought (CoT) is Not Universal:
  • Beneficial: Improved performance on CONTRACTNLI (+8.5 percentage points) where logical entailment is required.
  • Detrimental: Severely degraded performance on CASEHOLD (-16.0 pp) and ECTHR. The verbose reasoning process disrupted the strict format required for multiple-choice selection and multi-label classification.
  • Exception: Only Claude 3.5 Haiku maintained format fidelity during CoT generation on CASEHOLD.
• Few-Shot is Most Robust: Few-shot prompting emerged as the most consistently effective strategy across all tasks and models, avoiding the sharp penalties seen with CoT on multiple-choice tasks.

C. Retrieval Augmentation (RAG) Insights

• Method Irrelevance: There was no significant difference between BM25 (sparse) and Dense retrieval (0.3 pp mean difference).
• Task Dependency: RAG helped CONTRACTNLI but hurt CASEHOLD regardless of the retrieval method. This suggests the bottleneck lies in the LLM's ability to utilize retrieved context rather than the quality of the retrieval itself.

D. Parse Errors

• Instruction-following capability varied wildly. Nemotron-9B had a high parse error rate (16.4% average, up to 29.8% on CONTRACTNLI), contributing significantly to its low accuracy. In contrast, GPT-4o-mini and Qwen2.5-7B achieved near-zero parse errors.

4. Significance & Practical Implications

• Cost-Effective Legal AI: The study proves that open-weight small models (specifically MoE architectures like Qwen3-A3B) can serve as practical, cost-efficient alternatives to expensive commercial APIs for legal reasoning.
• Strategic Deployment:
  • Model Selection: Prioritize well-trained 3–8B models (especially MoE) over larger, poorly calibrated dense models.
  • Prompting: Use Few-shot as the default strategy. Reserve CoT only for entailment tasks (like contract analysis) and avoid it for multiple-choice or multi-label classification.
  • RAG: Simple BM25 retrieval is sufficient; the complexity of dense embeddings may not be necessary for legal passage retrieval.
• Accessibility: The entire evaluation was conducted for $62 via cloud APIs, establishing a blueprint for rigorous LLM evaluation without dedicated GPU infrastructure.

5. Limitations

• Prompting Only: The study did not include fine-tuning (e.g., LoRA), meaning results represent a lower bound for small model performance.
• Scope: Limited to English-language datasets (US and EU law) and did not cover generative tasks (summarization, drafting) or civil law systems.
• Data Contamination: Potential overlap between training data and public legal datasets (e.g., SEC filings) could not be ruled out for closed-weight models.

Conclusion

The paper concludes that architectural efficiency and training quality are more critical than parameter scale for legal reasoning. With the right prompting strategy (Few-shot) and model selection (MoE), sub-10B models can achieve commercial-grade performance at a fraction of the cost, making advanced legal NLP more accessible.