💬 NLP

LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification

This paper introduces LegalBench-BR, the first public benchmark for Brazilian legal decision classification using 3,105 annotated appellate proceedings, demonstrating that a LoRA-fine-tuned BERT model significantly outperforms leading commercial large language models in accuracy and domain-specific discrimination while eliminating systematic biases toward civil law.

Original authors: Pedro Barbosa de Carvalho Neto

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pedro Barbosa de Carvalho Neto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of legal documents from Brazil, but they are all mixed up in a giant, chaotic pile. Lawyers and judges need to sort these documents into five specific bins: Civil, Consumer, Tax, Administrative, and Criminal. Doing this manually is slow and expensive. So, the question is: Can we just ask a super-smart, general-purpose AI (like a high-tech librarian) to sort them for us?

This paper, LegalBench-BR, says: "Not quite. Not yet."

Here is the story of how they found out, explained simply.

1. The Problem: The "Generalist" vs. The "Specialist"

The researchers wanted to test if big, famous AI models (like GPT-4o mini and Claude 3.5) were good enough to sort Brazilian legal cases. These models are like generalist librarians: they know a little bit about everything, from cooking recipes to quantum physics.

But Brazilian law is tricky. It has its own slang, specific rules, and a very specific way of organizing things. The researchers built a new "test" (a benchmark) using 3,105 real court cases from Santa Catarina, Brazil. They made sure the test was fair by giving the AI an equal number of cases from each of the five legal categories, so the AI couldn't just guess the most common one and get a high score.

2. The Race: Who Wins?

They ran a race between three contestants:

GPT-4o mini (The expensive, famous cloud AI).
Claude 3.5 Haiku (Another expensive, famous cloud AI).
BERTimbau-LoRA (A small, custom-trained model running on a standard computer).

The Result: The custom model won by a huge margin.

The Custom Model: Got 87.6% correct.
The Famous AIs: Only got 60–65% correct.

3. The Big Surprise: The "Civil Law" Trap

The most interesting part of the story is how the famous AIs failed. They didn't just make random mistakes; they fell into a specific trap called the "Civil Default Bias."

The Analogy: Imagine a weather forecaster who is so used to seeing rain that, whenever they see a cloud, they just shout "RAIN!" without looking at the sky. Even if it's actually snowing or sunny, they insist it's rain.

What happened: When the famous AIs were unsure about a case, they defaulted to guessing "Civil Law" (the most common type of case in Brazil).
The Disaster: For Administrative Law cases (dealing with government bureaucracy), the famous AIs got 0% correct. They couldn't tell the difference between a government tax dispute and a regular civil lawsuit.
The Winner: The custom model, which had been specifically "studied" on Brazilian legal texts, learned the subtle clues (like specific court names or legal jargon) that separate these categories. It got 91% correct on Administrative Law.

4. Why the Custom Model Won (The "LoRA" Secret)

You might think the custom model is a giant super-computer. It's not. It's actually quite small and cheap.

The Analogy: Think of the big AIs as a Swiss Army Knife. It has a blade, a screwdriver, and a corkscrew. It's great for many things, but if you need to cut a specific type of wood, it's not the sharpest tool.
The Custom Model: This is like taking that Swiss Army Knife and sharpening just one specific blade (using a technique called LoRA).
The Magic: They only changed 0.3% of the model's brain. It's like teaching a generalist librarian to read only Brazilian legal codes for 30 minutes. Suddenly, they become a legal expert.
Cost: The custom model runs on a regular laptop (or even a free Google Colab GPU) and costs $0 to run. The big AIs cost money per question and take longer to answer.

5. Why This Matters for the Real World

This isn't just a science experiment; it has real consequences.

Privacy: Big AIs send your data to the cloud (the US or other countries). In Brazil, laws (LGPD) say you can't just send private legal data to a third party. The custom model keeps the data on your own computer.
Efficiency: If a law firm uses the "Civil Default" AI, they might send a government tax case to the wrong department, causing delays and confusion.
The Lesson: You can't just buy a "one-size-fits-all" AI for specialized jobs. If you want to sort legal documents in Brazil, you need a model that has actually studied Brazilian law, not just a model that knows a lot of English and Portuguese words.

Summary

The paper proves that for specialized tasks like sorting Brazilian legal cases, a small, cheap, custom-trained model beats expensive, general-purpose giants. The giants are too lazy (or biased) to look closely at the details, while the custom model, trained specifically for the job, sees the clues the others miss.

The takeaway: Don't just use the most famous tool for the job; sometimes, you need to sharpen a specific tool for the specific task.

1. Problem Statement

The intersection of Natural Language Processing (NLP) and legal text analysis has advanced significantly in English-speaking, common-law jurisdictions (e.g., via the LegalBench benchmark). However, Portuguese-language legal NLP, specifically for the Brazilian civil-law system, remains critically underserved.

Context: Brazil operates one of the world's largest court systems (80M+ active cases), yet lacks standardized benchmarks to evaluate how well language models perform on its specific legal data.
Core Question: Do general-purpose, multilingual Large Language Models (LLMs) possess sufficient domain knowledge to classify Brazilian legal proceedings without domain-specific adaptation, or are specialized models required?
Practical Need: Brazilian legal tech and the judiciary require automated classification for case routing and analytics, but current solutions lack empirical validation on balanced, representative datasets.

2. Methodology

A. Dataset Construction (LegalBench-BR)

Source: 3,678 appellate proceedings collected from the Santa Catarina State Court (TJSC) via the DataJud API (CNJ).
Annotation Pipeline: A two-stage hybrid approach was used to ensure high-quality labels:
1. LLM Classification: Claude 3.5 Haiku classified records using structured prompts. Records with low confidence (<0.7) were discarded.
2. Heuristic Correction: Rule-based overrides corrected misclassifications using high-confidence metadata (e.g., "Câmara Criminal" $\rightarrow$ Penal, "Execução Fiscal" $\rightarrow$ Tributário). This addressed the LLM's tendency to default to majority classes.
Label Set: Reduced to 5 classes based on jurisdictional reality:
1. Cível (Civil)
2. Consumidor (Consumer)
3. Tributário (Tax)
4. Administrativo (Administrative)
5. Penal (Criminal)
  Note: Labor law was excluded (different jurisdiction), and Family law was merged into Civil due to low appellate volume.
Class Balancing: The natural distribution is heavily skewed (52% Civil). To prevent models from achieving high accuracy via majority-class guessing, the dataset was rebalanced:
- Training: Oversampled minority classes to 363 examples each (1,815 total).
- Test/Validation: Undersampled to 21 examples per class (105 total).

B. Experimental Setup

Models Evaluated:
1. BERTimbau-LoRA: A domain-adapted model using BERTimbau (pre-trained on Brazilian Portuguese) with LoRA (Low-Rank Adaptation). Only 0.3% of parameters were updated. Trained on a consumer GPU (Google Colab T4) for 5 epochs.
2. Claude 3.5 Haiku: Zero-shot classification via API.
3. GPT-4o mini: Zero-shot classification via API.
Evaluation Metrics: Accuracy, Macro-averaged F1, Precision, Recall, and Inference Latency.

3. Key Contributions

First Public Benchmark: Release of LegalBench-BR, the first balanced, public benchmark for Brazilian legal text classification, available on HuggingFace.
Empirical Evidence of Domain Gap: Demonstrated that general-purpose LLMs fail significantly on specific legal domains compared to lightweight, fine-tuned models.
Identification of Systemic Bias: Uncovered a "Cível Default" bias in zero-shot LLMs, where models systematically misclassify ambiguous or minority classes (especially Administrative law) as Civil law.
Reproducible Pipeline: Provided a complete, open-source pipeline from data collection (DataJud API) to annotation, training, and evaluation, adaptable for other Brazilian courts.

4. Key Results

Overall Performance

Model	Accuracy	F1 Macro	Latency (ms)	Cost (per 1k)
BERTimbau-LoRA	87.6%	0.87	233	~$0.00
Claude 3.5 Haiku	65.7%	0.63	1,444	~$0.15
GPT-4o mini	60.0%	0.57	964	~$0.10

Performance Gap: BERTimbau-LoRA outperformed the best commercial LLM (Claude) by +22 percentage points in accuracy and +0.24 in F1 macro.
Efficiency: The fine-tuned model is ~6x faster and has zero marginal inference cost compared to API-based LLMs.

Per-Class Performance & The "Administrative" Failure

The most striking result occurred in the Administrativo (Administrative Law) class:

GPT-4o mini: F1 = 0.00 (Zero correct predictions).
Claude 3.5 Haiku: F1 = 0.08.
BERTimbau-LoRA: F1 = 0.91.
Gap: An 83-point F1 gap on a single class.

Error Analysis: The "Cível Default" Bias

Both commercial LLMs exhibited a systematic bias where, when uncertain, they defaulted to classifying cases as Cível (Civil).

Mechanism: The models appear to have internalized the statistical frequency of civil cases in the general corpus rather than learning discriminative legal features.
Consequence: In a production environment, this would lead to incorrect case routing and flawed analytics, particularly for Administrative and Tax cases.
Resolution: Domain-adapted fine-tuning on balanced data explicitly breaks this bias by exposing the model to discriminative metadata (e.g., court chamber names) and minority class examples.

5. Significance and Implications

Legal Tech Economics: For a mid-sized firm processing 10,000 cases/month, using commercial LLMs costs ~$150/month with high latency. A local fine-tuned model costs $0 in inference, runs in milliseconds, and ensures data privacy (critical for LGPD compliance and OAB ethics rules regarding client confidentiality).
Benchmarking Necessity: The paper argues that unbalanced benchmarks are misleading. On the original unbalanced dataset, Claude achieved 94.5% accuracy (driven by the 52% Civil majority). Only after balancing did the true performance gap (dropping to 65.7%) become visible.
Domain Adaptation is Essential: The results prove that general-purpose LLMs cannot substitute for domain-adapted models in specialized legal classification tasks, even for simple 5-class problems.
Annotation Caution: The study highlights that even high-end LLMs (like Claude 3.5 Haiku) can introduce systematic biases during dataset annotation, necessitating heuristic validation layers.

Conclusion

LegalBench-BR establishes that lightweight, parameter-efficient fine-tuning (LoRA) on consumer hardware is superior to zero-shot commercial LLMs for Brazilian legal classification. It offers a cost-effective, privacy-preserving, and highly accurate solution for the Brazilian legal sector, while providing a critical tool for future research in non-English legal NLP.