💬 NLP

From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text

This paper presents a dual-aspect evaluation framework combining quantitative benchmarks and qualitative error analysis to reveal that while large language models can simplify Vietnamese legal texts, their primary limitation lies in controlled, accurate legal reasoning rather than summarization, with specific models exhibiting distinct trade-offs between readability and factual precision.

Original authors: Van-Truong Le

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Van-Truong Le

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, dusty library filled with ancient, complicated rulebooks written in a secret code that only lawyers understand. This is the Vietnamese legal system. For the average person, trying to read these books is like trying to solve a puzzle while wearing blindfolded goggles.

This paper is about a new experiment where the researchers asked four super-smart AI robots (called Large Language Models) to act as legal translators. Their job was to take these scary, complex laws and rewrite them in simple, friendly language that a regular person could understand, complete with a real-life example.

Here is the story of what happened, explained simply:

1. The Setup: The "Taste Test"

The researchers didn't just ask the robots to write; they put them through a rigorous taste test. They picked 60 tricky laws (covering crimes, family matters, and land rights) and asked four top-tier AI models (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1) to translate them.

They measured the results in two ways:

The Scorecard (Quantitative): How good did the translation look? Was it easy to read? Did it stay consistent?
The Autopsy (Qualitative): They didn't just look at the final grade; they opened up the robot's brain to see exactly where it went wrong. They created a "menu of mistakes" to categorize every error.

2. The Characters: Four Robots with Different Personalities

The study revealed that these AI models aren't all the same. They have distinct "personalities" that make them good at some things and terrible at others.

Grok-1 (The Cautious Scribe):
- Personality: This robot is incredibly careful. It sticks very closely to the original text.
- Strengths: It wrote the clearest, most readable summaries and rarely changed its mind (consistent).
- Weakness: It was a bit boring and sometimes made up weird examples because it was afraid to take risks. It was like a student who copies the textbook perfectly but can't answer a "what if" question.
Claude 3 Opus (The Ambitious Lawyer):
- Personality: This robot tries to be the smartest in the room. It wants to explain everything deeply.
- Strengths: It got the core legal rules right more often than anyone else.
- Weakness: Because it tried so hard to be clever, it sometimes "overthought" things and twisted the meaning of complex words. It's like a lawyer who gives a brilliant speech but accidentally cites a law that doesn't exist.
GPT-4o (The Over-Simplifying Teacher):
- Personality: This robot wants to make things super easy for you.
- Strengths: It speaks very simply.
- Weakness: It was so eager to simplify that it cut out the important details. It's like a teacher who explains a complex math problem by saying "just do the math," but forgets to tell you which math to do. This is dangerous in law because the details are everything.
Gemini 1.5 Pro (The Mood-Swing Artist):
- Personality: This robot is unpredictable.
- Strengths: Sometimes it was perfect.
- Weakness: Other times, it contradicted itself or gave examples that had nothing to do with the law. You never knew which version of Gemini you were going to get.

3. The Big Discovery: The "Illusion of Competence"

The most important finding of the paper is a warning sign.

The researchers found that high scores can be misleading.

The Trap: A robot might get a high score for "Readability" because it sounds smooth and confident.
The Reality: Underneath that smooth voice, it might be making a critical logic error. For example, it might explain a law correctly but then give a fake example that breaks the law.

It's like a magician who performs a perfect trick (the readability) but actually stole your watch the whole time (the legal error). The audience is so impressed by the magic that they don't notice the theft.

4. The Verdict: We Need Human Supervision

The paper concludes that while these AI robots are amazing at rewriting words, they are still struggling with thinking like a lawyer.

They are great at summarizing.
They are bad at applying rules to new, tricky situations (like giving the right example).

The Takeaway:
We cannot just let these robots loose to explain the law to the public yet. If we do, people might get the "smooth" version of the law but miss the "critical" details that could cost them their rights or freedom.

The solution isn't to stop using AI, but to use it as a drafting assistant that is always double-checked by a real human lawyer. Think of the AI as a very fast, very confident intern, and the human lawyer as the boss who has to sign off on the final work.

Summary Analogy

Imagine you are building a house.

The Law is the blueprint.
The AI is a construction crew.
The Study found that some crews build walls that look beautiful (Readability) but are made of cardboard (Accuracy). Others build strong walls but forget to put in the windows (Oversimplification).

The paper says: "Don't just look at how pretty the house looks. You need a human inspector to check if the foundation is actually safe before anyone moves in."

1. Problem Statement

The complexity of Vietnamese legal texts ("legalese") creates a significant barrier to public access to justice. While Large Language Models (LLMs) offer a promising solution for simplifying these texts, current evaluation methods are insufficient.

The Gap: Existing evaluations rely heavily on surface-level metrics (e.g., general accuracy scores) that fail to explain why models fail. A model might achieve a high accuracy score by correctly summarizing general rules while missing critical exceptions or committing subtle reasoning errors.
The Challenge: There is a lack of holistic frameworks that combine large-scale quantitative benchmarking with deep qualitative diagnostics specifically for legal reasoning in non-English, civil law systems like Vietnam.

2. Methodology

The authors propose a Dual-Aspect Evaluation Framework applied to a curated dataset of 60 complex Vietnamese legal articles (20 from the Penal Code, 20 from the Civil Code, and 20 from the Land Law).

A. Experimental Setup

Models Evaluated: Four state-of-the-art LLMs: GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1.
Task: Zero-shot prompt requiring the model to act as a legal assistant, explain an article in simple terms for a layperson, and provide a practical example.
Data Volume: 480 total outputs (4 models × 60 articles × 2 runs per article).

B. Phase 1: Quantitative Performance Benchmarking

Three user-centric metrics were evaluated:

Legal Accuracy (Weighted by "Safety First"): Rated by 5 law students.
- Sub-criteria: Content Preservation (40%), Completeness (30%), Clarity (20%), Example Relevance (10%).
Readability: Rated by 253 non-expert participants.
- Sub-criteria: Ease of Language (40%), Structural Coherence (30%), Utility of Examples (30%).
Consistency: Measured stability across two independent runs.
- Sub-criteria: Semantic similarity (30%), Linguistic overlap (30%), Length stability (10%), and Manual consistency rating (30%).

C. Phase 2: Qualitative Error Analysis

A novel, expert-validated 9-category Error Typology was applied to the outputs to identify root causes of failure:

Omissions: Core Elements (1.1), Exceptions (1.2).
Hallucinations: Fabricated info (1.3).
Reasoning Flaws: Misinterpretation (1.4), Oversimplification (2.1), Cross-reference failure (2.2), Internal Contradiction (2.3).
Example Failures: Irrelevant Example (3.1), Incorrect Example (3.2).
Annotation Process: Two-stage annotation by law students with high inter-rater agreement ( $\kappa = 0.72$ ).

3. Key Contributions

Dual-Aspect Framework: Introduces a methodology that marries large-scale quantitative benchmarking with granular qualitative error analysis, moving beyond simple leaderboards.
Novel Error Typology: Develops a specific taxonomy for legal simplification errors, distinguishing between factual hallucinations and reasoning failures (e.g., oversimplification, misinterpretation of legal principles).
Large-Scale Vietnamese Legal Dataset: Provides a rigorous evaluation on 60 complex articles across three major legal domains (Criminal, Civil, Land), addressing the scarcity of benchmarks for Vietnamese civil law.
Diagnostic Insights: Reveals specific "error profiles" for top-tier models, demonstrating that high accuracy scores can mask critical reasoning fragility.

4. Key Results

A. Performance Trade-offs

Grok-1: Emerged as the top performer in Readability (4.93) and Consistency (4.28). It had the lowest total error count (89) and was the "safest" model, avoiding omissions and misinterpretations. However, it struggled with generating accurate examples.
Claude 3 Opus: Achieved the highest Legal Accuracy (4.71) but had the highest total error count (196). Its high score masked a critical weakness: a high frequency of Misinterpretation (41 errors) and Incorrect Examples (39 errors). It attempts sophisticated reasoning but often overreaches.
GPT-4o: Suffered primarily from Oversimplification (48 errors). It stripped away necessary legal nuances to achieve readability, leading to potentially misleading legal maxims.
Gemini 1.5 Pro: Showed high variance, producing both perfect responses and significant logical contradictions.

B. Critical Findings

The "Accuracy" Illusion: High aggregate accuracy scores (like Claude's) can be misleading if they are driven by correct handling of simple rules while failing on complex exceptions.
Readability vs. Fidelity: There is a direct trade-off. Models that prioritize faithful paraphrasing (Grok) are more consistent but less capable of novel reasoning. Models that prioritize "helpfulness" (Claude) or "conciseness" (GPT-4o) introduce reasoning errors.
Primary Failure Modes: The most prevalent failures were Incorrect Examples and Misinterpretation, indicating that current LLMs are better at rephrasing text than applying legal principles to new scenarios.

5. Significance and Implications

Architectural Insights: The study suggests that performance differences are driven by alignment strategies. Grok-1's "cautious" behavior may stem from looser safety constraints (less "alignment tax"), while GPT-4o's oversimplification may result from aggressive safety tuning that strips nuance.
Risk-Aware Deployment: The findings argue that deploying LLMs for public legal aid without human oversight is premature. The proposed error typology offers a blueprint for risk-aware human-in-the-loop systems, where specific error types (e.g., example generation) trigger mandatory professional review.
Future Research: Highlights the need for better reasoning capabilities in LLMs, the potential of Few-Shot/Chain-of-Thought prompting to mitigate errors, and the importance of evaluating open-weight models for reproducibility.

Conclusion: The paper concludes that while LLMs show promise for legal text simplification, they currently lack the robust, controlled reasoning required for high-stakes legal applications. A dual-aspect evaluation is essential to uncover the "illusion of competence" and guide the development of trustworthy legal AI.