💬 NLP

A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays

This study reveals that while LLM judges and human examiners converge on Thai bar-exam essays with clear rubrics, LLMs systematically fail to reproduce a minority human interpretation on ambiguous cases, instead uniformly aligning with the majority human view and thereby masking their inability to capture the full spectrum of expert disagreement.

Original authors: Pawitsapak Akarajaradwong, Wuttikrai Lertprasertphakorn, Chompakorn Chaksangchaichot, Sarana Nutanong

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pawitsapak Akarajaradwong, Wuttikrai Lertprasertphakorn, Chompakorn Chaksangchaichot, Sarana Nutanong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a high-stakes contest to judge the quality of legal essays. You have a very specific rulebook (the "rubric") that tells graders how to score answers. Usually, the rulebook is clear: if the answer is right, give a high score; if the reasoning is bad, give a low score.

But sometimes, the rulebook hits a "gray zone." It doesn't say exactly what to do when a student gets the final answer right but forgets to mention one specific, crucial law citation. This is the "regulation-silent" zone.

This paper is a two-part experiment to see how humans and Artificial Intelligence (AI) judges handle these gray zones.

Part 1: The "Exam" (Phase 1)

First, the researchers put 8 different AI models into the "student seat." They had to write legal essays just like 42 real Thai law students. The AI models were graded blindly by human experts.

The Result: The AI models performed about as well as the average human student. Some were great, some were average, and some struggled. They were all "in the game."

Part 2: The "Grading" (Phase 2)

Next, the researchers took the same essays and asked two groups to grade them:

The Human Panel: Three real, certified legal examiners.
The AI Panel: A massive group of 26 different AI models (including the ones that took the exam earlier, plus many others).

They gave everyone the exact same four things to look at: the question, the rulebook, the "perfect" answer key, and the student's essay.

The Big Discovery: The "One-Way Street"

The researchers found a fascinating split, but only on the tricky questions where the rulebook was silent.

The Human Split:
The three human experts didn't agree.

Two of them (B and C) said: "The student got the main point right and the reasoning was good enough. Give them a high score (6–8)."
One of them (A) said: "The student missed a crucial citation. That makes the reasoning 'unusable.' Give them a very low score (1–2)."
Think of it like a sports referee: Two refs say the player was "in bounds," and one ref says they were "out of bounds." Both are using the same rulebook, but they interpret the gray area differently.

The AI Split (The Asymmetry):
Here is where it gets weird. The researchers expected the 26 AI models to split up, maybe some siding with the strict human (A) and some with the lenient humans (B and C).

They didn't.
22 out of 26 AI models sided with the two lenient humans (B and C). They gave high scores.
3 AI models were confused and gave middle scores.
Zero AI models sided with the strict human (A). Even the one AI that tried to be strict (GPT-5.4 Nano) wasn't consistent enough to truly match the strict human's style.

The Metaphor:
Imagine a group of 26 different cameras taking a photo of a painting.

The three human art critics look at the painting and disagree: Two say it's a masterpiece, one says it's a failure.
You ask the 26 cameras to describe the painting.
The result: All 26 cameras agree with the two critics who called it a masterpiece. None of the cameras agree with the critic who called it a failure. Even though the cameras are different brands, sizes, and prices, they all "see" the painting the same way, missing the perspective of the one strict critic entirely.

Why Does This Matter?

The paper argues that when we build AI systems to grade essays, we usually pick the AI that agrees the most with the "average" human grader.

Because the majority of humans in this study (2 out of 3) were lenient, the AI learned to be lenient too.
The AI didn't learn to be "fair" in a balanced way; it learned to be asymmetric. It converged on the majority view and completely ignored the minority view, even though that minority view was a valid, legal interpretation.

The "Dual-Role" Surprise

The researchers also noticed something strange about the AI's personality.

The AI models that were bad students in Phase 1 (they got low scores on their own essays) actually became the most consistent lenient judges in Phase 2.
The AI models that were great students in Phase 1 became only "okay" judges.
The Lesson: Being good at writing the answer doesn't make you good at grading the answer. The skills are totally different.

The Bottom Line

The study shows that AI judges are very stable and consistent with each other, but they have a blind spot. When human experts disagree on a gray-area rule, AI doesn't split the difference or explore all valid opinions. Instead, it overwhelmingly picks the "majority" human opinion and ignores the "minority" one.

If you use AI to grade essays, you aren't just getting a second opinion; you are getting a mirror that reflects the majority view so strongly that it makes the minority view disappear completely.

Technical Summary: A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays

Problem Statement

Current Natural Language Processing (NLP) evaluation for free-form legal essays relies on two untested assumptions: (1) that inter-rater agreement among credentialed human experts defines a single, flat "stability ceiling" across all items, and (2) that an LLM judge's success is measured by how closely its agreement with this human panel approaches that ceiling. This study challenges these assumptions by investigating whether expert stability is uniform across items and whether LLM judge populations reproduce the full spectrum of legitimate human interpretations or merely converge on a single majority reading. The study focuses on the Thai Bar Examination, a high-stakes, free-form essay assessment where grading involves both deterministic correctness (matching a gold answer) and interpretive quality (reasoning depth), the latter of which often lacks explicit regulatory guidance.

Methodology

The research employs a two-phase, identical-inputs protocol to isolate the roles of examinee and judge while controlling for input variance.

Phase 1: LLMs as Examinees

Participants: 42 Thai law students and 8 LLMs (including Claude 4.6 Opus, Gemini 3.1 Pro, GPT-5.4 Pro, and others).
Task: Participants answered three Civil and Commercial Code essay questions under blinded conditions.
Grading: Three Bar Council-trained examiners (A, B, C) graded the answers using a published 8-tier rubric. The examiners were unaware of which participants were LLMs.
Outcome: Established a baseline capability distribution where LLMs sat within the human student distribution, ensuring the grading task was not trivial.

Phase 2: LLMs as Judges

Corpus: A subset of 15 cross-graded cells (5 answers $\times$ 3 examiners) was selected for deep analysis.
Inputs: All raters (3 humans and 26 LLMs) received the exact same four-tuple inputs: the question, the official grading regulation, the gold answer, and the candidate's essay.
LLM Panel:
- Anchor Sub-panel: Three closed-frontier models (Claude 4.6 Opus, Gemini 3.1 Pro, GPT-5.4 Pro) underwent rigorous instrumentation, including determinism probes (3 seeds at $T=0$ ), input ablations (regulation-withheld, gold-withheld), and bootstrap confidence intervals.
- Extended Panel: 23 additional models spanning closed-frontier (large/small), flagship open-weight, and small open-weight tiers were added to test population-level structural claims.
Metrics: Stability was measured using Krippendorff's $\alpha$ , pairwise Pearson correlations, and Cohen's weighted $\kappa$ .

Key Contributions

Population-Level Asymmetric Convergence: The study identifies a structural asymmetry in LLM judge behavior. On items where the rubric is silent regarding the interpretation of a correct final answer with omitted statutory citations (the "regulation-silent" boundary), the human panel splits into two coherent readings (a majority B/C reading and a minority A reading). The LLM population does not split symmetrically; 22 of 26 LLMs converge on the majority (B/C) reading, 3 sit in a middle gap, and only 1 (GPT-5.4 Nano) approaches the minority (A) reading without consistently scoring within it. Zero LLMs reproduce the minority reading.
Decoupling Examinee Capability from Judge Stability: Using the dual-role design (same 8 LLMs as examinees and judges), the paper demonstrates that a model's performance as an examinee does not predict its alignment as a judge. Weaker examinees often become stronger B/C-aligned judges, while top-scoring examinees do not necessarily align with the minority reading. Scaling capability does not push the LLM population toward the minority reading.
Instrumented Anchor Panel: The establishment of a three-LLM anchor sub-panel with full instrumentation (determinism, ablations, bootstrap CIs) achieving a panel $\alpha$ of 0.77, significantly higher than the human panel's 0.36 on the contested items.

Key Results

Stability Metrics

Non-Contested Items: On 10 of 15 cells (Questions 1 and 3), where the rubric prescribes both axes, human and LLM panels are stability-equivalent ( $\alpha \approx 0.75$ ).
Contested Items: On the 5 cells of Question 2 (regulation-silent boundary), the human panel collapses to $\alpha = -0.34$ due to the split between Examiner A (stricter, score 1–2) and Examiners B/C (lenient, score 6–8).
LLM Behavior on Contested Items: The LLM panel maintains high stability ( $\alpha = 0.68$ $α = 0.68$ ) but converges exclusively on the B/C reading.
- 22/26 LLMs: Score in the B/C band ( $m_c \in [5, 8]$ ).
- 3/26 LLMs: Score in the middle gap ( $3 < m_c < 5$ ).
- 1/26 LLMs (GPT-5.4 Nano): Approaches the A band ( $m_c = 2.33$ ) but fails to consistently score within it.
- Result: No LLM reproduces the minority human reading.

Failure Mode Diagnostics

The study identifies three distinct failure modes in LLMs that deviate from the majority cluster:

Scale Quantization (Claude Haiku 4.5): Collapses the scoring scale to a binary pass/fail bucket (3.5 or 7.5), failing to express mid-range scores.
Final-Answer-Only Grading (GPT-5.4 Mini): Ignores the interpretive axis, assigning high scores solely based on the correctness of the final conclusion, regardless of reasoning quality.
Reasoning-Strict Grading (GPT-5.4 Nano): Strictly penalizes omitted citations on the interpretive axis, aligning with the minority A reading, but lacks consistency across the full dataset.

Robustness Checks

Determinism: At $T=0$ , the anchor LLMs show high intra-judge stability (Claude $\alpha=0.998$ ), confirming that the inter-model diversity (0.77) is a feature of the population, not an artifact of noise.
Input Ablations: Removing the regulation or gold answer reduces LLM panel stability but does not invert the direction of the gap (LLM $\alpha$ remains higher than human $\alpha$ ).
Dual-Role Analysis: The correlation between examinee score and judge cluster alignment is exploratory and negative ( $r = -0.34$ ), suggesting that weaker examinees (who may rely more on the deterministic axis) align more strongly with the B/C majority as judges.

Significance and Claims

The paper claims that the "stability ceiling" in legal NLP is not a flat line but a fault line located at regulation-silent boundaries where the rubric leaves interpretive quality to expert judgment.

Asymmetry of Agreement: High LLM-panel agreement ( $\alpha$ ) does not imply balanced reproduction of human variance. Instead, it reflects systematic convergence on the majority human reading. A benchmark that selects an LLM judge by maximizing agreement with a human reference panel (which is typically skewed by the majority) will, by construction, inherit this asymmetry and filter out judges capable of the minority reading.
The Interpretive Bottleneck: The study defines the "rubric's interpretive bottleneck" as the specific zone where the regulation is silent on how to grade a correct answer with incomplete reasoning. In this zone, human panels naturally split into coherent, legitimate readings, while LLM populations converge on one side.
Limitations of Scaling: The findings suggest that simply scaling model capability does not resolve this asymmetry; larger models do not spontaneously adopt the minority reading. The convergence is a structural property of the LLM ensemble on this specific rubric, independent of model size, vendor, or price tier.

The paper concludes that current evaluation protocols obscure the full space of legitimate human readings. It calls for benchmarks that report panel stability per interpretive case and evaluate LLM judges against multiple legitimate human readings rather than a single averaged target.