CALRK-Bench: Evaluating Context-Aware Legal Reasoning in Korean Law

1. Problem Statement

Current Large Language Model (LLM) benchmarks for legal reasoning primarily evaluate rule application under the assumption that legal norms are fixed and facts are static. This approach relies heavily on the IRAC (Issue–Rule–Application–Conclusion) framework, focusing on knowledge recall or the reproduction of judgment outcomes.

However, real-world legal reasoning is context-aware. It requires understanding:

• Temporal Validity: Laws change over time (amendments, repeals, constitutional invalidations). A judgment valid today may be invalid tomorrow.
• Information Sufficiency: Determining if the provided legal facts and statutes are enough to reach a conclusion, or if additional information is needed.
• Judgment Shifts: Understanding why a legal outcome changes for the same issue (e.g., due to a change in social norms vs. a change in statutory text).

Existing benchmarks fail to capture these dynamic, contextual factors, leading to an overestimation of LLM capabilities in complex legal scenarios.

2. Methodology: CALRK-Bench

The authors propose CALRK-Bench, a benchmark based on the Korean legal system (a civil law system where statutes are the primary source of law, making temporal validity clearer than in common law systems). The benchmark consists of three distinct task types designed to stress-test context-aware reasoning:

A. Task Types

1. Type-TCR (Temporal Context Recognition):

   • Goal: Determine the specific time interval in which a legal conclusion holds true given a normative change (e.g., a statute amendment or constitutional ruling).
   • Mechanism: Models are given a case summary and a description of a legal change. They must select the correct combination of "time of illegal act" and "time of trial" that aligns with the applicable norm.
   • Challenge: Requires reasoning about the interaction between the act, the trial, and the effective date of the law.

2. Type-ISR (Information Sufficiency Recognition):

   • Goal: Assess whether the provided legal provisions are sufficient to answer a consultation question.
   • Mechanism: The model chooses between "Sufficient," "Partial," or "Insufficient" information.
   • Hard Negative Mining: To prevent trivial answers, incorrect options (distractors) are generated using hard negative mining. These distractors are statutes that frequently co-occur with the correct statute in real precedents but are legally irrelevant to the specific question, creating plausible but misleading alternatives.

3. Type-JSA (Judgment Shift Attribution):

   • Goal: Identify the specific cause of a shift in legal judgment for the same issue over time.
   • Mechanism: Models must categorize the cause of change into one of four mutually exclusive types:
     • Normative Premises: Changes in social consciousness or conventions.
     • Normative Text: Statutory amendments or constitutional invalidations.
     • Interpretive Rules: Changes in judicial precedent (e.g., Supreme Court en banc decisions).
     • Factual Premises: Changes in technology or socio-economic conditions.

B. Dataset Construction & Validation

• Sources: Legal precedents (Lawnb, AI Hub, Ministry of Government Legislation) and consultation records (Korea Legal Aid Corporation).
• Scale: 478 total items (100 Type-TCR, 279 Type-ISR, 100 Type-JSA). The scale is intentionally small to prioritize rigor and quality over volume, as clear doctrinal changes are rare.
• Validation: The dataset underwent rigorous review by law professors and legal practitioners to ensure legal consistency, correct categorization of change types, and accurate temporal boundaries.

3. Experimental Setup

• Models Evaluated: State-of-the-art LLMs including GPT-5, Gemini Flash, LLaMA 3.3 (70B), and Qwen3 (30B).
• Settings: Evaluated with varying reasoning efforts (minimal, low, medium, high) and Chain-of-Thought (CoT) prompting.
• Baselines: Random chance accuracy is 25% (4-option multiple choice).

4. Key Results

Experimental results indicate that even the most advanced LLMs struggle significantly with context-aware legal reasoning:

• Overall Performance: Models consistently fail to outperform random baselines significantly on Type-TCR and Type-JSA.
• Type-TCR (Temporal):
  • Models perform better when the post-change norm applies but struggle significantly with pre-change scenarios.
  • They fail to jointly reason about the timing of the act, the trial, and the law change.
  • Increasing reasoning effort sometimes degrades performance, suggesting models rely on internal parametric knowledge rather than the provided context.
• Type-ISR (Sufficiency):
  • Models achieve high accuracy when information is Sufficient but performance drops drastically when information is Partial or Insufficient.
  • Context Injection Experiment: When irrelevant context (Chinese legal QA) was injected, GPT-5's performance on "Partial" cases improved slightly, suggesting models often ignore provided context in favor of internal knowledge when cues are ambiguous.
• Type-JSA (Shift Attribution):
  • Models exhibit systematic biases. For example, Gemini tends to over-attribute changes to "Normative Premises" (social shifts), while GPT favors "Factual Premises."
  • This suggests models are influenced by training data biases (likely dominated by Common Law/English data where social shifts drive precedent changes more than in Civil Law systems).

5. Key Contributions

1. New Benchmark (CALRK-Bench): The first benchmark specifically designed to evaluate context-aware legal reasoning (temporal validity, information sufficiency, and judgment shift attribution) rather than static rule application.
2. Hard Negative Mining: A novel methodology for constructing Type-ISR distractors using co-citation patterns in legal precedents to create legally plausible but incorrect options.
3. Empirical Evidence of Limitations: Demonstrates that current SOTA LLMs lack the ability to reason about the validity of norms over time and cannot reliably distinguish between different causes of legal change, relying instead on shallow heuristics or training data biases.
4. Civil Law Focus: Provides a rigorous evaluation framework based on the Korean Civil Law system, offering a counterpoint to Common Law-centric benchmarks.

6. Significance

The paper highlights a critical gap in current AI legal tools: memorization of laws does not equate to legal reasoning. Legal professionals must navigate dynamic contexts where laws expire, change, or interact with specific facts. CALRK-Bench serves as a "stress test" to reveal that models currently cannot replace human judgment in scenarios requiring temporal logic or sufficiency analysis.

The authors argue that future legal AI must move beyond the IRAC framework to handle the dynamic nature of law, and that benchmarks must evolve to test these contextual capabilities before models can be safely deployed in high-stakes legal environments.

7. Limitations

• Jurisdiction Specificity: Based on Korean law; while concepts are transferable, direct application to other jurisdictions requires adaptation.
• Data Scarcity: Clear normative changes are rare, limiting the dataset size.
• Explicit Changes Only: The dataset excludes ambiguous or gradual legal shifts to ensure evaluation objectivity, which may not fully reflect the complexity of real-world legal disputes.