Leverage Knowledge Graph and Large Language Model for Law Article Recommendation: A Case Study of Chinese Criminal Law

This paper proposes a hybrid law article recommendation framework that integrates a Case-Enhanced Law Article Knowledge Graph (CLAKG) with Large Language Model reasoning to significantly improve judicial efficiency and accuracy in Chinese criminal law cases, achieving a notable performance boost from 0.549 to 0.694.

The Gist

Imagine you are a judge in a busy courtroom. Every day, you face piles of new cases. Your job is to read the messy story of what happened, find the exact law that applies, and write a fair verdict. It's like trying to find a specific needle in a haystack, but the haystack is made of thousands of laws, and the needle changes shape every time.

This paper introduces a new "super-assistant" for judges. It combines two powerful tools: a Massive Digital Library (Knowledge Graph) and a Super-Intelligent Brain (Large Language Model).

Here is how it works, broken down into simple concepts:

1. The Problem: The "Hallucinating" Brain

In the past, judges (and computers) tried to guess the right law just by reading the case story.

• Old Computers: They were like students who memorized keywords. If a case mentioned "stealing," they looked for the word "theft." But they missed the meaning. If the text was complex, they got confused.
• New AI (LLMs): These are like brilliant students who have read the entire library. They understand the story well. But, they have a bad habit called "hallucination." Sometimes, to sound confident, they make up laws that don't exist or cite the wrong article number. In law, making things up is dangerous.

2. The Solution: The "Case-Enhanced" Library (CLAKG)

The authors built a special, organized library called CLAKG (Case-Enhanced Law Article Knowledge Graph).

• The Metaphor: Imagine a giant, interactive spiderweb.
  • The Nodes (Dots): These are the laws (like Article 385) and the past court cases.
  • The Strings (Lines): These connect a past case to the law it used. They also connect specific details (like "bribery" or "abuse of power") to the laws.
• The Magic: Instead of just reading text, the computer looks at this web. It sees that "Case A" is connected to "Law X" because they both share the detail "bribery." It's not just guessing; it's following the map.

3. How the System Works (The "Closed Loop")

The system works in three steps, like a team of experts:

• Step 1: The Librarian (Building the Map)
  The system uses AI to read old court documents and automatically update the spiderweb. It links new cases to the right laws. If a human lawyer spots a mistake, they fix it, and the web gets smarter. This is the "Closed Loop"—the system learns from human feedback.

• Step 2: The Detective (Finding Clues)
  When a new case comes in, the system doesn't just read the text. It acts like a detective looking for "keywords" (like "taking money" or "abusing power"). It uses the spiderweb to find the top 5 laws that are most similar to these clues.

• Step 3: The Judge's Assistant (The Recommendation)
  The AI (the "Brain") is given a strict rule: "You can only choose from these 5 laws the Detective found. Do not make up new ones."
  The AI then compares the new case to these 5 options and says, "I recommend Article 385 because it matches the bribery details." Because it is forced to look at the real data from the web, it stops making things up.

4. The Results: Why It Matters

The researchers tested this on Chinese Criminal Law.

• The Old Way: Without this system, the AI was right about 55% of the time.
• The New Way: With the "Spiderweb Library" and the strict rules, the AI was right 69% of the time.
• The Comparison: It beat every other method they tried, including standard computer models and other AI tools that didn't use this special library.

The Big Picture Analogy

Think of it like GPS vs. a Guessing Game.

• Old AI: Like a driver guessing the route based on a vague memory. They might get close, but they often end up in a dead end or a fake street.
• This Paper's System: Like a GPS that has a live map of every road and traffic report. It doesn't guess; it calculates the exact path based on real data. If the driver (the human expert) sees a road is closed, they update the map, and the GPS gets even better for next time.

In short: This paper shows that if you give a super-smart AI a strict, organized map of the law and force it to stick to that map, it becomes a much more reliable and accurate tool for helping judges do their jobs.

Technical Summary

1. Problem Statement

The paper addresses the critical challenge of law article recommendation in judicial systems, specifically within the context of Chinese Criminal Law.

• Context: Grassroots courts face massive case backlogs, and judicial decisions rely heavily on human cognitive effort. Intelligent tools are needed to assist judges in identifying applicable laws efficiently.
• Limitations of Existing Methods:
  • Traditional ML (BERT, CNN, LSTM): These models treat the task as simple text classification (mapping case facts to Article IDs). They often ignore the semantic content of the law articles themselves, struggle with data scarcity, and are highly sensitive to label imbalance (e.g., collapsing to majority labels).
  • Large Language Models (LLMs): While LLMs offer reasoning capabilities, they suffer from hallucinations, frequently generating plausible but incorrect or non-existent legal citations.
  • Standard RAG (Retrieval-Augmented Generation): Existing RAG approaches often rely on lexical similarity (TF-IDF, BM25) or generic graph structures that fail to capture the fine-grained, structured relationships specific to legal domains (e.g., the link between specific case facts and key legal elements).

2. Methodology

The authors propose a closed-loop framework combining a Case-Enhanced Law Article Knowledge Graph (CLAKG) with Large Language Models (LLMs). The methodology consists of three main stages:

A. Case-Enhanced Law Article Knowledge Graph (CLAKG) Construction

The core innovation is a pre-designed schema that unifies law articles and historical cases.

• Schema Design: The graph consists of two sub-graphs:
  1. Law Article Knowledge Graph (LAKG): Contains nodes for original law articles, Article IDs, and "Key Information" (extracted legal elements).
  2. Adjudicated Cases Knowledge Graph (ACKG): Contains nodes for case names, court session times, prosecution reasons, and specific case details.
• Integration: The two graphs are linked via relationships such as Applicable Law (Case $\to$ Article) and Agree With (Case $\to$ Key Information).
• Automated Construction: An LLM is used to extract key information and case details from raw judgment documents. Crucially, a human-in-the-loop mechanism validates and refines these extractions to ensure the graph schema remains consistent and free of hallucinated concepts.

B. Graph Embedding and Retrieval

• Embedding: The CLAKG is processed using Relational Graph Convolutional Networks (RGCN). This generates low-dimensional vector embeddings for all nodes (cases, articles, key info), capturing the topological structure and multi-relational nature of the legal data.
• Retrieval Pipeline:
  1. Keyword Matching: For a new case input, an LLM identifies relevant "Key Information" nodes from the CLAKG.
  2. Candidate Selection: The system traverses the graph to find law articles connected to these key nodes.
  3. Ranking: Cosine similarity is calculated between the new case's key information embeddings and the embeddings of candidate law articles. The top $q$ articles are selected as candidates.

C. LLM-Enhanced Recommendation (Closed-Loop)

• Prompt Engineering: The retrieved candidate law articles and linked historical cases are fed into the LLM as a structured prompt.
• Constraints: The prompt explicitly restricts the LLM to choose only from the retrieved candidate set, preventing the generation of non-existent articles.
• Human Feedback Loop: Legal experts review the system's recommendations. If correct, the new case and its validated links are added to the CLAKG. If incorrect, the graph is updated to remove erroneous links and add correct ones. This continuously refines the knowledge base.

3. Key Contributions

1. CLAKG Schema & Construction: Proposed a novel, task-oriented schema that integrates statutory law and adjudicated cases, along with an automated, LLM-driven construction pipeline that includes expert validation to ensure data quality.
2. Closed-Loop Framework: Designed a retrieval-augmented generation (RAG) system where the LLM is grounded in the CLAKG. This mitigates hallucinations by forcing the model to reason over verifiable, retrieved evidence rather than relying solely on parametric memory.
3. Empirical Validation: Conducted extensive experiments on a dataset of 1,170 Chinese criminal judgments (covering Amendment XI of the Criminal Law), demonstrating significant performance gains over both traditional classifiers and state-of-the-art RAG baselines.

4. Experimental Results

The study compared the proposed LLM + CLAKG method against several baselines:

• Classical Text Classification: BERT (0.289), GRU (0.427), DPCNN (0.495). These models struggled with label imbalance and data sparsity.
• Standard RAG & LLMs:
  • Plain LLM (GPT-4): 0.549
  • TFIDF-RAG: 0.597
  • Light-RAG: 0.636
  • Graph-RAG (Generic): 0.179 (Failed due to mismatch with legal structure).
• Proposed Method:
  • LLM + CLAKG (without case info): 0.676
  • LLM + CLAKG (with case info): 0.694

Key Findings:

• The proposed method improved accuracy from 0.549 (LLM only) to 0.694, outperforming all baselines.
• The inclusion of historical case information within the CLAKG provided a significant boost, proving that grounding the LLM in specific precedents is crucial.
• Ablation studies using Llama-3.1-405B confirmed the framework's robustness, achieving 0.786 accuracy with CLAKG augmentation.

5. Significance and Future Work

• Significance: The paper demonstrates that combining structured legal knowledge (KG) with the reasoning power of LLMs effectively solves the "hallucination" problem in legal AI. It offers a practical solution for judicial efficiency, particularly in civil-law systems with codified statutes.
• Scalability: The framework is designed to be adaptable to other jurisdictions, though it requires schema adjustments for common-law systems (which rely more on precedent networks).
• Future Directions:
  • Expanding the dataset as more judgments become public.
  • Exploring synthetic data generation (using LLMs to create hypothetical cases for training, verified by experts) to address data sparsity.
  • Refining the information extraction pipeline to reduce noise in the initial graph construction.
  • Developing quantitative metrics to evaluate the efficiency of the human-in-the-loop feedback mechanism itself.

Availability: All source code, the constructed CLAKG, and processed datasets are publicly available on GitHub.