An Senegalese Legal Texts Structuration Using LLM-augmented Knowledge Graph

This study leverages large language models to extract and structure nearly 8,000 articles from Senegalese legal texts into a comprehensive knowledge graph, thereby enhancing access to judicial information and clarifying rights and responsibilities for citizens and legal professionals.

The Gist

Imagine the legal system of Senegal as a gigantic, ancient library. Inside this library, there are thousands of books (laws, decrees, and codes) written in French. These books are messy: some pages are torn, some are just scanned pictures, and the chapters are written in a confusing code that only a few experts can decipher. For an average citizen trying to find out their rights regarding land or taxes, walking into this library is like trying to find a specific needle in a haystack while wearing blindfolded.

This paper is about building a super-smart, AI-powered librarian and a digital map to help everyone navigate this library.

Here is a breakdown of what the researchers did, using simple analogies:

1. The Problem: The "Messy Attic"

The researchers started by looking at the Senegalese legal system. They found that while there are many laws, they are scattered and hard to connect.

• The Analogy: Imagine your attic is full of boxes. One box says "Land Rules," another says "Tax Rules," but inside the "Land Rules" box, there are notes referencing the "Tax Rules" box. If you want to know how much rent you can charge, you have to dig through three different boxes to find the answer.
• The Goal: They wanted to clean up this attic, organize every single rule, and draw a map showing how every rule connects to every other rule.

2. The Solution: The "Digital Twin" (Knowledge Graph)

The team didn't just scan the documents; they built a Knowledge Graph.

• The Analogy: Think of a Knowledge Graph as a giant subway map or a social network for laws.
  • Nodes (The Stops): Each stop on the map is a piece of information (e.g., "Law 98-03," "Article 5," "President," "Minister").
  • Edges (The Tracks): The tracks connecting the stops show the relationships. For example, a track connects "Law 98-03" to "Article 5" with a label saying "contains." Another track connects "Article 5" to "Minister" with a label saying "signed by."
• The Result: They successfully pulled 7,967 articles from 20 different legal documents and turned them into a map with 2,872 stops and 10,774 tracks. Now, instead of reading a whole book, a user can see exactly how one law leads to another.

3. The Engine: The "AI Brain" (LLMs)

To build this map, they couldn't just use a simple computer script because legal language is tricky. They used Large Language Models (LLMs) like GPT-4o and Mistral-Large.

• The Analogy: Imagine you have a robot that has read every book in the world. You give it a messy legal paragraph and say, "Hey Robot, tell me: Who signed this? What law does it change? What other laws does it mention?"
• The "Triples": The robot answers in a specific format called "Knowledge Triples" (Subject → Verb → Object).
  • Example: "Article 5" → "references" → "Law 2020."
• The Experiment: They tested different "Robots" (AI models) to see which one was the best at reading these laws and building the map.
  • The Winner: GPT-4o was the smartest reader, getting the details right 86% of the time.
  • The Speedster: Mistral-Large was the fastest, finishing the job in record time while still doing a great job.

4. Why This Matters

Before this project, if a Senegalese farmer wanted to know their rights regarding public land, they might have to hire a lawyer or spend days in a dusty archive.

• The New Reality: With this system, the farmer (or a lawyer) can ask the AI, "Show me all the rules about land rent in the Dakar region." The AI instantly looks at its "Subway Map," follows the tracks, and gives a clear, organized answer.

Summary

The researchers took a chaotic, hard-to-read pile of legal documents and used advanced AI to:

1. Clean them up (extracting 7,967 rules).
2. Connect the dots (building a graph database).
3. Test the best AI brains to see which one understands the law best.

The Bottom Line: They are building a GPS for the law. Just as a GPS helps you navigate a confusing city without getting lost, this AI system helps citizens and lawyers navigate the complex world of Senegalese law, making justice more accessible and transparent for everyone.

Technical Summary

Here is a detailed technical summary of the paper "An Senegalese Legal Texts Structuration Using LLM-augmented Knowledge Graph."

1. Problem Statement

The study addresses the critical challenge of accessing and navigating legal texts within the Senegalese judicial system. Key issues identified include:

• Complexity and Fragmentation: Legal documents (laws, decrees, codes) are often interconnected, contradictory, or scattered across various formats (PDFs, DOCX, scanned images).
• Lack of Structured Access: There is no centralized, machine-readable repository for legal entities and their relationships, making it difficult for citizens and legal professionals to track amendments, citations, and rights.
• Data Scarcity: Recent jurisprudence and updates are often unavailable or exist only in unstructured scanned formats, hindering the tracking of legislative changes.
• Need for Automation: Traditional rule-based methods are insufficient for the nuanced extraction of relationships and metadata required to build a comprehensive legal knowledge base.

2. Methodology

The research proposes a hybrid approach combining rule-based extraction with Large Language Model (LLM) augmentation to construct a Knowledge Graph (KG).

A. Data Extraction and Pre-processing

• Source Material: The team processed 20 legal documents, including the Land and Public Domain Code, Public Procurement Code, Criminal Code, and others.
• Rule-Based Algorithm: A custom algorithm (Algorithm 1) was developed to parse DOCX files based on Senegalese drafting guidelines.
  • It identifies hierarchical structures (Parts, Books, Titles, Chapters).
  • It extracts metadata (Law/Decree number, signature date, domain) and content.
  • It handles specific formatting nuances, such as multiplicative adverbs (e.g., "bis," "ter") and specific rental price regulations.
• Output: The algorithm successfully extracted 7,967 articles and their associated metadata.

B. Knowledge Graph Construction (Neo4j)

• Schema Design: A graph database was built using Neo4j.
  • Node Labels: Domain, Law, Decree, Article, Official Journal, Ministerial Order, Declaration, Uniform Act, Legal Code, and Person.
  • Relationships: Defined interactions such as publish, possess, is associated, modify, repeal, frame, execute, and based on.
• Scale: The resulting graph contains 2,872 nodes and 10,774 relationships, visualizing the intricate web of Senegalese land and public domain laws.

C. LLM-Augmented Knowledge Triple Extraction

• Objective: To automate the extraction of "Knowledge Triples" (Subject-Predicate-Object) from the unstructured text of legal articles to populate and validate the graph.
• Technique: The authors employed a Few-Shot Chain-of-Thought (Few-Shot-CoT) prompting strategy.
  • Prompting: The LLMs were provided with examples of legal articles, metadata, and the desired triple output format. A "Let's think step by step" directive was included to enhance reasoning.
  • Models Evaluated: The study benchmarked several models:
    • Mistral Family: Mistral-Large, Pixtral-Large, Mistral-Nemo.
    • OpenAI Family: GPT-4o, GPT-4, GPT-4o-Mini, GPT-3.5-Turbo.
• Evaluation Metrics: Performance was measured using ROUGE metrics (R-1, R-2, R-L, R-Lsum) to compare generated triples against ground-truth data.

3. Key Contributions

1. First Large-Scale Structuration of Senegalese Law: The creation of a comprehensive dataset containing nearly 8,000 legal articles, specifically focusing on the complex Land and Public Domain Code.
2. Hybrid Extraction Framework: A novel pipeline combining deterministic rule-based parsing for structural integrity with probabilistic LLMs for semantic relationship extraction.
3. Benchmarking LLMs in Legal Contexts: A rigorous comparative analysis of state-of-the-art LLMs on the specific task of legal triple extraction, providing insights into their reasoning capabilities and formatting adherence in a non-English (French) legal context.
4. Open Graph Database: The development of a Neo4j-based graph database serving as a foundational tool for future legal AI applications in Senegal.

4. Results

• Data Volume: Successfully extracted 7,967 articles from 20 documents. The Land and Public Domain Collection contributed the majority (2,039 articles).
• Graph Statistics: The Neo4j database comprises 2,872 nodes and 10,774 relationships, successfully mapping the hierarchy and interdependencies of the legal texts.
• LLM Performance:
  • Top Performer: GPT-4o achieved the highest ROUGE scores across all metrics (R-1: 86.00, R-Lsum: 85.99), demonstrating superior accuracy in extracting complex relationships and formatting.
  • Runner-ups: GPT-4 and Mistral-Large also performed well, with scores exceeding 80% on most metrics.
  • Efficiency vs. Accuracy Trade-off:
    • Mistral-Large was the most efficient, processing 24 articles in ~2 minutes (approx. 6 seconds/article) with high accuracy (R-1: 82.90).
    • GPT-4 had the highest accuracy but the slowest inference time (~13 minutes for the test set).
    • Smaller Models: GPT-3.5-Turbo and GPT-4o-Mini showed lower accuracy (often failing to capture ellipses for article ranges or misformatting prefixes like "R." or "L.") but offered faster inference.
• Qualitative Findings: Larger models (GPT-4o, GPT-4, Mistral-Large) demonstrated better logical reasoning for identifying antecedent articles and handling complex citation ranges. Smaller models often failed to adhere to strict formatting rules or missed contextual references.

5. Significance and Future Work

• Access to Justice: The framework provides a solid foundation for building an intelligent legal assistant that can help Senegalese citizens and professionals understand their rights and obligations more effectively.
• Transparency: By structuring legal texts, the system enhances the transparency of the judicial system and facilitates the tracking of legislative amendments.
• Scalability: The O(n) time complexity of the extraction algorithm ensures the system can scale to larger datasets.
• Future Directions: The authors propose integrating Retrieval-Augmented Generation (RAG) and Reasoning and Acting (ReAct) techniques to build a production-ready legal assistant. Future work will also focus on optimizing OCR for scanned documents and expanding the knowledge graph to cover jurisprudence and court judgments.

In conclusion, the paper demonstrates that LLM-augmented Knowledge Graphs are a viable and powerful solution for modernizing legal information systems in developing jurisdictions, offering a path toward more accessible and efficient justice.