Retrieval Augmented Generation Framework for the Nepali Legal Domain Question Answering

Technical Summary: Retrieval Augmented Generation Framework for the Nepali Legal Domain Question Answering

Problem Statement
The Nepali legal domain faces significant barriers to information accessibility due to the fragmentation, poor digitization, and archaic terminology of legal texts. While Artificial Intelligence (AI) has transformed legal question answering (QA) in high-resource languages like English, the scarcity of annotated data in low-resource languages like Nepali has hindered the training of specialized Large Language Models (LLMs). The dual challenges of limited centralized legal documents and the jargon-heavy nature of these texts make manual research inefficient and preclude the creation of fine-tuned legal LLMs. Consequently, there is a critical need for a system that can provide accurate, verifiable legal answers to both the public and professionals without relying on extensive, manually annotated training datasets.

Methodology
This study proposes a Retrieval-Augmented Generation (RAG) framework tailored for Nepali legal QA, utilizing case laws extracted from the Nepal Kanun Patrika (NKP) digital archive. The methodology is structured around three core components:

1. Dataset Curation:

   • Source: 10,320 legal documents (Supreme Court rulings) were scraped from the NKP archive, covering decisions from 1992 to 2022.
   • Preprocessing: Data was cleaned to remove anomalies, normalized using Unicode NFC, and segmented into chunks of maximum 500 tokens with a 10% overlap to maintain context. The final dataset comprised 10,265 unique documents divided into 110,626 chunks.
   • Evaluation Set: A test set of 100 queries was curated by legal scholars, consisting of 79% fact-based objective questions and 21% subjective questions regarding legal procedure and reasoning.

2. Retrieval Strategies:
   The study compared sparse retrieval against dense retrieval using the following configurations:

   • Sparse Retrieval (BM25): Implemented in two modes:
     • BM25_Docs: Retrieval on complete documents.
     • BM25_Chunks: Retrieval on the segmented 500-token chunks.
   • Dense Retrieval (Embedding Models): Three models were evaluated on the chunked corpus using FAISS for indexing:
     • multilingual-e5-large (1024 dimensions).
     • multilingual-e5-base (768 dimensions).
     • LaBSE (768 dimensions).
   • Context Selection: For generation, the top-ranked document from the best-performing sparse and dense methods was selected. An ensemble "Count Method" was also tested, selecting the most frequent document among top-5 results from multiple retrievers, though it underperformed.

3. Generation Strategy:

   • Model: OpenAI's GPT-03 model was used for answer generation.
   • Prompting: A two-step prompting strategy was employed: (1) Evidence extraction, where the model identifies verbatim legal sentences supporting the answer; and (2) Answer generation based strictly on that evidence.
   • Refusal Mechanism: The model was instructed to refuse answering if sufficient legal grounding was not found in the retrieved context to prevent hallucinations.
   • Parameters: A moderate temperature (0.5) and top-p (1.0) were used to balance factuality and fluency.

4. Evaluation:

   • Retrieval Metrics: Precision@1, Hit@k, and Mean Reciprocal Rank (MRR@k) were used, alongside latency measurements.
   • Generation Metrics: Quality was assessed via Groundedness (alignment with retrieved sources), Truthfulness (alignment with reference answers), and Human Evaluation. Both "LLM-as-a-Judge" (using Gemini 2.5 Pro) and human experts evaluated the outputs.

Key Results

• Retrieval Performance: Sparse retrieval significantly outperformed dense retrieval. BM25_Chunks achieved the highest Precision@1 of 91%, followed by BM25_Docs at 88%. The best dense model, multilingual-e5-large, achieved a Precision@1 of 75%. Other dense models (e5-base, LaBSE) performed considerably worse.
• Efficiency Trade-offs: While BM25_Chunks had the highest accuracy, it suffered from high latency (1.2 seconds per query) due to the large number of chunks (110,626). BM25_Docs offered nearly comparable performance (88%) with significantly lower latency (0.1 seconds), making it the more viable strategy.
• Generation Performance: Using BM25_Docs as the retrieval backbone yielded the best generation results:
  • Successful Answer Rate: 92% (92 out of 100 queries).
  • Groundedness: 74% (automated judge).
  • Truthfulness: 85% (automated judge) and 84% (human evaluation).
  • The "Count Method" and dense retrieval methods resulted in lower success rates and higher error rates in truthfulness.

Significance and Contributions
The authors claim this study as the first systematic application of a RAG-based QA pipeline specifically for Nepali legal case laws. The primary contributions include:

1. Dataset Creation: The compilation and curation of the first public dataset of Nepali Supreme Court case laws from the NKP archive.
2. Baseline Establishment: The first baseline application of a standard RAG pipeline for Nepali legal QA.
3. Methodological Comparison: The first empirical comparison of sparse (BM25) versus dense retrieval methods in the Nepali legal domain.

Significance of Findings
The paper concludes that for the Nepali legal domain, characterized by formulaic language and specific terminology, sparse lexical matching (BM25) is more effective than semantic dense retrieval using off-the-shelf multilingual embeddings. This finding suggests that in low-resource, domain-specific settings where pre-training on legal text is unavailable, exact-word overlap provides more reliable grounding than semantic approximation. The study demonstrates that a well-constructed RAG pipeline can effectively bridge the gap in legal information accessibility for low-resource languages, providing a foundation for reliable AI systems without the need for extensive fine-tuning. The results also validate the "LLM-as-a-Judge" approach for automated evaluation, showing strong alignment with human expert assessments.