Lightweight Domain Adaptation of a Large Language Model for Legal Assistance in the Indian Context

1. Problem Statement

The paper addresses the critical gap in access to legal assistance in India. While the Indian Constitution guarantees the right to legal resources, many citizens cannot effectively utilize their rights due to a lack of awareness and access to accurate, up-to-date legal information.

• Limitations of General LLMs: Existing general-purpose Large Language Models (LLMs) like GPT-3.5 Turbo suffer from two main issues in the legal domain:
  1. Outdated Knowledge: They lack training on recent legislative changes (e.g., the newly enacted Bharatiya Nyaya Sanhita (BNS) and Bharatiya Nagarik Suraksha Sanhita (BNSS)).
  2. Hallucinations: They frequently generate plausible but factually incorrect information, which is unacceptable in high-stakes legal contexts.
• Computational Inefficiency: Training or fine-tuning massive models (e.g., 175B parameters) for specific domains is computationally expensive and often unnecessary if a lighter model can achieve similar results with the right architecture.

2. Methodology

The authors propose Legal Assist AI, a lightweight framework that combines a quantized 8-billion parameter model with a Retrieval-Augmented Generation (RAG) system and strategic prompt engineering.

A. Data Preparation and Corpus Engineering

• Corpus Curation: A high-quality dataset of over 600 legal documents was curated from public government and judicial sources. This includes the Indian Constitution, labor laws, judicial judgments, and the new BNS and BNSS codes.
• Preprocessing: Raw text was cleaned and segmented into 1,000-character chunks with a 20-character overlap to preserve contextual continuity while ensuring precise vector representation.
• Vector Indexing:
  • Embedding: The sentence-transformer/all-MiniLM-L6-v2 model was used to convert text chunks into dense vector embeddings.
  • Storage: Vectors were indexed using FAISS (Facebook AI Similarity Search) to enable fast, real-time semantic retrieval.

B. Core Model Selection

• Base Model: Llama 3.1-8B was selected for its balance of reasoning capabilities and computational efficiency.
• Quantization: To optimize for resource-limited environments, the model was quantized to Q4_0, reducing its size from 16 GB to **4.7 GB**.
• Architecture: The model utilizes modern components including Group Query Attention (GQA), Rotary Positional Embeddings (RoPE), RMS Normalization, and SwiGLU activation.

C. System Framework (RAG + Prompt Engineering)

The system operates on a Retrieve-then-Generate workflow:

1. Retrieval Module: When a user query is received, it is embedded and searched against the FAISS vector store to retrieve the top $k$ relevant legal text chunks.
2. Prompt Construction: A structured system prompt is engineered with four components:
   • Proxy Feature: Defines the persona as an Indian legal expert.
   • Legal Constraint: Enforces safety, accuracy, and objectivity.
   • Input: Inserts the user query and retrieved context.
   • Signifier: Provides formatting cues (e.g., "Answer:") for structured output.
3. Generation Module: The quantized Llama 3.1-8B processes the enriched prompt to generate the final response.
4. Hallucination Guardrail: If the retrieval module returns no relevant context, the system explicitly states, "I don't know," rather than hallucinating an answer.

3. Key Contributions

1. Legal Assist AI Framework: A novel, lightweight framework demonstrating that an 8B parameter model, when combined with RAG and a specialized corpus, can outperform much larger generalist models in the Indian legal domain.
2. Comprehensive Indian Legal Corpus: The creation and integration of a dataset containing the latest Indian laws, including the recently enacted BNS and BNSS, which are often missing from standard LLM training data.
3. Parameter Efficiency Index (PEI): The introduction of a new metric to quantify model efficiency relative to parameter size:
   $$PEI(M) = \frac{A(M)}{\Pi(M)}$$
   Where $A(M)$ is the accuracy score and $\Pi(M)$ is the parameter count in billions.
4. Mitigation of Hallucinations: The strict RAG implementation ensures that responses are grounded in the provided corpus, significantly reducing the risk of factual errors.

4. Experimental Results

The framework was evaluated on two primary dimensions:

A. Qualitative Reasoning (Lawyer_GPT_India Dataset)

• Metric: BERT Score (measuring semantic similarity).
• Result: The model achieved a mean BERT score of 76.90%, indicating high semantic and contextual relevance in its responses to open-ended legal queries.

B. Factual Accuracy (All-India Bar Examination - AIBE)

• Dataset: 1,134 valid questions from 12 years of AIBE exams.
• Performance Comparison:
  • Legal Assist AI: 60.08% (Passed)
  • GPT-3.5 Turbo (175B): 58.72% (Passed)
  • Base Llama 3.1-8B: 43.73% (Passed)
  • Mistral 7B: 23.48% (Fail)
• Key Finding: The lightweight Legal Assist AI framework outperformed the 175B GPT-3.5 Turbo despite having 21 times fewer parameters.

C. Parameter Efficiency Index (PEI) Analysis

• Legal Assist AI PEI: 7.51
• GPT-3.5 Turbo PEI: 0.34
• Conclusion: Legal Assist AI is 22 times more parameter-efficient than GPT-3.5 Turbo. This validates that specialized, smaller models with RAG are superior to massive generalist models for domain-specific tasks.

5. Significance and Impact

• Democratization of Legal Access: The framework offers a cost-effective solution (operational costs reduced by up to 99.97% compared to traditional methods) to provide accurate legal information to the general public.
• Scalability: By using a quantized 8B model, the system can be deployed on consumer-grade hardware, making it scalable for law firms and public legal aid services without massive infrastructure costs.
• Paradigm Shift: The study challenges the notion that "bigger is better" in LLMs, proving that domain adaptation via RAG and prompt engineering is a more efficient strategy than relying on parameter count alone.
• Practical Application: The system successfully handles complex, evolving Indian laws (like BNS/BNSS) that general models fail to address accurately, offering a viable tool for legal professionals to extract relevant sections and articles efficiently.

6. Limitations and Future Work

• Limitations: The system currently lacks explicit source citation (traceability), which affects user trust. There is also a risk of user misinterpretation of AI output as formal legal advice.
• Future Directions:
  1. Implementing a continuous data pipeline to update the corpus with real-time case law.
  2. Enhancing the RAG module to explicitly cite source passages for verifiability.
  3. Introducing "expert-in-the-loop" validation to ensure data quality.