💻 computer science

Augmented Question-guided Retrieval (AQgR) of Indian Case Law with LLM, RAG, and Structured Summaries

This paper proposes Augmented Question-guided Retrieval (AQgR), a framework that enhances Indian case law retrieval by generating legal issues for queries, using structured RAG summaries for context, and employing LLMs for reranking and relevance explanation, achieving significantly higher Mean Average Precision than existing benchmarks.

Original authors: Vishnuprabha V, Daleesha M Viswanathan, Rajesh R, Aneesh V Pillai

Published 2026-10-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vishnuprabha V, Daleesha M Viswanathan, Rajesh R, Aneesh V Pillai

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of law, finding the right past decision to support a current case is often the most time-consuming part of a lawyer's job. This is especially true in India, where court judgments are massive, unstructured documents that mix facts, arguments, and rulings into long, continuous blocks of text. Traditionally, computers have tried to help by searching for cases that look similar to the current situation based on the specific words used. If a lawyer describes a dispute about a factory worker's firing, the computer looks for other cases mentioning factories and workers. However, this approach often fails because the law cares more about the underlying legal principle than the specific details of the story. Two cases can involve completely different people and settings but hinge on the exact same legal question, such as whether a government action was fair. To solve this, researchers have turned to a new generation of artificial intelligence known as large language models. These systems can read and understand human language, but they face a significant hurdle: they cannot read thousands of pages of text at once. To work effectively, they need information compressed into a manageable size, and they need to be guided by the right questions rather than just raw facts.

A team of researchers from the Cochin University of Science and Technology and the Naval Physical & Oceanographic Laboratory set out to build a system that could navigate these challenges to retrieve Indian case law. They developed a framework called Augmented Question-guided Retrieval, which operates in two main stages. First, the system takes a long, messy court judgment and uses an artificial intelligence model to extract the most important parts, such as the facts, the legal issues raised, and the final ruling, organizing them into a clean, structured summary. This step is crucial because it shrinks a document that might be twenty thousand words long down to a few key fields that fit inside the computer's memory. Second, instead of searching for cases using the raw story of the current dispute, the system asks the artificial intelligence to generate the specific legal questions that the story raises. It then uses these abstract legal questions to search for past precedents, and finally, a second artificial intelligence model re-ranks the results to explain why each past case is relevant.

The researchers tested this system on a large collection of Indian Supreme Court judgments. They found that the way they structured the information mattered more than the specific search terms used. When the system searched for cases using the generated legal questions rather than the raw facts, it found more relevant precedents, but only when the past judgments had been converted into those short, structured summaries. If the system tried to search through the full, unedited text of the judgments, the short legal questions were too sparse to find a match. The study also revealed a surprising detail about how the system handles citations. Including the names of past cases in the summary actually hurt performance because it added too much noise. However, when the system summarized the legal principle behind each citation instead of just listing the name, the results improved significantly. This suggests that the computer learns better when it understands the rule behind a reference rather than just seeing the reference itself.

To ensure the system was robust, the researchers ran dozens of experiments, changing how the queries were formed and how the data was organized. They discovered that combining different types of searches—using both the facts of the case and the generated legal questions—helped the system find more relevant cases overall, even though it sometimes made the final ranking slightly less precise. They also found that while the system could handle most documents well, the very longest judgments sometimes failed to be summarized correctly, which limited the system's ability to find the perfect match for certain queries. Despite these limitations, the system achieved a level of accuracy that was significantly better than previous methods, reaching a score of roughly 0.312 on a standard test, compared to about 0.157 for the best existing systems. This improvement was achieved without needing to retrain the artificial intelligence on legal data, proving that a smart way of organizing information can be just as powerful as teaching the computer new facts.

The study concludes that the key to making artificial intelligence useful for legal research lies not in making the computer smarter, but in presenting the information in a way the computer can actually use. By compressing long judgments into structured summaries and guiding the search with generated legal questions, the researchers created a pipeline that helps lawyers find the right precedents faster. The system provides not just a list of cases, but a one-sentence explanation for why each case was chosen, allowing a lawyer to quickly decide if the full document is worth reading. While the system is not perfect and still struggles with the very longest documents, it demonstrates a practical path forward for legal technology. It shows that with the right structure, artificial intelligence can help navigate the vast and complex landscape of Indian law, turning hours of searching into minutes of reading.

Technical Summary: Augmented Question-guided Retrieval (AQgR) of Indian Case Law

Problem Statement
Legal professionals in India face significant challenges in retrieving relevant case law precedents due to the unstructured nature of judgments, which often lack consistent headings separating facts, issues, and reasoning. Traditional retrieval systems rely heavily on fact-based similarity matching (e.g., BM25, dense embeddings), which often fails to identify precedents that share the same legal principle but differ in factual details. Furthermore, existing automated systems rarely explain why a precedent is relevant, forcing lawyers to read full judgments to verify applicability. The core challenge is retrieving cases based on shared legal doctrine rather than surface-level factual similarity, all while managing the context window limitations of Large Language Models (LLMs) and avoiding hallucination without fine-tuning.

Methodology: The AQgR Framework
The proposed Augmented Question-guided Retrieval (AQgR) framework operates in two primary stages: structured summary generation and an ensemble retrieval pipeline with LLM-based reranking.

Structured Summary Generation (RAG):
- To address context window limits and the unstructured nature of Indian judgments, the system uses Retrieval-Augmented Generation (RAG) to extract specific fields from raw judgments into a structured JSON format.
- Phase 1 utilized Gemini Pro on a 50-judgment subset, employing Chain-of-Thought (CoT) prompting to handle long documents by processing them in chunks.
- Phase 2 scaled this to 496 judgments using Llama 3 70B. It introduced a robust pipeline with greedy decoding (temperature=0) and a validation/repair mechanism to ensure consistent schema adherence (21 fields, including Facts, Issues, Reasoning, and New Legal Principle).
- Crucially, the system extracts "New Legal Principles" and "Citation Principles" (the rule behind a citation) rather than just citation names, to reduce noise.
Retrieval Pipeline:
- Query Formulation: The system generates three abstract legal issue phrases from the input facts using an LLM. These are concatenated into a pipe-separated string ( $q_I$ ).
- First-Stage Retrieval: A hybrid "B5" retriever combines BM25 (weighted 0.7) and FAISS dense retrieval (weighted 0.3). The system tests three query types: Facts only ( $q_F$ ), Issues only ( $q_I$ ), and Facts+Issues ( $q_{FI}$ ).
- Second-Stage Reranking: An LLM reranks the top 100 candidates from the first stage. The reranking prompt uses the combined Facts+Issues query ( $q_{FI}$ ) to reason over the candidate pool, which is represented by the structured JSON fields. The LLM outputs a ranked list of the top 20 cases with a one-line relevance explanation for each.

Key Contributions
The paper identifies several specific contributions to legal NLP and AI:

Query-Document Density Alignment: The study establishes that retrieval performance depends on matching the lexical density of the query to the document representation. Short, issue-based queries perform poorly against long, unstructured plain-text judgments but excel against compressed structured summaries. Conversely, long, descriptive queries are necessary for plain-text retrieval.
Citation-Content Finding: The paper demonstrates that including full citation names in structured summaries adds noise and consumes context space. However, including the legal principle behind a citation significantly improves reranking performance.
Zero-Shot LLM Pipeline: The authors present a retrieval and reranking pipeline that achieves competitive results without task-specific fine-tuning, relying instead on structured summaries and prompt engineering.
Context Window Management: The work demonstrates that reducing judgments to a specific set of six key fields (Facts, Issues, Reasoning, New Legal Principle, Important Subjects, and Citation Principles) allows 227 of the 496 judgments to fit within a single LLM context window simultaneously, compared to only 98 when all 21 fields are retained, making large-scale reranking practical without needing a larger context window.
Systematic Ablation: Through eight controlled experiments, the paper isolates the impact of query formulation, document format, pooling strategies, and reranking prompts, revealing that document representation is a stronger lever for performance than query type alone.

Results

Phase 1 (Gemini Pro): On a 50-judgment corpus (14 queries), the system achieved a Mean Average Precision (MAP) of 0.365 and Recall of 0.672. Including full precedent citations in summaries degraded performance, confirming the need for selective extraction.
Phase 2 (Llama 3 70B): On a 496-judgment corpus (50 queries), the best configuration (B5 hybrid retriever with short issue queries, V4 structured format, and Facts+Issues reranking) achieved a MAP of approximately 0.312.
- Comparison to Baselines: This result is well above the FIRE 2020 leaderboard MAP of 0.1573, though the authors note the datasets and protocols are not directly comparable.
- Query Performance: On structured summaries, issue-based retrieval outperformed fact-based retrieval. However, a union pool (combining facts, issues, and combined queries) increased recall ceiling but diluted ranking quality; the issue-only pool yielded the best MAP.
- Document Format: Format V4 (including "citation principles" rather than names) outperformed all other formats, including those with full reasoning or citation names.
- Stability: The study found that retrieval is highly reproducible (95.5% match), but LLM reranking introduces run-to-run variance of approximately ±0.03–0.04 MAP, primarily due to the LLM's inconsistent selection of the number of documents.
- Metric Note: While the best MAP configuration (V4, short issue query, issues-only prompt) achieved a MAP of 0.3124, its corresponding MRR was 0.3379. Higher MRR values (e.g., 0.4121) were observed in configurations with lower MAP scores (such as the Union Pool with Facts+Issues prompt), indicating a trade-off between ranking quality and coverage in different setups.

Significance and Claims
The paper claims that AQgR provides a practical, zero-shot route to explainable case law retrieval for Indian law. Its significance lies in demonstrating that:

Structured Summaries are Critical: They enable LLMs to reason over legal doctrine rather than skimming unstructured text, making retrieval feasible without fine-tuning.
Issue-Based Retrieval is Viable: Generating legal issues from facts allows the system to find precedents based on legal principles rather than just factual similarity, aligning with how lawyers actually search.
Explainability: The system provides a relevance reason for every retrieved case, allowing lawyers to quickly assess if a full judgment needs to be read.
Practical Utility: The approach requires no fine-tuning and runs on modest hardware, making it accessible to individual practitioners and small firms.

The authors remain modest, acknowledging limitations such as run-to-run variance, the inability to summarize the very longest judgments, and the lack of exhaustive human validation for the 496 generated summaries. They conclude that while the system is a significant step forward, future work is needed to close the recall ceiling for cases with no surface vocabulary overlap and to improve the stability of LLM-generated queries.

More like this