Structured Legal Document Generation in India: A Model-Agnostic Wrapper Approach with VidhikDastaavej

This paper introduces VidhikDastaavej, a large-scale anonymized dataset of 133 categories of Indian private legal documents, and proposes a Model-Agnostic Wrapper (MAW) framework that significantly enhances the factual accuracy, coherence, and completeness of structured legal document generation compared to fine-tuned baselines.

The Gist

Imagine you are a lawyer in India. You have a client who needs a very specific legal document, like a contract for selling a business or a power of attorney. In the old days, you would spend hours typing, checking laws, and making sure every clause was perfect. It's tedious, expensive, and prone to human error.

Now, imagine you have a super-smart AI assistant that can write this for you in seconds. Sounds great, right? But here's the catch: AI is notoriously bad at writing long, structured documents without making things up (hallucinating) or forgetting important details. It's like asking a brilliant but scatterbrained architect to build a skyscraper without a blueprint; they might get the bricks right, but the building could collapse because the rooms don't connect logically.

This paper introduces a solution to that problem, specifically for the Indian legal system. Here is the breakdown in simple terms:

1. The Missing Ingredient: A Specialized Library (VidhikDastaavej)

Before this paper, AI models trying to write Indian legal documents were like chefs trying to cook a traditional Indian feast without ever seeing the ingredients. They only had access to public court judgments (which are like reading news reports about fights), not the actual private contracts and agreements lawyers use daily.

The authors created VidhikDastaavej (which translates to "Legal Documents" in Hindi).

• What is it? A massive, secret library of over 11,000 real-world private legal documents (like employment contracts, stock options, and leases) that have been scrubbed of names and private info.
• Why it matters: It's the first time anyone has given AI a "cookbook" of actual Indian legal drafting styles. It covers 133 different types of documents, teaching the AI the specific "flavor" and structure of Indian law.

2. The Smart Assistant: The Model-Agnostic Wrapper (MAW)

The authors realized that simply feeding this data to an AI and saying "Write a contract" doesn't work well. The AI gets overwhelmed and starts making things up.

So, they built a Model-Agnostic Wrapper (MAW). Think of this not as a new brain, but as a super-organized project manager that sits between you and the AI.

Here is how the "Project Manager" works:

• Step 1: The Blueprint (Planning): Instead of asking the AI to write the whole document at once, the wrapper first asks the AI: "What are the chapter titles for this contract?" (e.g., Introduction, Parties, Payment, Termination). The user can review and fix these titles. This is like drawing the floor plan before laying a single brick.
• Step 2: The Construction (Retrieval): Now, the wrapper asks the AI to write just one section at a time. But here's the magic trick: before the AI writes the "Payment" section, the wrapper looks back at the "Introduction" section it just wrote and feeds that context back to the AI.
• The Analogy: Imagine writing a novel. If you ask a writer to write the whole book in one go, they might forget the character's name from page 1 by page 100. But if you ask them to write Chapter 1, then give them a summary of Chapter 1 before asking for Chapter 2, the story stays consistent. The wrapper does this for legal documents, ensuring the "Payment" section matches the "Parties" section perfectly.

3. Why "Model-Agnostic" Matters

"Model-Agnostic" is a fancy way of saying "It works with any brain."

• You don't need to retrain the AI from scratch (which is expensive and slow).
• You can use a free, open-source AI, a paid commercial one (like GPT-4), or a private one your company owns.
• The "Wrapper" is the same for all of them. It's like a universal remote control that works on any TV brand.

4. The Results: Does it actually work?

The team tested this against standard AI methods and even against the very expensive, top-tier GPT-4.

• The Problem with Standard AI: When they just fine-tuned (trained) the AI on the new data, it actually got worse. It became rigid and started repeating patterns too much, missing critical details.
• The Wrapper Success: When they used the Wrapper (the Project Manager approach), the results were amazing.
  • Fewer Mistakes: The AI stopped making up fake laws or changing the contract type (e.g., turning a lease into a sale).
  • Better Structure: The documents looked like they were written by a human lawyer, with all the right sections in the right order.
  • Expert Approval: Real Indian lawyers reviewed the AI's work. They gave the Wrapper-generated documents high scores for accuracy and completeness, often rating them higher than the expensive GPT-4.

5. The Human Safety Net (Human-in-the-Loop)

The authors know AI isn't perfect yet. They built a system where a human lawyer can step in, tweak the chapter titles, and approve the final draft. It's a collaboration, not a replacement. The AI does the heavy lifting of drafting, and the human ensures the final product is legally sound.

Summary

This paper is about teaching AI to be a better legal secretary.

1. They gave it a library of real Indian legal documents it had never seen before.
2. They gave it a project manager (the Wrapper) that forces it to plan the structure first and write section-by-section, preventing it from getting confused or making things up.
3. They proved that this method works better than just "training" the AI, and it works with any AI model, making legal drafting faster, cheaper, and more accurate for everyone in India.

It's a step toward a future where lawyers spend less time typing and more time thinking, with AI handling the heavy lifting of the paperwork.

Technical Summary

1. Problem Statement

The paper addresses the significant gap in automating the generation of private legal documents within the Indian legal context. While Large Language Models (LLMs) have been applied to public legal tasks (e.g., judgment prediction, summarization), their application to drafting private contracts and agreements remains underexplored due to:

• Data Scarcity: Private legal documents are confidential, leading to a lack of public, high-quality training datasets.
• Complexity: Legal drafting requires long-form generation with strict structural coherence, factual accuracy, and adherence to specific legal conventions.
• Model Obsolescence: Traditional Supervised Fine-Tuning (SFT) approaches are resource-intensive and quickly become outdated as new, more powerful LLMs are released. End-users (law firms) often lack the resources to retrain models or prefer to keep proprietary models private.

The core challenge is to develop a system that generates high-quality, structured legal drafts that are factually accurate, coherent, and model-agnostic (compatible with any LLM without retraining).

2. Key Contributions

The authors present four primary contributions:

1. VidhikDastaavej Dataset: A novel, large-scale, anonymized dataset of private Indian legal documents.
   • Scale: 11,825 documents (11,692 for training, 133 for testing).
   • Diversity: Covers 133 distinct legal document categories (e.g., License Agreements, Stock Options, Power of Attorney).
   • Privacy: Rigorously de-identified using NER-based redaction and expert manual verification.
2. Model-Agnostic Wrapper (MAW): A two-stage generation framework that decouples the generation process from specific model architectures.
3. Expert-Based Evaluation Metrics: A structured evaluation protocol focusing on Factual Accuracy and Completeness, rated by legal professionals on a 1–10 Likert scale, moving beyond standard lexical metrics.
4. Human-in-the-Loop (HITL) System: An interactive interface allowing users to define document types, customize section structures, and iteratively generate drafts.

3. Methodology

A. The VidhikDastaavej Dataset

Curated in collaboration with an Indian law firm, the dataset underwent a multi-stage pipeline:

• Categorization: LLM-based classification (Mixtral) followed by expert validation to map documents to 133 categories.
• Structuring: Section headers were generated and manually reviewed to ensure alignment with professional drafting norms.
• Anonymization: A Named Entity Recognition (NER) pipeline (using spaCy) replaced entities (PERSON, ORG, LOC, etc.) with placeholders (e.g., [PERSON]). This was supplemented by human verification to ensure no residual PII remained.

B. Model-Agnostic Wrapper (MAW)

The MAW is a lightweight, two-phase workflow designed to work with any base LLM (open-source or closed-source):

• Phase 1: Section Title Generation (Planning)
  • The user provides a document title and description.
  • The LLM generates a structured list of section titles (e.g., "Parties," "Obligations," "Termination").
  • Human-in-the-Loop: Users can review, rename, insert, or delete sections before proceeding.
• Phase 2: Section Content Generation (Iterative)
  • Content is generated iteratively for each finalized section title.
  • Retrieval-Augmented Generation (RAG): As each section is generated, a concise summary is stored in a vector database (ChromaDB).
  • For subsequent sections, the model queries the vector DB to retrieve relevant summaries of previous sections. This context is injected into the prompt to ensure global coherence and prevent hallucinations.
  • The final document is assembled and refined.

4. Experimental Setup & Evaluation

• Models Tested: Open-source models (Qwen3-14B, LLaMA-3.1-8B-Instruct, Gemma-3-12B-It) were tested in three configurations:
  1. Base (Zero-shot).
  2. Supervised Fine-Tuned (SFT).
  3. Wrapped with MAW.
  • Baseline: GPT-4o (proprietary, used for inference only).
• Evaluation Metrics:
  • Lexical/Semantic: ROUGE-L, BLEU, METEOR, BERTScore, BLANC.
  • Automatic LLM: G-Eval (using GPT-4o-mini to score factuality and coherence).
  • Expert Evaluation: Three legal professionals rated drafts on Factual Accuracy (adherence to instructions/facts) and Completeness (coverage of mandatory clauses) on a 1–10 scale.
  • Reliability: Inter-Annotator Agreement (IAA) measured via Fleiss' κ, ICC, and Krippendorff's α.

5. Results and Analysis

A. Performance of SFT vs. MAW

Contrary to expectations, Supervised Fine-Tuning (SFT) degraded performance for most open-source models.

• Reasoning: SFT models overfit to dominant patterns in frequent categories and failed to generalize to rare document types. They also struggled with the "flat" instruction style of SFT compared to the structured, iterative nature of legal drafting.
• MAW Superiority: The Wrapper approach significantly outperformed both base and SFT models.
  • Example: Gemma-3-12B-It SFT achieved an expert factual accuracy score of 1.00, whereas the same model with the MAW wrapper achieved 8.82.
  • Comparison: MAW-enhanced open-source models achieved scores comparable to or exceeding GPT-4o (e.g., Gemma-MAW: 8.82 vs. GPT-4o: 8.80).

B. Ablation Study

• Retrieval Module: Removing the retrieval component caused a drop of 2.2 points in factual accuracy and 1.7 points in completeness, proving that grounding generation in retrieved context is critical.
• Structure vs. Retrieval: Neither "Structure Only" nor "Retrieval Only" was sufficient; the combination of both was necessary for high performance.

C. Inter-Annotator Agreement (IAA)

• Wrapper-based outputs showed significantly higher consistency among expert raters (Fleiss' κ > 0.80, ICC > 0.90) compared to base/SFT models (κ ~ 0.35–0.45). This indicates that the structured generation approach produces more predictable and evaluable outputs.

D. Hallucination Reduction

The MAW framework drastically reduced hallucinations (e.g., inventing clauses, altering contract types, or citing wrong laws) by enforcing a step-by-step planning and retrieval process.

6. Significance and Conclusion

• Paradigm Shift: The paper demonstrates that for complex, high-stakes domains like legal drafting, structured prompting and retrieval are more effective than traditional fine-tuning, especially when data is scarce or diverse.
• Accessibility: The Model-Agnostic approach allows legal practitioners to use existing open-source or proprietary models without the cost and technical barrier of fine-tuning.
• Benchmark: VidhikDastaavej establishes the first standard benchmark for private legal document generation in India, filling a critical gap in Legal AI resources.
• Practicality: The HITL system ensures that AI acts as an assistive tool, maintaining human oversight for legal validity and ethical compliance.

Future Work: The authors plan to expand the test set size, integrate case precedents for more robust drafting, and optimize computational overhead (currently ~1.4–1.6x inference time) for real-time deployment.