DALDALL: Data Augmentation for Lexical and Semantic Diverse in Legal Domain by leveraging LLM-Persona

1. Problem Statement

Data Scarcity in Low-Resource Domains: Information Retrieval (IR) in specialized fields like law suffers from a lack of high-quality, annotated training data. Creating such datasets requires expensive legal expertise.
Limitations of Current Augmentation: While Large Language Models (LLMs) can generate synthetic data, existing methods (e.g., few-shot prompting) often prioritize quantity over quality. They tend to reuse vocabulary from seed examples, leading to low lexical diversity (repetitive wording) and insufficient semantic diversity (lack of varied perspectives). This limits the robustness of retrieval models, particularly in legal IR where queries vary significantly in structure and terminology.

2. Methodology: DALDALL

The authors propose DALDALL (Data Augmentation for Lexical and Semantic Diversity in the Legal Domain by Leveraging LLM-Persona), a framework that uses persona-based prompting to generate diverse synthetic legal queries.

Core Components:

• Persona-Based Prompting: Instead of generic prompting, the LLM is assigned specific professional roles (personas) such as attorneys, prosecutors, judges, and law professors. Each persona brings a distinct perspective, objective, and linguistic pattern to query formulation.
• Two-Stage Generation Process:
  1. Essential Extraction: Before augmentation, the system extracts invariant legal components from the original query using the IRAC framework (Issue, Rule, Application, Conclusion). This ensures the generated queries preserve the core legal meaning and facts.
  2. Query Augmentation: The LLM generates synthetic queries based on the extracted essentials and the assigned persona.
     • Vanilla Prompt: Generates queries without role assignment.
     • Persona Prompt: Generates queries by simulating a specific legal professional's viewpoint.
• Experimental Setup:
  • Datasets: Evaluated on COLIEE (long-form case statements, ~6,000 tokens) and CLERC (citation retrieval, ~350 tokens).
  • Models: Tested on five dense retrievers (DPR, LegalBERT, BGE-base, multilingual-e5-base, BGE-M3) and a sparse baseline (BM25).
  • Fine-tuning: Models were fine-tuned on original data, vanilla-augmented data, and persona-augmented data (both "only" and "mix" configurations).

3. Key Contributions

1. Novel Augmentation Strategy: Introduced a persona-based approach tailored for legal IR, demonstrating that assigning specific professional roles induces significantly higher lexical and semantic diversity compared to standard few-shot prompting.
2. Empirical Analysis of Diversity: Provided a rigorous analysis showing that persona-based augmentation improves Self-BLEU scores (indicating lower similarity among generated samples) and alters the semantic space distribution, even when input text lengths vary drastically.
3. Performance Validation: Demonstrated that retrievers fine-tuned on persona-augmented data achieve competitive or superior recall performance compared to those trained on original data or generic augmentations, particularly for models like DPR and BGE-base.

4. Key Results

• Lexical Diversity: Persona-based augmentation consistently outperformed Vanilla prompting.
  • On CLERC, Self-BLEU improved by 15% (0.368 vs. 0.435).
  • On COLIEE, Self-BLEU improved by 23% (0.318 vs. 0.414).
  • Insight: Diversity gains stem from the distinct professional perspectives of the personas, not just the length of the input text.
• Semantic Diversity:
  • On COLIEE (long texts), personas produced lower Intra-Cosine Similarity (0.778 vs. 0.799), indicating they explored different semantic regions of the case facts.
  • On CLERC (short texts), personas were lexically diverse but semantically converged, suggesting that input richness influences semantic variation.
• Retrieval Performance:
  • Dense retrievers (DPR, BGE-base) fine-tuned on Persona-mix data achieved better recall scores than those trained on Vanilla data.
  • BM25 Recall: Interestingly, persona-augmented queries showed slightly lower BM25 recall than original queries, suggesting they introduce more lexical variation (challenging the model) rather than just keyword matching.
  • Model Sensitivity: The benefits were most pronounced for general dense retrievers. Specialized models like LegalBERT did not outperform baselines, likely because the "natural" phrasing of persona-generated queries conflicted with its pre-trained legal domain representations.

5. Significance and Conclusion

• Quality over Quantity: The paper establishes that in low-resource, high-stakes domains like law, generating high-quality, diverse synthetic data is more critical than generating massive volumes of mediocre data.
• Practical Framework: DALDALL offers a practical, reproducible method for enhancing training datasets in specialized IR tasks without requiring new human annotations.
• Generalizability: While focused on English legal data, the methodology (using domain-specific personas to drive diversity) is applicable to other specialized fields (e.g., medicine) where distinct professional roles exist.
• Future Work: The authors note limitations regarding the lack of human legal verification for the generated content and the need to explore cross-lingual generalization and optimal persona combinations.

In summary, DALDALL successfully leverages the reasoning capabilities of LLMs through role-playing to solve the data scarcity problem in legal IR, producing training data that is both lexically rich and semantically faithful to the original legal context.