Cross-Domain Data Selection and Augmentation for Automatic Compliance Detection

1. Problem Statement

Automating regulatory compliance detection is critical for organizations but remains challenging due to the complexity, variability, and scarcity of annotated legal datasets.

• The Core Challenge: Models trained on one regulatory domain (e.g., GDPR) often fail to generalize to another (e.g., HIPAA) due to differences in terminology, structure, and reasoning styles.
• The Negative Transfer Risk: Simply augmenting a small target dataset with a large volume of data from a source domain often leads to negative transfer, where the introduction of poorly aligned samples degrades model performance rather than improving it.
• The Gap: While data augmentation is a known strategy for data scarcity, there is a lack of principled methods to select which source data to include to ensure effective cross-domain adaptation in compliance detection.

2. Methodology

The authors frame compliance detection as a Natural Language Inference (NLI) task, where a regulatory requirement serves as the premise and a contractual clause (e.g., a Data Processing Agreement) serves as the hypothesis. The model must determine if the hypothesis entails the premise (compliance) or not.

Experimental Setup

• Source Domain: GDPR-DPA dataset (General Data Protection Regulation), containing ~362k training instances.
• Target Domain: HIPAA dataset (Health Insurance Portability and Accountability Act), containing ~394 training instances (highly scarce).
• Models: Fine-tuned encoders (BERT-large, RoBERTa-large, Legal-BERT) and decoder-based LLMs (GPT-2-XL, Llama-3) for zero-shot/one-shot baselines.
• Strategy: The study evaluates how varying proportions (1% to 90%) of selected source data affect performance when transferred to the target domain.

Data Selection Strategies

The paper compares four approaches for selecting augmentation data from the source domain:

1. Random Sampling: A baseline where data is selected without filtering.
2. Moore-Lewis Cross-Entropy Difference: Uses two language models (source and target) to calculate the cross-entropy difference for source instances. Lower differences indicate higher similarity to the target distribution.
3. Importance Weighting (Density Ratio): Trains a classifier to distinguish source from target samples. The inverse probability (density ratio) is used to weight or filter source instances, prioritizing those that resemble the target distribution.
4. Embedding-Based Retrieval (DSIR): Encodes both domains using pre-trained models (e.g., BERT/RoBERTa) and retrieves source instances with the highest cosine similarity to target instances.

3. Key Contributions

1. Systematic Cross-Domain Study: The first systematic investigation into how data selection strategies mitigate negative transfer when moving from GDPR to HIPAA.
2. Comparative Evaluation: A rigorous comparison of four selection strategies (Random, Moore-Lewis, Importance Weighting, Embedding Retrieval) across various data ratios.
3. Scaling Analysis: An analysis of how performance fluctuates as the proportion of selected data increases, identifying specific "tipping points" where negative transfer occurs.
4. Empirical Guidelines: Demonstration that targeted selection is superior to bulk augmentation, providing a practical path for scalable compliance automation.

4. Key Results

• Baseline Performance: Fine-tuned models significantly outperform zero-shot/few-shot LLMs. However, models trained solely on GDPR perform poorly on HIPAA (F1 ~0.55), confirming the domain gap.
• Effectiveness of Selection:
  • Importance Weighting achieved the highest peak performance (F1 0.97 on validation) using only 5% of the source data.
  • Embedding-Based Retrieval showed the most robustness, maintaining high performance (F1 0.90) even at extreme ratios (1% and 75%).
  • Moore-Lewis provided consistent, stable improvements across all ratios.
• Negative Transfer Phenomenon:
  • Performance drops significantly at intermediate ratios (10–20%) for most methods. This indicates that adding a moderate amount of "noisy" or loosely aligned data harms the model more than it helps.
  • Performance often recovers at very high ratios (75%+) as the model leverages a broader, more representative pool of data, but this is less efficient than using small, curated subsets.
• Lexical vs. Semantic Alignment: Analysis revealed that high lexical overlap (e.g., Jaccard scores) does not guarantee semantic alignment. Later-ranked data often contained generic legal boilerplate ("may include but is not limited to") that increased lexical scores but introduced semantic noise, leading to overfitting.

5. Significance and Implications

• Efficiency over Volume: The study proves that small, carefully selected subsets of source data are often sufficient and superior to large, unfiltered datasets. This reduces computational costs and training time.
• Mitigating Negative Transfer: The findings highlight that the quality of augmentation data is more critical than the quantity. Using methods like Importance Weighting or Embedding Retrieval is essential to filter out data that causes negative transfer.
• Practical Application: For organizations dealing with multiple regulations, this approach offers a scalable solution to automate compliance checks without needing massive, manually annotated datasets for every new regulation.
• Future Directions: The authors suggest moving beyond selection to controlled augmentation (paraphrasing, back-translation) and analyzing model internal representations to further improve trustworthiness and explainability in legal AI.

Conclusion

The paper concludes that cross-domain data selection is a viable and necessary strategy for automatic compliance detection. By employing principled selection methods (specifically Importance Weighting and Embedding Retrieval), practitioners can effectively transfer knowledge from data-rich domains (GDPR) to data-scarce domains (HIPAA) while avoiding the pitfalls of negative transfer.