Legal2LogicICL: Improving Generalization in Transforming Legal Cases to Logical Formulas via Diverse Few-Shot Learning

1. Problem Definition

The core challenge addressed is Legal Semantic Parsing: the task of transforming unstructured, natural language legal case descriptions into structured, machine-interpretable logical formulas (specifically PROLEG fact formulas).

• Current Limitations: Existing approaches rely on fine-tuned models (e.g., NER-based systems or neural machine translation) which suffer from:
  • Data Scarcity: High costs and difficulty in obtaining high-quality annotated training data.
  • Poor Generalization: Models often overfit to surface-level patterns or specific entity names, failing when encountering unseen legal structures or entity variations.
  • Entity Bias: In legal texts, specific entity mentions (names, dates, contract IDs) often dominate semantic representations, causing retrieval systems to retrieve examples based on superficial entity overlap rather than underlying legal logic.
• Goal: To develop a framework that achieves high accuracy and robust generalization in converting legal text to logic without requiring task-specific supervised fine-tuning (SFT), leveraging instead the reasoning capabilities of Large Language Models (LLMs) via In-Context Learning (ICL).

2. Methodology: Legal2LogicICL

The authors propose Legal2LogicICL, a novel framework based on Retrieval-Augmented Generation (RAG) and Diverse Few-Shot Learning. The system operates in two main stages:

A. Dataset Construction: Legal2Proleg

To support evaluation, the authors created a new dataset, Legal2Proleg, which aligns legal cases with PROLEG logical formulas.

• Features: It covers diverse contract types (loan, lease, purchase, copyright) and legal issues.
• Augmentation: It utilizes a data augmentation pipeline to generate diverse templates and entity pairs, ensuring that training and test sets do not share identical templates (preventing overfitting to specific templates).

B. The Core Framework

The framework uses a hybrid few-shot retrieval strategy to select the best examples (demonstrations) to prompt the LLM. It balances semantic similarity with structural diversity through two mechanisms:

1. DiverseSim Algorithm (Latent Semantic Level):

   • A retrieval algorithm that selects exemplars by optimizing a trade-off between similarity to the query and diversity among selected exemplars.
   • It uses a tunable parameter $\lambda$ to balance these factors.
   • Mechanism: It iteratively selects the top-$k$ examples that maximize relevance to the query while penalizing redundancy with already selected examples.

2. Entity-Agnostic Template Retrieval (Structural Level):

   • Problem Mitigation: Standard semantic retrieval is biased by specific entity names (e.g., "Sam" vs. "Eva").
   • Solution: The system abstracts legal cases into templates by replacing concrete entities with typed placeholders (e.g., {Borrower}, {Agreement}).
   • Process: It retrieves exemplars based on template similarity rather than raw text similarity. This ensures the retrieved examples share the same legal logic structure even if the surface entities differ.

3. Hybrid Prompting Construction:

   • The final prompt combines two types of exemplars:
     • Case-based exemplars: Semantically similar to the query (real-world context).
     • Template-based exemplars: Structurally aligned but entity-agnostic (logical diversity).
   • This hybrid set is fed to the LLM to generate the target PROLEG logical formulas.

3. Key Contributions

1. Legal2LogicICL Framework: A novel, training-free approach for legal semantic parsing that utilizes diverse few-shot learning to improve generalization.
2. Diversity-Aware Hybrid Retrieval: The introduction of a two-level retrieval strategy (semantic + template) that explicitly mitigates entity-induced retrieval bias, a critical issue in legal domains where specific names often overshadow logical structure.
3. Legal2Proleg Dataset: A new benchmark dataset with annotated alignments between legal cases and PROLEG formulas, designed with strict template separation between training and testing to rigorously evaluate generalization.
4. DiverseSim Algorithm: A specific ranking algorithm for ICL that dynamically balances relevance and diversity using a hyperparameter $\lambda$.

4. Experimental Results

The authors evaluated the framework on Legal2Proleg and LegalCaseNER using various LLMs (Qwen3, Llama-3.1, Phi-4, and GPT-5.2).

• Generalization Performance:
  • In low-resource settings (20% seen data), Legal2LogicICL achieved 83% accuracy, whereas the state-of-the-art NER-based supervised model (LegalCaseNER) dropped to 16.61%.
  • The framework maintained high performance (95% with Phi-4) even with 80% seen data, significantly outperforming supervised baselines.
• Impact of Diversity ($\lambda$):
  • A moderate diversity setting ($\lambda = 0.6$) consistently yielded the best results across all models, confirming that balancing similarity and diversity is crucial.
• Hybrid Strategy Efficacy:
  • The hybrid strategy (3 case-based + 3 template-based exemplars) outperformed pure case-based retrieval (5 case-based exemplars) across all models.
  • This confirms that structural diversity (via templates) is more effective than simply increasing the number of semantically similar examples.
• Error Analysis:
  • NER Baseline Errors: Suffered from shallow entity boundary detection and failure to handle composite legal concepts (e.g., treating a complex harm clause as multiple unrelated entities).
  • Legal2LogicICL Errors: Primarily consisted of minor surface-form inconsistencies (e.g., "a" vs. "the", missing punctuation). These did not affect the logical structure or legal reasoning validity.
  • Soft-Match Accuracy: When accounting for semantic equivalence of entities (ignoring minor surface variations), accuracy rose significantly (e.g., from 84.72% to 92.18% for Qwen3-8B), indicating the model captures the correct legal logic even if entity strings vary slightly.

5. Significance

• Interpretability and Reliability: By generating structured PROLEG formulas, the system enables transparent, explainable legal reasoning, bridging the gap between unstructured legal text and symbolic reasoning engines.
• Scalability: The approach eliminates the need for expensive, task-specific fine-tuning and large annotated datasets, making it adaptable to new legal domains or jurisdictions with minimal data.
• Robustness: The explicit handling of entity bias and the focus on structural diversity make the system robust against the variability inherent in real-world legal documents, addressing a major bottleneck in current Legal AI systems.
• Future Foundation: The work establishes a principled paradigm for using LLMs in logic-based legal reasoning, offering a scalable path toward automated, reliable, and interpretable legal decision support systems.