Legal$Δ$: Enhancing Legal Reasoning in LLMs via Reinforcement Learning with Chain-of-Thought Guided Information Gain

Technical Summary: Legal$\Delta$

Legal$\Delta$ is a novel reinforcement learning (RL) framework designed to enhance the legal reasoning capabilities of Large Language Models (LLMs). It specifically addresses the tendency of existing legal LLMs to engage in "fast-thinking" (producing direct answers without explicit reasoning), which limits their reliability and interpretability in complex judicial scenarios.

1. The Problem: Superficial Reasoning in Legal LLMs

While current LegalAI systems have made strides in automation, they face two primary challenges:

• Lack of Rigorous Justification: Models often default to direct answers, bypassing the multi-step Chain-of-Thought (CoT) necessary for complex legal logic.
• Limitations of Existing RLVR: Current Reinforcement Learning with Verifiable Rewards (RLVR) focuses primarily on the correctness of the final answer. This approach overlooks the quality and depth of the reasoning process, potentially rewarding models that arrive at the right answer through flawed or superficial logic.

2. Methodology: Information-Gain Guided Reinforcement Learning

The authors propose a two-stage training pipeline built upon the Group Relative Policy Optimization (GRPO) paradigm:

• Stage 1: Supervised Fine-Tuning (SFT): The model is first trained on high-quality, reasoning-enhanced legal datasets (distilled from DeepSeek-R1) to imitate expert-like reasoning behaviors.
• Stage 2: Reinforcement Learning with Information Gain: This is the core innovation. Instead of just rewarding the correct answer, Legal$\Delta$ introduces an Information-Gain Enhanced Reward ($R_{info}$).

The Information-Gain Mechanism:
The framework quantifies the "value" of a reasoning chain by measuring the semantic shift it induces in the model's internal representation. It leverages the mathematical equivalence between model logits and Q-values:

1. Dual-Mode Comparison: The model compares its confidence (logits) when generating an answer directly versus when it uses a reasoning chain ($r$).
2. $\Delta Q$ Calculation: It calculates the difference in Q-values: $\Delta Q(r) = \text{logits}_{\theta}(a|q, r) - \text{logits}_{\theta}(a|q)$.
3. Information-Theoretic Interpretation: This $\Delta Q$ is decomposed into two parts:
   • Pointwise Mutual Information (PMI): Measures how much the reasoning process correlates with the specific answer.
   • Global Confidence Regularizer: Measures whether the reasoning sharpens the model's overall decisiveness (reducing ambiguity).
4. Reward Modulation: The final reward is a product of the legal correctness ($R_{legal}$) and a sigmoid-scaled version of this information gain, ensuring the model is rewarded for reasoning that is both correct and informative.

3. Key Contributions

• Novel Reward Design: Introduced a reward mechanism that uses differential Q-value analysis to quantify the semantic contribution of reasoning steps.
• CoT Optimization: Successfully moved beyond "verifiable correctness" to optimize the process of reasoning, preventing redundant or superficial explanations.
• Mathematical Grounding: Provided a theoretical link between next-token prediction logits and reinforcement learning Q-functions to justify the information-gain metric.

4. Experimental Results

The framework was tested on various LLM scales (Qwen2.5-7B and 14B) across multiple in-domain and out-of-domain legal tasks (e.g., sentencing prediction, criminal charge prediction, and case analysis).

• Superior Performance: Legal$\Delta$ outperformed strong baselines (including DPO and R1-Distill), achieving an average improvement of approximately 10% across multiple tasks.
• Generalization: The model maintained its advantages on out-of-domain benchmarks, proving it learned robust legal logic rather than just memorizing patterns.
• Enhanced Confidence & Quality: Ablation studies and logit analysis confirmed that Legal$\Delta$:
  • Increased model confidence (higher average logits).
  • Increased "Legal Token Prominence" (the model pays more attention to legally significant terms).
  • Produced lower perplexity in reasoning chains, indicating higher-quality text generation.

5. Significance

Legal$\Delta$ represents a significant step toward trustworthy LegalAI. By incentivizing models to "think" more deeply and providing a mathematical way to reward the quality of thought, the framework addresses the critical need for interpretability and rigor in judicial decision-support systems. It provides a blueprint for applying information-theoretic principles to optimize reasoning in specialized, high-stakes domains.