LLM-Assisted Causal Structure Disambiguation and Factor Extraction for Legal Judgment Prediction

1. Problem Statement

Legal Judgment Prediction (LJP) aims to automatically predict court rulings based on case fact descriptions. While current state-of-the-art methods rely on Pre-trained Language Models (PLMs) like BERT, they suffer from two fundamental limitations:

1. Spurious Correlations: PLMs rely on statistical correlations between text and outcomes rather than explicit legal logic. This makes them prone to learning noise (e.g., punctuation changes or irrelevant entities) and lacking robustness.
2. Causal Discovery Bottlenecks: Existing causal inference approaches in LJP face two critical challenges in real-world legal texts:
   • Noisy Factor Extraction: Traditional statistical methods (e.g., keyword extraction) fail to distinguish substantive legal elements from background noise (names, dates, places), leading to inaccurate causal variables.
   • Structural Uncertainty: Causal discovery algorithms (like GFCI) often output Partial Ancestral Graphs (PAGs) with ambiguous edge directions due to Markov equivalence and the sparsity of legal text features. Resolving these ambiguities typically requires arbitrary heuristics, weakening legal interpretability.

2. Methodology: The LLM-Knowledge-GCI Framework

The authors propose an enhanced inference framework that integrates Large Language Model (LLM) priors with statistical causal discovery. The framework operates in four stages:

A. Legal Factor Discovery (Coarse-to-Fine Hybrid Extraction)

To extract clean, standard legal constituent elements from unstructured text:

1. Statistical Screening: Uses the YAKE algorithm to generate a candidate set of keywords based on saliency scores. To avoid high-frequency noise, a uniform sampling process is applied to ensure coverage of long-tail evidence.
2. Semantic Refinement (RAG + Prompting):
   • Retrieval-Augmented Generation (RAG): Retrieves relevant legal lexicons and contexts for candidate keywords.
   • LLM Reasoning Chain: An LLM acts as a "criminal law expert" using a structured prompt (Role, Goal, Chain-of-Thought, Output). It performs:
     • Semantic Parsing: Resolves polysemy.
     • Element Relevance: Maps keywords to legal constituent elements.
     • Multi-dimensional Filtering: Removes non-legal entities (e.g., PERSON, GPE, DATE) based on POS and NER constraints.
3. Node Unification: Semantic clustering (K-means) unifies synonymous terms (e.g., "steal" and "pilfer") into single causal nodes.

B. Causal Structure Learning (LLM-Assisted Disambiguation)

To resolve directional uncertainty in causal graphs:

1. Initial Discovery: The GFCI (Greedy Fast Causal Inference) algorithm generates a PAG containing deterministic, confounding, and ambiguous edges.
2. Probabilistic Disambiguation:
   • The LLM serves as a constrained soft knowledge base.
   • For ambiguous edges ($u \circ-\circ v$), the LLM evaluates the plausibility of three relations ($u \to v$, $v \to u$, $u \leftrightarrow v$) based on injected legal provisions and case facts.
   • Constraints: The process enforces Judicial Logic (facts determine judgments; no $Y \to X$) and Temporal Constraints (cause precedes effect).
3. Graph Sampling: Multiple candidate causal graphs are generated via random sampling. The quality of each graph is weighted using the Bayesian Information Criterion (BIC).

C. Causal Strength Estimation

To quantify the impact of factors on judgments:

• Propensity Score Matching (PSM): Used to estimate the Average Treatment Effect (ATE) for edges in the sampled graphs.
• Aggregation: The overall causal strength between a factor and a charge is calculated as a weighted sum of strengths across all sampled graphs, where weights are determined by the graph's BIC fit.

D. Decision Making (Causal-Aware Prediction)

• Model: Uses ALBERT as the backbone.
• Attention Constraint: An auxiliary loss function ($L_{causal}$) is introduced to align the model's attention weights with the estimated causal strengths. This forces the model to focus on tokens that have high causal relevance to the judgment, rather than just statistical correlation.

3. Key Contributions

1. Hybrid Factor Extraction: A novel "coarse-to-fine" mechanism combining statistical sampling with LLM semantic reasoning to achieve high-precision, noise-free extraction of legal elements.
2. LLM-Assisted Disambiguation: A probabilistic strategy to resolve Markov equivalence in causal graphs by leveraging LLMs as a domain knowledge base, avoiding random or deterministic assumptions.
3. Causal-Aware Prediction Model: A framework that explicitly constrains text attention mechanisms using causal graph information, significantly improving robustness and interpretability.

4. Experimental Results

The method (LLM-Knowledge-GCI) was evaluated on five datasets: LEVEN, QA, CAIL, LEDGAR, and Overruling.

• Performance: The proposed model outperformed 12 baselines (including BERT, Legal-RoBERTa, and other causal models like CASAM) across all datasets.
  • Achieved 89.31% accuracy on CAIL (vs. 88.45% for BERT).
  • Achieved 97.05% on Overruling (vs. 95.69% for BERT).
• Low-Resource Efficiency: In few-shot scenarios (1% training data), the model achieved 61.17% average accuracy, significantly outperforming baselines (e.g., BERT dropped to ~1.79% on LEVEN). This demonstrates superior data efficiency.
• Ablation Studies:
  • Removing the LLM enhancement in factor extraction reduced accuracy by ~2.9%.
  • Removing the knowledge augmentation in edge disambiguation reduced accuracy by ~1.42%.
  • Removing the causal attention constraint reduced accuracy by ~3.1%.
• Robustness: Sensitivity analyses (random confounding, placebo treatment) confirmed that the causal structures generated are robust to perturbations, unlike purely statistical models.
• Factor Analysis: Performance gains were most significant in complex cases with moderate numbers of causal factors, showing diminishing returns only when factors became excessive (redundant).

5. Significance

This paper bridges the gap between statistical NLP and legal reasoning. By integrating LLMs not just as text generators but as causal reasoning agents, the framework addresses the "black box" nature of PLMs in legal AI.

• Interpretability: It provides a transparent causal graph explaining why a judgment was made, aligning with legal requirements for reasoning.
• Robustness: It mitigates the risk of models learning spurious correlations (e.g., predicting guilt based on a defendant's name or location), which is critical for fair judicial outcomes.
• Practicality: The method is particularly effective in low-resource settings and for distinguishing between highly similar criminal charges, a persistent challenge in legal AI.