Fenced Citation-Context Retrieval for Case Law: Temporal Leakage and Degree Control Across Two Jurisdictions

Technical Summary: Fenced Citation-Context Retrieval for Case Law

1. Problem Statement

Legal precedent retrieval (PCR) aims to find cases that a new legal analysis relies upon. While a case's own text is a primary signal, "incoming citation context" (the text surrounding citations to a case in other judgments) has emerged as a powerful retrieval signal, notably via the PAT (Preceding citation Anchor Text) method. However, standard evaluations of PAT-style methods suffer from two critical flaws that compromise their deployability and interpretability:

1. Temporal Leakage: Standard methods aggregate citation contexts regardless of date. In a real-world deployment, a query case cannot be cited by cases filed after the query date. Evaluations that include these future citations effectively "cheat" by using evidence unavailable at query time, inflating performance metrics.
2. Popularity Confounding: In benchmarks where relevance is defined by citations, a candidate's performance is partially driven by its total citation count (popularity). Without controlling for this, citation-context gains may simply reflect popularity rather than the informational value of the citation text.

Furthermore, existing literature often lacks rigorous controls (temporal fences, degree-matched baselines, and placebo tests) to isolate the true signal from these artifacts.

2. Methodology

The authors propose a rigorous, zero-training framework for citation-context retrieval that enforces strict temporal fencing and popularity controls across two distinct jurisdictions: CLERC (US federal case law, 1.84M documents) and ECtHR-PCR (European Court of Human Rights, 15.7k cases).

Core Components:

• Temporal Fencing: The system constructs an "anchor channel" that aggregates citation windows (text surrounding citations) but strictly admits only citers dated before the query date.
  • CLERC: Uses a per-query year fence (admits citers from years strictly before the query year).
  • ECtHR-PCR: Uses a split-static fence (admits only citers dated before the evaluation split start date) to ensure byte-mechanical verifiability and prevent any leakage from the test set.
• Retrieval Architecture:
  • Body Channel: Standard BM25 over the case text.
  • Anchor Channel: Aggregates BM25 scores of citation windows from admitted citers.
  • Fusion: The two channels are fused using Reciprocal Rank Fusion (RRF).
• Controls and Audits:
  • Degree Controls: Explicit baselines that rank candidates by citation count (degree) or window count, and fusion baselines (BM25 + degree) to ensure gains are not merely popularity effects.
  • Placebo Tests:
    • CLERC: Family/time-matched placebo (swapping real contexts with permuted ones) to test if the signal is content-specific.
    • ECtHR-PCR: Candidate-identity placebo (swapping window scores between candidates with matched degree/count) to test if the signal is structure/mass-dominated.
  • Provenance Audits: Byte-invariance audits and manual precision checks to ensure no data leakage occurred during mapping or artifact construction.

Temporal-Admission Decomposition:
The paper introduces a novel decomposition to quantify exactly how much an "unfenced" (naive) evaluation over-credits itself. For ECtHR-PCR, the naive gain ($c - a$) is split into:

1. Non-prior-date Evidence Premium ($c - b$): The "clean leakage" (evidence from dates $\ge$ query date).
2. Split-Static Admission Cost ($b - a'$): The loss incurred by a strict fence excluding legitimate pre-query citers that a dynamic fence would catch.
3. Index/Retrieval-Set Effect ($a' - a$): Differences arising from index construction or candidate pool changes.

3. Key Contributions

1. Quantification of Temporal Leakage: The paper provides the first precise measurement of "phantom fraction"—the share of naive citation-context gains that are actually future evidence. On ECtHR-PCR, the naive gain of +4.97 R@1000 is decomposed, revealing that only +2.70 (14.9%) is genuine leakage, while the majority is an admission cost or index effect.
2. Cross-Jurisdiction Validation: The study validates the necessity of fencing and degree control across two very different legal corpora (US federal vs. European Court), demonstrating that the "deployability lesson" transfers.
3. Zero-Training Performance: The proposed fenced system achieves state-of-the-art results without training:
   • CLERC: +16.1 R@1000 over tuned BM25 on the eligible universe, clearing a fairly-tuned degree control.
   • ECtHR-PCR: 79.56 R@1000, matching the strongest published trained system (LeCoPCR-HT, 79.39) and outperforming it at shallow recall cuts and on MAP, despite using zero training.
4. Mechanism Differentiation: Through placebo controls, the paper observes that the mechanism of the gain differs by regime:
   • CLERC: Gains collapse under a content-matched placebo, suggesting the signal is content-dominated (specific text in the citation window).
   • ECtHR-PCR: Gains persist under a structure-matched placebo (retaining ~96%), suggesting the signal is structure/mass-dominated (score mass from the citation network topology) rather than specific candidate identity.

4. Results

• CLERC (US Federal):
  • The full-universe policy (fenced anchor + BM25) achieves 70.21 R@1000 vs. 54.11 for BM25 alone (+16.1 points, 95% CI [+14.3, +18.0]).
  • On the mapped subset (2,058 queries), the gain is +22.3 R@1000.
  • The system clears a "fairly-tuned" degree control (where the optimal degree weight was found to be zero, effectively making the control identical to BM25).
  • Relaxing the fence (allowing same-year or future citers) results in a monotonic dose-response increase in performance, peaking at +4.95 R@1000 over the strict fence, confirming significant over-crediting in unfenced evaluations.
• ECtHR-PCR (European Court):
  • The fenced system achieves 79.56 R@1000, beating BM25+degree (70.89) by +8.68 points.
  • It matches the performance of the strongest trained system (LeCoPCR-HT) at R@1000 but significantly outperforms it at R@50, R@100, and MAP.
  • Decomposition: The naive "relax-the-fence" gain of +4.97 is split into:
    • Clean Leakage (non-prior-date): +2.70 (14.9% of the naive gain).
    • Admission Cost (legitimate pre-query citers excluded by static fence): +3.29.
    • Index Effect: -1.02.
  • The "phantom fraction" (leakage share) is 14.9%.

5. Significance and Claims

The paper positions itself as a "rigorous Findings-tier contribution" focused on deployability audits rather than proposing a new retrieval mechanism (the PAT mechanism is borrowed). Its primary significance lies in:

1. Correcting Evaluation Hygiene: It demonstrates that without strict temporal fencing and degree controls, legal citation-context retrieval evaluations are fundamentally flawed and over-credit themselves.
2. Measurable Over-Credit: It proves that the "phantom fraction" of future evidence in unfenced evaluations is large enough to matter (measurable and reportable) and provides a decomposition method to isolate it.
3. Zero-Training Viability: It shows that a strictly fenced, zero-training approach can match or exceed the performance of complex trained systems, provided the evaluation is honest.
4. Observational Mechanism Insight: It suggests that the nature of the citation-context signal may vary by jurisdiction (content vs. structure), though this is presented as an observational hypothesis rather than a proven law due to differences in placebo designs across corpora.

The authors explicitly state they do not claim the mechanism is new, nor do they claim to be the first to use temporal induction or degree controls in adjacent fields. The novelty is the decomposition of the over-credit and the cross-jurisdiction validation of the necessity for these controls.