💬 NLP

TypedCSIP: Typed Counterfactual Pretraining for Chinese Legislative Conflict Classification

This paper introduces TypedCSIP, a typed counterfactual pretraining method that leverages expert-written minimal revisions to improve Chinese legislative conflict classification, achieving statistically significant gains in macro-F1 scores on the LCR-CN benchmark while demonstrating task-specific specialization.

Original authors: Yao Liu

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yao Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge trying to figure out if two laws are fighting each other. One law is the "Big Boss" (a national statute), and the other is a "Little Rule" (a local regulation). Sometimes, the Little Rule breaks the Big Boss.

Your job is to look at the pair of laws and answer two questions:

Are they fighting? (Yes or No)
How are they fighting? Is it about who is responsible? Is it about the conditions required? Is it about the punishment? Or is it about how a word is defined?

This is the task the paper tackles: Chinese Legislative Conflict Classification.

The Problem: The "Black Box" Verdict

Usually, computer programs trained to do this just look at the two laws and guess the answer. They are like a student who memorized the final answer key but doesn't understand why the answer is correct. They see the conflict, but they can't point to the specific sentence that caused the fight.

The Secret Weapon: The "Expert Editor"

The researchers found a hidden treasure in their data. Alongside every pair of fighting laws, there was a tiny, expert-written note. This note showed exactly how a human legal expert fixed the Little Rule to make it stop fighting the Big Boss.

Think of it like this:

The Conflict: A student writes, "I will eat apples and oranges."
The Rule: "You can only eat fruit."
The Expert Fix: The expert changes the sentence to, "I will eat apples." (Removing "oranges" because they aren't fruit in this context).

The paper argues: Why not teach the computer to learn from the expert's fix?

The Solution: TypedCSIP (The "Counterfactual" Teacher)

The authors created a method called TypedCSIP. They split the training into two stages, like a two-semester course.

Semester 1: The "What-If" Training (Pretraining)
Instead of just showing the computer the fighting laws, they show it three things:

The Big Boss Law.
The Little Rule (that is fighting).
The Expert Fix (the version where the fight is gone).

The computer is asked to play a game: "Look at the fighting version. Now look at the fixed version. What specific change did the expert make to stop the fight?"

If the expert removed a word about "punishment," the computer learns: "Ah, this is a Sanction Conflict."
If the expert changed a definition, the computer learns: "This is a Definition Conflict."

The computer learns to recognize the type of conflict by studying how the expert solved it. It's like learning to spot a typo by watching someone correct it.

Semester 2: The Final Exam (Fine-tuning)
Now, the computer takes the knowledge it learned in Semester 1 and applies it to a new test. It looks at a new pair of laws (without seeing the expert fix) and has to guess: "Are they fighting? If so, what type of fight is it?"

The Results: Did it Work?

The researchers were very careful. They didn't just run the test once; they ran it 18 times with different random starting points (seeds) to make sure the results weren't just luck. They also locked their rules before they started, so they couldn't change the rules if the results looked bad.

The Verdict:

Yes, it worked. The new method beat the old, standard methods.
It improved the accuracy by about 1% (which sounds small, but in this high-stakes legal field, it's a big deal).
It worked best on the specific types of conflicts (like "Definition" or "Condition") rather than just guessing "No Conflict."

The "Surprise" Discovery

The researchers had two slightly different versions of their method:

Version 1 (The Simple One): Learn from the expert fix, then throw away the "expert fix" tool and just use the knowledge.
Version 2 (The Complex One): Learn from the expert fix, keep the "expert fix" tool, and use it again while taking the final exam to help remember the lessons.

They thought Version 2 would be better because it "remembers" the lessons better. But when they tested it, Version 1 (the simple one) did just as well.

The Lesson: The magic wasn't in the complex memory trick; the magic was in the Semester 1 training itself. Just learning from the expert's edits was enough to make the computer smarter.

The "Cold Start" Test

The researchers also worried: "Did the computer just memorize the specific examples it saw in training?"
To check this, they tested it on a group of laws where the "Big Boss" and the "Expert Fix" combination had never been seen before.

Result: The computer still performed better than the old methods. This proves it actually learned the concept of legal conflict, not just memorized the answers.

Summary

The paper introduces a smart way to teach computers to find legal conflicts. Instead of just showing them the problem, they show the computer the problem and the expert's solution. By studying the solution, the computer learns to spot the specific type of legal fight. It's a simple, effective method that works better than previous attempts, and it proves that learning from "how to fix it" is a powerful way to learn "how to spot it."

Technical Summary: TypedCSIP

Problem Definition
The paper addresses the task of Chinese Legislative Conflict Classification using the LCR-CN benchmark. The goal is to classify a pair of legal provisions—a superior law ( $A$ ) and a subordinate provision under review ( $B$ )—into one of five mutually exclusive categories: Responsibility conflict, Condition conflict, Sanction conflict, Definition conflict, or No-Conflict.

This task presents specific challenges:

Granularity: The evidence for a conflict type is often a small, localized span within long legal texts.
Data Scarcity and Imbalance: The dataset is relatively small (696 test records), with the Definition class being particularly rare ( $n=48$ ).
Structural Overlap: The benchmark design includes structural overlap between training and test sets. Specifically, distinct subordinate provisions constrained by the same superior law often share the same expert-written minimal revision ($eB$), creating $(A, eB)$ tuple overlaps that could allow models to "shortcut" learning by memorizing revision patterns rather than understanding conflict logic.
Lack of Specialized Methods: Prior work on Chinese legal NLP focuses heavily on judgment prediction or retrieval. Existing baselines for LCR-CN treat the task as a standard classification problem, compressing the decision into a single label without utilizing the expert revision annotations provided in the dataset.

Methodology: TypedCSIP
The authors propose TypedCSIP (Typed Counterfactual Selective Intervention Pretraining), a two-stage method that leverages the expert-written minimal revision ($eB$) as a counterfactual supervision signal during pretraining.

Stage 1: Typed Counterfactual Pretraining
- Input: Triplets $(A, B, eB)$, where $A$ is the superior law, $B$ is the conflicting subordinate provision, and $eB$ is the expert revision that resolves the conflict.
- Architecture: A shared encoder (e.g., chinese-roberta-wwm-ext or SAILER) produces a [CLS] representation. This is fed into a Typed Factor Head ( $W_\phi$ ) with four rows corresponding to the four conflict types.
- Monotone-Complement Classifier: Instead of a standard 5-way softmax, the model uses a specialized classifier where the No-Conflict logit is derived as a strictly decreasing function of the four conflict factor evidences. This enforces the domain constraint that "No-Conflict" is the absence of any conflict evidence.
- Loss Function: The pretraining objective consists of three components:
  1. Positive Loss: Forces the true conflict factor of $(A, B)$ to fire.
  2. Counterfactual Loss: Forces all four factors of the expert revision $(A, eB)$ to be silent (zero evidence).
  3. Selectivity Loss: Penalizes changes in non-target factor scores between $B$ and $eB$, ensuring the model learns to localize the conflict signal to the specific factor being edited.
- Goal: To train the encoder to recognize conflict types by observing how an expert resolves them, treating the revision as a "counterfactual positive" that carries no conflict evidence.
Stage 2: Transfer to Classification
The paper evaluates two variants of the transfer:
1. TypedCSIP-v1 (Typed-Discard): The typed factor head and monotone-complement parameters are discarded. A fresh 5-way classification head is initialized and fine-tuned using standard class-weighted cross-entropy.
2. TypedCSIP-v2 (Anti-Forget Replay): The typed factor head is retained as a live parameter group during fine-tuning. An auxiliary replay loss (identical to Stage 1) is applied to independently sampled triplets to prevent the fine-tuning process from erasing the conflict-discrimination structure learned in Stage 1.

Experimental Protocol and Rigor
The study is a pre-registered confirmatory study registered on the Open Science Framework (OSF) before any measurements were observed.

Seeds: 18 seeds were derived deterministically from a cryptographic hash of a previous pre-registration, eliminating post-hoc seed selection.
Decision Rule: A "locked seed-as-unit" rule requires the mean per-seed improvement to exceed 0.8 percentage points (pp) and for both the seed-bootstrap and Student-t 95% confidence intervals to have lower bounds strictly above zero.
Backbones: Experiments were run on chinese-roberta-wwm-ext (primary) and SAILER (cross-backbone replication).
Baseline: The method is compared against C2, a reimplementation of a strong single-model dense classifier that concatenates $(A, B)$ and uses class-weighted cross-entropy, without access to $eB$.

Key Results

Confirmatory Success: The pre-registered TypedCSIP-v2 variant passed the locked decision rule on both backbones:
- RoBERTa: Mean improvement of +0.916 pp (CI: [+0.512, +1.346]).
- SAILER: Mean improvement of +1.288 pp (CI: [+0.676, +1.922]).
Auxiliary Findings (Matched-Seed Audit): A re-run of the simpler TypedCSIP-v1 (without replay) on the same 18 seeds achieved a mean improvement of +1.137 pp on RoBERTa. This suggests that the Stage-1 typed CSIP pretraining is the "load-bearing" component of the family, while the Stage-2 anti-forget replay mechanism (v2) is not strictly necessary for the gain on this task.
Stratified Evaluation (Cold-Start): When evaluated on the Unseen-gB stratum (test records where the $(A, eB)$ tuple is novel relative to training), the gains remained positive and significant on both backbones. The gains were concentrated in the conflict-type classes (especially Definition and Condition) rather than the No-Conflict class, refuting the hypothesis that the improvement is driven by memorizing exact tuple overlaps.
Cross-Task Limitation: The Stage-2 encoder, specialized for conflict classification, performed poorly when reused as a bi-encoder for the Superior Law Retrieval task (Task 1), showing a significant drop in performance.

Significance and Claims
The paper claims to introduce a novel combination of typed counterfactual contrastive pretraining and expert-written legal revisions for the specific domain of Chinese legislative conflict review.

Methodological Contribution: The core contribution is the use of expert revisions ($eB$) as counterfactual positives to train a typed factor head, allowing the model to learn the "resolution" of a conflict to understand the conflict itself.
Reproducibility: The study demonstrates that a pre-registered, locked-decision-rule protocol is feasible in NLP, providing a template for rigorous evaluation on small, applied benchmarks where structural data overlaps exist.
Modest Scope: The authors explicitly state that the absolute gain (~1 pp) is modest but consistent and reproducible. They do not claim TypedCSIP solves the entire problem of legislative review or that it is the only way to utilize $eB$. They also clarify that the method is scoped to the conflict-classification task and does not extend to retrieval or generation tasks.
Family-Level Gain: The results support the "TypedCSIP family" concept, where the typed pretraining signal transfers effectively to a fresh classification head, regardless of whether the Stage-2 replay mechanism is employed.