Technical Summary: LineageMark

Problem Statement

In open Large Language Model (LLM) ecosystems, models frequently undergo multi-stage derivation chains where different contributors adapt a base model for specific tasks, domains, or applications. This creates complex lineage structures where a final derived model contains contributions from multiple historical stages. Current watermarking methods face two critical limitations in this context:

1. Lineage Decay: Historical watermarks embedded by early contributors often fade or become unextractable due to parameter drift caused by subsequent full-parameter fine-tuning and model updates.
2. Multi-User Interference: When new contributors attempt to insert their own watermarks into an already watermarked model, existing methods often fail to preserve the integrity of previous watermarks, or the new insertion disrupts the signals of prior contributors.

Existing white-box methods are primarily designed for single-user, one-time embedding and lack mechanisms to support incremental, multi-user verification where each contributor must independently verify their presence without access to the original model or other users' keys.

Methodology: LineageMark

LineageMark is a multi-user white-box watermarking framework designed to trace contributions in model derivation chains. It operates through three core phases:

1. Stable Carrier Selection

To mitigate the risk of watermark drift during subsequent derivations, LineageMark selects "stable" parameter coordinates for embedding.

• Functional Relevance: It computes an empirical hidden-space Fisher matrix ($F_\ell$) using calibration data to identify directions strongly associated with model behavior, avoiding inactive parameter regions.
• Perturbation Stability: It constructs a compression second-moment matrix ($C_\ell$) using perturbation operators (random projection, Gaussian quantization noise, structured dropout) to measure sensitivity to common model modifications.
• Selection Criterion: Stable directions are selected by solving a generalized eigenvalue problem ($F_\ell u = \lambda(C_\ell + \epsilon I)u$) to maximize functional relevance per unit of perturbation. A spectral truncation strategy selects eigenvectors within a specific range to balance stability and utility.
• Mapping: These stable subspaces are mapped back to specific weight coordinates in linear matrices, creating a mask ($\Omega$) of candidate carriers.

2. Sign-Projection Watermark Embedding

Instead of modifying individual weights to specific values, LineageMark encodes each watermark bit as a sign-projection statistic over a group of weights.

• Key-Driven Grouping: Using a private key, each contributor generates pseudo-random mappings to define disjoint groups of coordinates and projection coefficients ($\phi_{ij} \in \{-1, +1\}$).
• Redundancy: Watermark payloads are split into chunks and cyclically assigned across multiple linear matrices. This ensures that even if fine-tuning weakens the signal in some matrices, the aggregate signal remains recoverable.
• Margin Constraints: The embedding process enforces a decision margin ($\mu$). If the current projection statistic does not satisfy the target sign with sufficient margin, a local closed-form update is applied to the stable coordinates to push the statistic toward the target. This creates a buffer against future parameter perturbations.

3. Key-Driven Watermark Extraction

Verification is performed independently by each candidate contributor without requiring the original model, calibration data, or other users' information.

• Detector Reconstruction: The verifier uses their private key to reconstruct the exact same coordinate groups and projection coefficients used during embedding.
• Weighted Voting: The verifier computes projection statistics on the current model parameters. Instead of simple majority voting, LineageMark uses the magnitude of the projection as a confidence weight.
• Aggregation: Local decisions from multiple matrices are aggregated. A bit is decoded based on whether the accumulated positive or negative weighted votes dominate. This mechanism allows the system to recover bits even if some matrices have been corrupted by later derivation stages.

Key Contributions

1. Formalization of Multi-User Watermarking: The paper defines the requirements for model derivation chains, specifically lineage preservation (historical watermarks remain extractable), incremental embedding (new watermarks do not destroy old ones), and key-only independent verification.
2. Stable Carrier Selection: A novel method using Fisher information and perturbation sensitivity analysis to identify parameter regions robust to full-parameter fine-tuning and compression.
3. Margin-Constrained Sign-Projection: An embedding strategy that represents bits as aggregate projection signs with enforced margins, enhancing robustness against parameter drift and multi-user interference.
4. Comprehensive Evaluation: The framework is evaluated across multiple model scales (OPT-125M to OPT-1.3B) and domains (medical, legal, mathematical reasoning), demonstrating robustness against fine-tuning, re-watermarking, quantization, and pruning.

Experimental Results

• Lineage Preservation: In multi-stage derivation chains involving full-parameter fine-tuning, LineageMark achieved 100% Watermark Extraction Rate (WER) for all historical watermarks across all stages and domains. In contrast, baseline methods (ELLMark, EmMark) showed significant decay (WER dropping to ~40-60% for historical marks) and interference issues.
• Incremental Embedding: New contributors could successfully embed watermarks without degrading the extractability of existing watermarks.
• Robustness:
  • Fine-tuning: Maintained 100% WER against parameter-efficient fine-tuning attacks (LoRA, Prefix-tuning, etc.) and standard full-parameter fine-tuning.
  • Quantization: Achieved 100% WER under 8-bit quantization and >90% WER under aggressive 4-bit quantization.
  • Pruning: Maintained >96% WER even with 60% random pruning.
• False Positive Resistance: LineageMark demonstrated a large gap between correct-key and invalid-key extraction scores. With a verification threshold of 0.75, it achieved a 0.50% False Positive Rate (FPR) and 0% False Negative Rate (FNR), significantly outperforming baselines which suffered from high FPRs (e.g., 88% for ELLMark) due to ambiguous extraction scores.
• Utility Preservation: Zero-shot performance on general benchmarks (PIQA, HellaSwag, WinoGrande) remained stable, with negligible degradation compared to the original model.

Significance and Claims

The paper claims that LineageMark addresses a critical gap in the open LLM ecosystem by enabling verifiable multi-contributor provenance. Unlike previous methods that focus on single-owner verification, LineageMark allows a derived model to serve as a digital asset where the contributions of multiple historical contributors can be independently audited.

The authors emphasize that the framework does not aim to recover the complete derivation topology or the chronological order of insertion, but rather to verify the presence of specific contributor watermarks in a downstream model. The work demonstrates that extending watermarking from single-owner scenarios to multi-user derivation chains is feasible through stable carrier selection and statistical projection encoding, providing a practical mechanism for intellectual property protection and contribution tracing in evolving model ecosystems.

The paper also modestly acknowledges limitations, noting that the preprocessing cost (Fisher information estimation) is the primary computational bottleneck and that the method is not immune to sufficiently aggressive fine-tuning (e.g., very high learning rates) or extreme compression that exceeds the designed stability margins.