🤖 AI

Functional Subspace Watermarking for Large Language Models

This paper proposes Functional Subspace Watermarking (FSW), a robust framework that anchors ownership signals into a stable, low-dimensional functional backbone of Large Language Models to ensure reliable detection against parameter-level perturbations like fine-tuning and quantization while preserving model utility.

Original authors: Zikang Ding, Junhao Li, Suling Wu, Junchi Yao, Hongbo Liu, Lijie Hu

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zikang Ding, Junhao Li, Suling Wu, Junchi Yao, Hongbo Liu, Lijie Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you own a incredibly talented, custom-built robot chef. You've spent years training it to make the perfect soufflé. Now, you want to sell this robot, but you're worried: what if someone buys it, tweaks the recipe slightly, changes the robot's internal wiring, or even copies its "brain" into a cheaper, smaller robot, and then claims they invented the soufflé?

This is the problem Functional Subspace Watermarking (FSW) tries to solve for Large Language Models (AI chatbots).

Here is the simple breakdown of how this paper works, using everyday analogies.

The Problem: The "Fragile Tattoo"

Currently, if you want to prove an AI belongs to you, you might try to give it a "digital tattoo" (a watermark).

The Old Way: Imagine tattooing a tiny dot on the robot's finger. If the owner of the robot decides to shrink the robot (quantization), cut off a finger (pruning), or retrain the robot on new data (fine-tuning), that tattoo might get erased or distorted. The thief can say, "See? No tattoo. This is my robot."
The Challenge: AI models are huge and complex. Changing their settings to make them faster or smaller often accidentally wipes out these simple watermarks.

The Solution: The "Soul of the Machine"

The authors of this paper propose a smarter way. Instead of tattooing the finger, they tattoo the robot's "soul"—the specific part of its brain that actually makes it good at cooking.

They call this the Functional Subspace. Think of it like this:

Every AI has millions of internal settings (parameters).
Most of these settings are just "noise" or minor adjustments.
But a tiny, specific group of settings is critical. If you mess with these, the robot stops making soufflés and starts making mud pies.
The Insight: Because these critical settings are so important to the robot's job, the robot cannot change them without losing its ability to work. Therefore, these settings are the most stable place to hide a watermark.

How It Works (The 3-Step Recipe)

1. Finding the "Golden Thread" (The Math Part)

The researchers use a special mathematical tool (called a Generalized Eigenvalue Problem) to find that tiny, stable group of settings.

Analogy: Imagine a spinning top. If you push it hard, it wobbles. But there is one specific axis it spins around that never changes, no matter how hard you push. The researchers find that "axis" in the AI's brain. They know that if the AI tries to change this axis, it will break its own ability to answer questions. So, this axis is safe to use for a watermark.

2. The "Goldilocks" Filter (Adaptive Truncation)

They don't just pick any stable part; they have to be careful.

Too sensitive: If they pick a part that changes too easily, the watermark disappears when the AI is tweaked.
Too dull: If they pick a part that doesn't matter, the AI can easily change it to hide the watermark.
The Fix: They use a "Goldilocks" strategy. They pick the "just right" settings—stable enough to survive changes, but important enough that the AI can't ignore them.

3. The "Ghost in the Machine" (Embedding)

Once they find this stable "soul," they inject a secret code (the watermark) into it.

The Safety Net: They add a rule: "You can change the code, but you must promise the robot still tastes the same." This ensures that while the secret code is hidden inside, the robot's ability to write stories or solve math problems doesn't get worse.

Why This is a Game-Changer

The paper tested this against the "thieves" (attacks):

Fine-tuning: The thief retrained the AI on new data. Result: The watermark survived.
Quantization: The thief compressed the AI to make it smaller (like turning a high-res photo into a low-res JPEG). Result: The watermark survived.
Distillation: The thief copied the AI's knowledge into a smaller, cheaper model. Result: The watermark survived.

Even when the AI was heavily modified, the "soul" remained intact, and the secret code was still there, proving the original owner's rights.

The Bottom Line

Think of this method as hiding a watermark in the foundation of a house instead of painting it on the front door.

If a burglar paints over the door (changes the surface), the watermark is gone.
But if the burglar tries to move the foundation (change the core logic), the house collapses. Since they need the house to stand, they can't touch the foundation.
Therefore, the watermark in the foundation is safe forever.

This paper gives AI creators a way to prove, "This AI is mine," even if someone tries to disguise it, shrink it, or copy it. It's a robust, invisible shield for intellectual property in the age of AI.

1. Problem Statement

The rapid proliferation of Large Language Models (LLMs) has intensified concerns regarding model ownership and provenance. While content watermarking (embedding signals in generated text) exists, it is vulnerable to distillation attacks where a student model reproduces capabilities without preserving the output-level signals.

Model watermarking, which embeds ownership into model weights or internal representations, faces a critical challenge: robustness against parameter-level perturbations. Existing methods often fail when the model undergoes realistic modifications such as:

Fine-tuning (SFT/LoRA): Adjusting weights for specific tasks.
Quantization: Reducing precision (e.g., INT4/INT8).
Knowledge Distillation: Compressing a large model into a smaller one.
Pruning: Removing neurons or connections.

These modifications inevitably distort internal representations, causing conventional watermarks to be erased or rendered undetectable. The core problem is identifying a representation subspace that is critical for task performance (to prevent removal) yet stable under compression (to survive modifications).

2. Methodology: Functional Subspace Watermarking (FSW)

The authors propose FSW, a framework that anchors ownership signals into a low-dimensional "functional backbone" of the model. The methodology follows a structured pipeline:

A. Representation Extraction

The system extracts hidden state representations ( $r(x)$ ) from a fixed intermediate layer of the LLM. This layer acts as an information bottleneck for next-token prediction, making it an optimal carrier for semantic-level watermarking.

B. Geometry Analysis & Subspace Construction

To find the stable subspace, FSW analyzes two competing properties of representation directions:

Functional Sensitivity: Measured via the Fisher Information Matrix ( $F$ ). Directions with high sensitivity encode core knowledge; perturbing them significantly alters output distribution, making them resistant to pruning.
Compression Invariance: Measured via a Compression Invariance Matrix ( $C_A$ ). This matrix models the variance introduced by compression operators (Linear Projection, Quantization Noise, Structural Dropout). Directions with low variance in $C_A$ remain stable under compression.

C. Generalized Eigenvalue Problem (GEVP)

FSW formulates the search for the optimal subspace as a GEVP:
$F u_i = \lambda_i C_A u_i$
The generalized eigenvalue $\lambda_i$ represents the robustness score of a direction. High $\lambda_i$ indicates a direction that is both critical for the task (high $F$ ) and stable under compression (low $C_A$ ).

D. Adaptive Spectral Truncation

Simply selecting the top- $k$ eigenvalues can degrade model utility, while selecting low ones leads to easy removal. FSW introduces an adaptive spectral truncation strategy to select a specific range of eigenvalues ( $\tau_{lower} \cdot \lambda_1 \le \lambda_i \le \tau_{upper} \cdot \lambda_1$ ) that balances robustness and utility.

E. Watermark Embedding & Verification

Anchored Projection: The model projects representations into the identified subspace $U^*$ relative to a global mean.
Vector Consistency Constraint: A loss function ( $L_{con}$ ) ensures that the watermarked model's projections remain close to the original model's, preserving semantic performance.
Embedding: A margin-based objective injects orthogonal secret keys into the subspace.
Verification: Ownership is verified via a statistical test on an aggregated detection score ( $S$ ), calculated over a challenge set.

3. Key Contributions

FSW Framework: A novel method that embeds watermarks into a functional backbone identified via GEVP, explicitly decoupling task-critical sensitivity from compression-induced variance.
Adaptive Spectral Truncation: A strategy to select the optimal subspace dimensions, avoiding the utility collapse associated with naive top- $k$ selection.
Vector Consistency Constraint: Ensures watermark injection does not compromise the model's original generation capabilities.
Statistical Verifiability: Transforms ownership verification into a rigorous statistical test with a provably low False Positive Rate (FPR).

4. Experimental Results

The authors evaluated FSW across diverse architectures (LLaMA-2/3, Qwen2.5, Mistral, DeepSeek) and datasets.

Functional Preservation: FSW maintains high task performance. For example, on LLaMA-3-8B, the perplexity (PPL) increase was only +0.37, and accuracy remained competitive with clean fine-tuning.
Robustness to Attacks:
- Distillation: Even after "Backbone Distillation," FSW maintained high bit accuracy (e.g., 100% on DeepSeek-7B-Chat).
- Quantization & Pruning: Detection scores often increased or remained stable after INT4 quantization and magnitude pruning.
- LoRA Fine-tuning: The watermark survived parameter-efficient adaptation with minimal degradation in detection scores.
Statistical Significance: The method achieved near-perfect bit accuracy (100%) and high AUC (up to 1.000) across models. The detection margin remained significant even at extreme significance levels ( $\alpha = 10^{-8}$ ), effectively rejecting the null hypothesis of random noise.
Ablation Studies: Removing components like "Adaptive Thresholding" or "Anchored Projection" caused catastrophic drops in detection accuracy (e.g., dropping to 12.5% bit accuracy), proving the necessity of the full framework.

5. Significance

This paper addresses a critical gap in LLM security: the vulnerability of internal watermarks to model compression and distillation. By mathematically identifying a stable functional subspace, FSW provides a robust mechanism for ownership verification that survives the most aggressive model modifications.

Practical Impact: It offers a viable solution for model developers to protect intellectual property in an era where models are frequently fine-tuned, quantized, and distilled for deployment.
Theoretical Advance: The use of GEVP to balance functional sensitivity and compression invariance provides a new theoretical foundation for robust watermarking, moving beyond heuristic weight modifications.
Reliability: The rigorous statistical verification protocol ensures that ownership claims can be defended in legal or commercial disputes with high confidence and low false positives.