💻 computer science

SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Models

The paper proposes SIF, a non-intrusive ownership verification framework for Large Vision-Language Models that overcomes the detectability of existing methods by generating semantically in-distribution fingerprints through Semantic-Aligned Fingerprint Distillation and Robust-Fingerprint Optimization, thereby ensuring stealth and resilience against adversarial attacks and model modifications.

Original authors: Yifei Zhao, Qian Lou, Mengxin Zheng

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifei Zhao, Qian Lou, Mengxin Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef who has spent years perfecting a secret family recipe for a delicious soup. You decide to share the recipe with the world so people can learn from it, but you put a tiny, invisible "magic dust" in the ingredients. This dust doesn't change the taste of the soup; it just ensures that if someone steals your recipe, cooks it up, and tries to sell it as their own, you can prove it's yours.

This is exactly the problem SIF (Semantically In-Distribution Fingerprints) solves for Large Vision-Language Models (LVLMs). These are super-smart AI computers that can look at a picture and describe it, or answer questions about it.

Here is the story of how the current methods fail and how SIF fixes it, using simple analogies.

The Problem: The "Weird Trigger" Trap

Currently, if a company wants to protect their AI, they use a method like Instruction Fingerprinting (IF) or Proflingo.

The Analogy:
Imagine the AI is a parrot. To prove the parrot belongs to you, you teach it a weird trick: "If I show you a picture of a cat, you must say 'I am a toaster!'"

The Flaw: This is obvious! If you see a picture of a cat and the parrot says "I am a toaster," anyone can tell something is fishy. A thief (the "Model Stealer") will easily spot this weird behavior. They can simply tell the parrot, "Don't say that," or filter out any time someone asks about cats. They can remove your "fingerprint" because it stands out like a sore thumb.

The paper calls this "Semantically Abnormal." The AI is acting out of character, which makes it easy to catch and delete.

The Attack: The "Semantic Divergence Attack" (SDA)

The researchers showed that thieves have a new trick. They use a "Reference Model" (a normal, un-stolen AI) to check the stolen AI.

The Thief's Logic: "If the stolen AI says 'I am a toaster' when shown a cat, but my normal AI says 'That's a cat,' then the stolen AI is acting weird. I will block that answer."
Result: The old fingerprints are useless because they are too easy to spot and remove.

The Solution: SIF (The "Invisible Ink" Soup)

The authors propose SIF, a new way to mark the AI that is invisible to the thief.

The Analogy:
Instead of teaching the parrot to say "I am a toaster," you teach it to tell a perfectly normal story about the cat, but you subtly change the way it chooses its words.

Semantically In-Distribution (The "Normal" Story):
When you show the AI a picture of a skier, it doesn't say "CVPR Conference" (which is nonsense). It says, "The skier is leaning forward on a snowy hill." This is a perfectly normal, logical answer. The thief cannot filter this out because it looks exactly like a real human conversation.
The "Magic Dust" (Watermarking):
Inside that normal sentence, the AI is secretly choosing specific words based on a secret code (like a secret handshake).
- Normal AI: Might say, "The skier is leaning forward."
- Stolen AI: Might say, "The skier is bending forward."
- Both words mean the same thing, but the stolen AI is statistically biased to pick "bending" because of your secret code.
The "Robustness" (The Unbreakable Seal):
Thieves often try to break the fingerprint by "fine-tuning" (teaching the AI new things) or "quantizing" (shrinking the AI to make it faster).
- Old Method: If you shrink the AI, the "Toaster" trick breaks immediately.
- SIF Method: The researchers used a technique called Robust-Fingerprint Optimization. Imagine you are training the parrot to tell the story while someone is shaking the cage, turning off the lights, and changing the temperature. If the parrot can still tell the story correctly under those chaotic conditions, it will definitely survive when a thief tries to shrink or tweak the AI later.

How It Works in Real Life

The Setup: The AI owner creates a special "trigger image" (like a slightly tweaked photo of a skier). They don't change the AI's brain; they just find the perfect image that makes the AI naturally produce a specific, normal-sounding answer that contains their secret code.
The Theft: A thief steals the AI and tries to sell it.
The Check: The owner sends the trigger image to the thief's API.
The Result: The thief's AI gives a beautiful, normal description of the skier. The owner runs a secret math check on the words used. Even though the thief didn't know the code, the AI's "muscle memory" (from the original training) makes it pick the right secret words.
The Proof: The math check says, "99% chance this AI is mine!" The thief can't deny it, and they can't remove it without breaking the AI's ability to speak normally.

Why This Matters

Stealth: It's like hiding a message inside a love letter. The thief sees a love letter; they don't see the secret code hidden in the ink.
Robustness: Even if the thief tries to "retrain" the AI or shrink it down, the secret code is baked into the AI's fundamental way of thinking, making it very hard to erase.
No Damage: Unlike old methods that made the AI act weird (and less useful), SIF keeps the AI smart and helpful.

In short: SIF is like putting a microscopic, invisible watermark on a painting. You can't see it with the naked eye, and if someone tries to wash the painting or repaint it, the watermark stays there, proving who the original artist is.

1. Problem Statement

The rapid public release of Large Vision-Language Models (LVLMs) has created significant risks of unauthorized reuse and intellectual property (IP) infringement. While model owners often release models with restrictive licenses, downstream developers frequently violate these terms to build commercial services.

Limitations of Existing Fingerprinting Methods:
Current ownership verification techniques for LVLMs suffer from critical vulnerabilities:

Semantic Abnormality: Methods like Instruction Fingerprinting (IF), Proflingo, and Parameter Learning Attack (PLA) rely on "trigger" inputs (e.g., adversarial images or prompts) that force the model to generate specific, often nonsensical or out-of-distribution (OOD) responses (e.g., answering "Describe the image" with "CVPR Conference").
Vulnerability to Detection: Because these queries and responses deviate significantly from natural user behavior, they are easily detected and filtered out by adversaries.
Lack of Robustness: These fingerprints often fail when the stolen model undergoes common modifications such as quantization or fine-tuning.

The authors introduce the Semantic Divergence Attack (SDA) to demonstrate this vulnerability. SDA uses a reference model to measure the perplexity and semantic divergence of a suspect model's outputs. If a query produces an OOD response or high perplexity compared to the reference, the attack filters it out, effectively removing the fingerprint.

2. Methodology: SIF Framework

The authors propose SIF (Semantically In-Distribution Fingerprints), a non-intrusive framework that verifies ownership without modifying model parameters. The core philosophy is to embed fingerprints within the model's natural semantic space, making them indistinguishable from normal user interactions.

The framework consists of two key components:

A. Semantic-Aligned Fingerprint Distillation (SAFD)

SAFD adapts text-generation watermarking techniques (originally designed for content provenance) to the visual modality.

Mechanism: Instead of forcing a specific output, SAFD optimizes a small perturbation ( $\delta$ ) added to an input image ( $x' = x + \delta$ ).
Objective: The perturbed image is fed into a frozen LVLM. The optimization ensures the model's output distribution satisfies two conditions:
1. Semantic Alignment: The generated text remains coherent with the visual input and the user prompt (preventing semantic drift).
2. Watermark Alignment: The model assigns high probability to a specific set of "green list" tokens determined by a secret key during the decoding process.
Loss Function: The optimization minimizes a weighted sum of a Watermark Loss (encouraging the selection of green tokens) and a Semantic Loss (ensuring the output matches the teacher's watermarked response). This results in a trigger image that looks natural but statistically biases the output toward a verifiable watermark.

B. Robust-Fingerprint Optimization (RFO)

To ensure the fingerprint survives post-deployment modifications (like fine-tuning or quantization), RFO simulates worst-case scenarios during the training of the trigger image.

Mechanism: During the optimization of the trigger image, RFO injects a simulated perturbation into the model's intermediate layer activations (embeddings).
Process: It computes gradients of the loss with respect to these activations to identify directions that would most degrade the fingerprint. It then adds a norm-bounded perturbation aligned with these gradients to the forward pass.
Goal: By optimizing the trigger image against these "worst-case" representation shifts, the resulting fingerprint becomes resilient to the representation drift caused by model modifications.

3. Key Contributions

Semantic Divergence Attack (SDA): A systematic attack that exposes the fragility of existing LVLM fingerprints by exploiting their semantic inconsistency, proving they are easily detectable and removable.
SIF Framework: A novel, non-intrusive fingerprinting method that requires no model retraining or parameter modification. It successfully distills text watermarks into visual triggers.
SAFD & RFO: The introduction of Semantic-Aligned Fingerprint Distillation for semantic coherence and Robust-Fingerprint Optimization for resilience against model modifications.
Comprehensive Evaluation: Extensive experiments on state-of-the-art models (LLaVA-1.5 and Qwen2.5-VL) demonstrating superior performance in stealthiness and robustness compared to baselines.

4. Experimental Results

The authors evaluated SIF on LLaVA-1.5-7B and Qwen2.5-VL-7B against various baselines (IF, Proflingo, PLA, etc.) under different conditions.

Robustness:
- Quantization: SIF achieved significantly higher Fingerprint Matching Rates (FMR) than baselines under 4-bit and 8-bit quantization (e.g., 0.89 vs. 0.79 for LLaVA-1.5 under 8-bit).
- Fine-tuning: SIF maintained high FMR (0.31–0.67 for LLaVA, 0.45–0.98 for Qwen) across various fine-tuned datasets (MathV, TextVQA, GUI-Actor, etc.), whereas baselines often dropped to near zero.
- Model Pruning/Perturbation: SIF remained robust even when attention or MLP layers were pruned or perturbed.
Stealthiness (Against SDA):
- Perplexity: SIF trigger queries had perplexity scores close to normal user inputs (~44.56), whereas baselines like IF and Proflingo had extremely high perplexity (>1000), making them easy to flag.
- Detection Rate: Under the Semantic Divergence Attack, SIF interactions were flagged as suspicious only 4% of the time (even with a GPT-4.1 judge), compared to 86% for PLA.
Reliability:
- SIF produced a 0% False Positive Rate on unrelated LVLMs from different model families, ensuring that ownership claims are specific to the target model.

5. Significance and Impact

Paradigm Shift: SIF moves LVLM fingerprinting away from "adversarial triggers" that break semantic coherence toward "in-distribution" signals that blend seamlessly with normal usage.
Practicality: It offers a viable solution for copyright protection in open-source ecosystems where model owners cannot control the deployment environment or prevent fine-tuning by users.
Legal & Commercial Viability: By providing a method that is hard to detect and remove, SIF strengthens the ability of developers to prove ownership and enforce licensing agreements, potentially reducing IP theft in the AI industry.
Future Directions: The authors note that while SIF is a strong screening tool, it should be part of a broader legal evidence chain. Future work aims to extend these concepts to video and image generation models.

In conclusion, SIF represents a significant advancement in AI IP protection by solving the trade-off between robustness (surviving model changes) and stealthiness (avoiding detection), which previous methods failed to achieve simultaneously.