Technical Summary: FLIPS – Instance-Fingerprinting for LLMs via Pseudo-random Sequences

1. Problem Statement

Current Large Language Model (LLM) identification techniques, primarily designed for Intellectual Property Protection (IPP), focus on verifying if a target model is a copy of a reference model based on its original weights. These methods deliberately ignore "instance-level parameters"—such as system prompts, sampling configurations (e.g., temperature), and quantization—to achieve robustness against minor model alterations.

However, this approach creates a critical gap for AI regulation. Legislation like the EU AI Act and California's SB 533 requires compliance assessments based on the actual deployed behavior of an LLM, not just its provenance. A model with safe weights can generate toxic content under different configurations (e.g., specific system prompts or high temperatures). Existing IPP methods fail to distinguish between a safe model and its "abliterated" (uncensored) or differently configured variants, rendering them ineffective for regulators who need to identify specific deployed instances to verify behavioral compliance.

The paper defines Instance-level Fingerprinting (IF) as the task of distinguishing a specific model instance defined by its precise weights, instructional framing, and generation hyperparameters.

2. Methodology: FLIPS

The authors propose FLIPS (Instance-Fingerprinting for LLMs via Pseudo-random Sequences), a black-box, query-efficient method designed to detect instance-specific biases in pseudo-random sequence generation.

Core Rationale

The method exploits the observation that LLMs are poor at generating true random sequences. When prompted to generate binary sequences (e.g., alternating tokens like "0" and "1" or arbitrary pairs like "car" and "sun"), LLMs exhibit distinct, instance-specific biases. These biases arise from the complex interplay of weights, attention mechanisms, context, and hyperparameters (like temperature), which create a unique "blueprint" for each instance.

Technical Pipeline

1. Querying Strategy: The system uses a fixed prompt template instructing the model to generate a random binary sequence using a specific pair of tokens (e.g., $t_A$ and $t_B$). The authors utilize a set of distinct token pairs to elicit different bias signatures.
2. Feature Extraction: The generated output sequences are converted into binary bitstreams. These bitstreams are analyzed using the NIST Statistical Test Suite (a standard cryptographic test suite for randomness). The suite produces a vector of $d$ test statistics (e.g., frequency of transitions, run lengths, pattern occurrences) that serve as the fingerprint features.
3. Detector Training: The fingerprinting task is framed as a multi-class classification problem. An XGBoost classifier is trained on the NIST feature vectors extracted from a reference set of known instances.
4. Verification: To verify a target instance, the system queries it with the same token pairs, extracts NIST statistics, and feeds them to the trained classifier. The classifier predicts the most likely reference instance or labels the target as "Unseen" if the confidence falls below a predefined threshold.

Operational Constraints

FLIPS is designed to operate under strict black-box conditions (no access to weights or logits) and prioritizes query efficiency. It distinguishes between:

• Extraction Budget ($N_r$): Used to build the reference database (e.g., 40 queries per instance).
• Verification Budget ($N_t$): Used for routine checks (e.g., as few as 8 queries).

3. Key Contributions

1. Paradigm Shift: The paper introduces and motivates Instance-level Fingerprinting, shifting the focus from weight-based identification (IPP) to configuration-aware identification for regulatory compliance.
2. FLIPS Algorithm: A novel, query-efficient method that leverages biases in pseudo-random generation and NIST test statistics to distinguish between model instances.
3. Comprehensive Benchmark: The authors constructed a benchmark of 237 instances derived from 25 open-weight LLMs. This includes variations in temperature (0.4–1.0), system prompts (safety-oriented, harmful, persona-based), quantization (int4, fp8), and abliterated models.
4. Empirical Validation: The study demonstrates that FLIPS significantly outperforms adapted state-of-the-art IPP methods in distinguishing behavioral variants.

4. Experimental Results

Evaluated on the 237-instance benchmark, FLIPS achieved the following performance metrics:

• Closed-Set Accuracy: 96% with 8 verification queries ($N_t=8$).
• Open-Set Accuracy: 90% (where 10% of targets are unknown/unseen instances) with 8 verification queries.
• Baseline Comparison: The adapted state-of-the-art IPP method, LLMmap, achieved only 35% accuracy in the closed-set setting and was not evaluated in the open-set setting due to poor performance.
• Query Efficiency: High accuracy was achieved with as few as 40 extraction queries and 8 verification queries per instance.
• Robustness: The method remained effective even when sequence lengths were truncated to a fixed length, confirming that the signal comes from bias rather than raw length differences.
• Failure Analysis: The method showed lower recall for instances with higher temperatures (increasing output entropy) and specific "abliterated" models, though it still maintained significant discriminative power compared to baselines.

5. Significance and Claims

The paper claims that FLIPS provides a practically feasible solution for AI regulators to verify the compliance of deployed LLMs without requiring costly full audits or white-box access.

• Regulatory Necessity: The authors argue that because instance-level parameters (prompts, temperature) drastically alter model behavior (e.g., safety vs. toxicity), regulators must be able to identify specific configurations, not just model weights.
• License Plate Analogy: The authors liken FLIPS to a "license plate" for LLMs: a simple, efficient mechanism to identify an instance. While it can be forged (e.g., by a provider rerouting queries to a true PRNG), such deception is detectable (as true PRNGs pass NIST tests perfectly) and represents a higher barrier than the current blind spot in regulation.
• Limitations: The paper acknowledges that as LLMs improve their internal random generation capabilities, the discriminative power of this method may decrease. However, it suggests that the task can be escalated to more complex randomness challenges (e.g., Poisson processes) to maintain efficacy.

In conclusion, FLIPS demonstrates that exploiting subtle biases in pseudo-random generation allows for the effective, low-cost identification of specific LLM instances, addressing a critical gap in current AI governance frameworks.