Technical Summary: SLAM (Structural Linguistic Activation Marking)

Problem Statement

The widespread deployment of Large Language Models (LLMs) necessitates reliable provenance attribution to distinguish AI-generated text from human-authored content. Current watermarking schemes face a fundamental trade-off between detectability, text quality, and robustness.

Dominant paradigms (e.g., KGW, EWD, Unigram) operate by biasing the model's next-token distribution (e.g., via green-list token selection). While efficient, this approach incurs a measurable quality cost: on Gemma-2 models, these methods lose 7.5–11.5 reward points and exhibit collapsed conditional perplexity ratios (0.30–0.42), indicating repetitive and unnatural output. Conversely, recent sparse autoencoder (SAE) based methods like SAEMark achieve near-lossless quality but suffer from high computational overhead (sampling $N=50$ candidates per step) and vulnerability to semantic shifts and simple text edits.

Methodology: SLAM

SLAM (Structural Linguistic Activation Marking) proposes a white-box watermarking scheme that decouples detection from token-frequency bias. Instead of altering which tokens are sampled, SLAM writes the watermark into the structural geometry of the model's residual stream.

Core Mechanism

1. Structural Feature Mining:

   • The authors construct a contrastive dataset of 46,579 sentence pairs covering 104 linguistic phenomena (syntactic alternations, tense, morphology, discourse). These pairs hold semantic content constant while varying structural forms (e.g., active vs. passive voice).
   • Using Sparse Autoencoders (SAEs) trained on the residual stream, the method identifies specific, localized subspaces that encode these structural features.
   • A composite scoring mechanism (contrastive magnitude $\times$ purity $\times$ cross-domain consistency) isolates SAE directions that are robustly responsive to structural changes rather than semantic content.

2. Direction Construction (SVD):

   • For phenomena where structural concepts are distributed across multiple weakly-discriminative features (common in larger models like Gemma-2 9B), SLAM performs Singular Value Decomposition (SVD) on the contrastive difference matrix.
   • This yields $k$ orthogonal structural modes. The method supports both single-feature steering ($k=1$) and composite subspace steering ($k>1$), allowing it to scale effectively to larger models.
   • Bidirectional Mining: The system mines directions for both the target structure and its inverse, doubling the pool of steerable directions to reduce collision rates during selection.

3. Causal Steering and Injection:

   • At generation time, SLAM hooks into the residual stream at specific layers (determined by the phenomenon's peak activation layer).
   • It adds a scaled vector ($\alpha \cdot v_{structure}$) to the residual stream of newly generated tokens. This biases the model toward producing a specific syntactic variant (e.g., passive voice) without constraining the lexical choice of individual tokens.
   • Feature Selection: A per-document subset of features is selected via HMAC-keyed weighted sampling to ensure uniqueness and security.
   • Early-Stop Generation: The system generates up to $N=4$ candidates and returns the first that clears a calibrated detection threshold ($\hat{z} \ge 2.0$), significantly reducing overhead compared to candidate-scoring methods.

4. Detection:

   • Detection requires a single forward pass of the model. The residual stream activations are projected onto the pre-computed structural directions to compute a Stouffer $z$-score.
   • No SAE forward pass is required at detection time, making the process computationally efficient.

Key Contributions

1. Contrastive Mining Pipeline: A pipeline isolating structural SAE directions via a composite score across 104 phenomena and 5 semantic domains, ensuring domain invariance.
2. PCA-Bidirectional Structural Directions: An SVD-based method to construct orthogonal composite directions ($k>1$), enabling SLAM to handle distributed structural representations in larger models (9B+) and offering a tunable quality-detection trade-off.
3. Efficient White-Box Scheme: A complete watermarking system with HMAC-keyed selection and an early-stop generation loop. It requires an order of magnitude fewer LLM calls than SAEMark ($N \le 4$ vs. $N=50$) and operates with a modest overhead ($\sim 1.6\times$ generation time).
4. Empirical Validation: Comprehensive evaluation on Gemma-2 2B and 9B against six baselines across four quality metric families and seven robustness attacks.

Experimental Results

Evaluated on Gemma-2 2B and 9B pre-trained models:

• Detection Performance: SLAM achieves 100% True Positive Rate (TPR) at a 2.3% False Positive Rate (FPR) on both model sizes.
• Quality Preservation:
  • Reward Gap: SLAM incurs a reward penalty of only $\Delta$Reward = -1.3 (2B) and -1.9 (9B). This is significantly lower than token-distribution baselines (KGW, EWD, Unigram), which lose 7.5–11.5 points.
  • Naturalness: SLAM preserves output distribution metrics (distinct-n, Self-BLEU, MAUVE) and conditional perplexity ratios (1.24/1.36) much closer to the unwatermarked baseline than other methods. Unlike token-distribution methods, it does not induce repetitive text (PPL ratio collapse).
  • Grammar: Grammar error rates remain near baseline levels.
• Robustness Profile:
  • Word-Level Attacks: SLAM is 100% robust to synonym substitution, word deletion, word substitution, and sentence reordering.
  • Paraphrase Attacks: SLAM is vulnerable to syntax-restructuring paraphrase (e.g., DIPPER), where TPR drops to ~10–12%. However, the paper notes that such attacks impose a quality cost on the attacker (degrading the text), and this vulnerability is the converse of token-distribution methods (which are robust to paraphrase but vulnerable to word-level edits).

Significance and Claims

The paper claims that SLAM demonstrates that quality and detection need not trade off when the watermark is encoded at the appropriate level of linguistic abstraction (structural geometry vs. token frequencies).

• Mechanistic Insight: By steering which structural variant the model produces rather than which tokens it samples, SLAM leaves the lexical surface and semantic meaning essentially unconstrained. This explains its superior performance on reward models, perplexity, and diversity metrics.
• Complementary Robustness: The paper positions SLAM's vulnerability to syntax-restructuring paraphrase as a calculated, acceptable cost for achieving near-lossless generation quality, contrasting it with the "repetitive text" signature of token-biasing methods.
• Scalability: The use of composite subspaces ($k>1$) is identified as the critical enabler for scaling structural watermarking to larger models where features are distributed.

The authors conclude that while SLAM requires white-box access and is sensitive to severe structural rewriting, it offers a superior balance of detectability and quality preservation for provenance attribution compared to existing token-distribution and semantic-scoring approaches.