🤖 machine learning

The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models

This paper introduces a lightweight, training-free audit method that detects prefix invariance violations in attention, state-space, and hybrid models by identifying causality leaks that standard attention-mask inspections fail to catch.

Original authors: Taebong Kim, Youngsik Hong, Minsik Kim, Sunyoung Choi, Jaewon Jang, Minseo Kim

Published 2026-08-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Taebong Kim, Youngsik Hong, Minsik Kim, Sunyoung Choi, Jaewon Jang, Minseo Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence systems that generate text, speech, or images often rely on a fundamental rule: they must process information in a strict, one-way timeline. Imagine reading a book where you are only allowed to see the words on the current page and everything before it, but never the pages that come after. This rule, known as autoregression, is the backbone of how these models learn and function. If a model accidentally glances at a future word while trying to predict the current one, it gains an unfair advantage, making its performance look artificially high during testing while failing in real-world use. For years, the scientific community assumed that checking the "mask"—a digital barrier meant to block future information—was enough to guarantee this rule was being followed. However, a new study from researchers at VIDRAFT AI Research and QuantumOS in Seoul suggests this assumption is dangerously incomplete. They have developed a simple, rigorous test that proves specific released models are leaking information from the future, and they have pinpointed exactly where the breakdown happens.

The researchers began by formalizing a concept they call "prefix invariance." In plain terms, this means that the internal representation of a word at any given moment should depend only on the words that came before it. If you change a word at the very end of a sentence, the internal state of the model for the first few words should not change at all. If it does change, the model is using future knowledge to influence the past. To test this, the team created a lightweight audit that requires no training, no complex gradients, and no special hardware. The process is straightforward: they feed the model a sequence of random tokens, record the internal state of every layer, and then run the exact same sequence again with just the final token changed. By comparing the two runs, they can see if the earlier parts of the sequence shifted in response to the change at the end. If the earlier parts remain perfectly identical, the model is honest. If they shift, even slightly, the model has a causal leak.

What makes this discovery significant is that the researchers found this method catches errors that traditional checks completely miss. In the past, engineers would inspect the attention masks—the digital barriers mentioned earlier—to ensure causality. The new study shows that this inspection is insufficient because leaks can occur through other mechanisms, such as how data is scanned or normalized, even when the mask itself is correct. To prove this, the team injected 192 different types of "faults" into eight different model checkpoints. These faults were designed to simulate common coding errors that would allow future information to leak backward. While the traditional mask inspection failed to find a single one of these 192 faults, the new audit detected and localized every single one, identifying the exact layer where the leak began. This ability to pinpoint the specific location of the error is crucial, as it allows engineers to fix the code rather than just knowing a problem exists.

The team did not stop at testing artificial faults; they applied their method to real, publicly available models to see if the issue existed in the wild. Their investigation uncovered a serious, structural defect in two specific hybrid models: Zamba2 and Nemotron-H. These models use a specific technique called a "chunked scan" to process long sequences of data efficiently. The researchers discovered that the code responsible for linking these chunks together was oriented incorrectly. Instead of ensuring that each chunk only looked at previous chunks, the code allowed information from future chunks to bleed into the current one. This defect was invisible when the models were tested on short sequences, but as soon as the sequence length exceeded a specific threshold—the size of a single "chunk"—the leak appeared. For the Zamba2 model, this threshold was 256 tokens; for Nemotron-H, it was 128. Once the sequence crossed these boundaries, the model began to contaminate its earlier predictions with information from the future.

The researchers were able to trace this error to a specific three-line block of code that was identical in both models, suggesting a shared lineage in their development. By correcting the orientation of the data reduction in that code block, they were able to eliminate the leak entirely, driving the contamination down to exactly zero. This confirmed that the defect was not a random glitch or a result of the model's training, but a fundamental error in how the code was written. The study also highlighted a broader lesson for the field: the length of the test sequence matters immensely. If a test is shorter than the model's internal processing window, the specific code paths that contain these defects are never exercised, and the model will appear clean even when it is not. The researchers found that their own initial census of models, which used short sequences, would have missed these defects entirely had they not extended the test length.

Crucially, the study also revealed that the audit tool itself can fail if the models do not load correctly. In three specific cases involving the Falcon-H1 hybrid family, the test returned a "clean" verdict because the models failed to respond to input due to loading errors, effectively rendering the audit instrument inert. The researchers realized that a "clean" result is meaningless unless the testing tool is proven to be active on that specific checkpoint. This lesson was reinforced when the team's own 7B model release initially could not be audited due to a device-placement conflict during loading. Only after resolving these technical hurdles and confirming the models were responsive did the audit proceed, ensuring that the results reflected actual model behavior rather than a silent system failure.

This work establishes a new standard for verifying the integrity of sequence models. The authors argue that just as model releases routinely report parameter counts and benchmark scores, they should also include a "causal correctness certificate." This certificate would detail the results of the prefix invariance test, including the specific sequence length used, the precision of the calculation, and proof that the testing instrument was working correctly. The study concludes that relying solely on attention masks is no longer sufficient for the complex, hybrid architectures being built today. By using a simple, two-pass check that compares internal states, developers can now detect and locate these subtle leaks, ensuring that the models they build and deploy truly respect the timeline of the information they process. The findings serve as a reminder that in the rapidly evolving landscape of artificial intelligence, the most critical safeguards are often the ones that are easiest to overlook.

Technical Summary: The Mask Is Not the Model

1. Problem Statement

Autoregressive language models rely on a fundamental structural constraint: the representation at any position $t$ must depend only on inputs at positions $\le t$ . This property, termed prefix invariance, is silently assumed by all practical applications, including training, perplexity evaluation, and speculative decoding.

However, modern sequence models increasingly utilize heterogeneous architectures combining attention, state-space models (SSMs), linear recurrences, and convolutions. In these systems, the traditional method of auditing causality—inspecting the explicit attention mask—is incomplete. Masks govern only a subset of layers or may be absent entirely (e.g., in pure SSMs or recurrent layers). Consequently, "causal leakage" (where future information influences past representations) can occur via scans, normalization, or aggregation steps without violating any mask, leading to artificially improved training metrics while causing failures during free-running generation.

2. Methodology

The authors propose a lightweight, gradient-free audit based on metamorphic testing. The core hypothesis is that if a model is causally correct, the representations of a prefix must remain invariant under perturbations to future tokens.

The Audit Procedure

The method requires only two forward passes and no training or gradients:

Input Construction: Create two sequences, $x^1$ and $x^2$ , of length $T$ . They are identical except for the token at the final position ( $T-1$ ).
Forward Passes: Run both sequences through the model with caching disabled to ensure structural identity.
Layer-wise Comparison: For every layer $\ell$ $ℓ$ , compute the maximum absolute difference ( $\Delta_\ell$ $Δ_{ℓ}$ ) between the representations of the two sequences over the prefix positions $[0, T-1)$ $[0, T - 1)$ .
- Note: The differing position $T-1$ is excluded from the comparison.
Verdict:
- If $\Delta_\ell = 0$ (within numerical precision) for all layers, the model is CLEAN.
- If $\Delta_\ell > \tau$ (threshold) for any layer, the model has a LEAK. The first such layer $\ell^*$ is reported as the source of the violation.

Design Choices

Per-layer Hooks: Unlike output-logit checks, this method captures intermediate representations to localize the exact layer where causality breaks.
No Caching: Disabling incremental caching ensures both passes traverse the exact same computational graph, preventing state-reuse artifacts.
Deterministic Precision: In a correct implementation, $\Delta_\ell$ should be exactly zero (bitwise identical arithmetic), making the test robust to threshold selection.

3. Key Contributions

Formalization of Prefix Invariance: Defined a testable property where prefix representations must not depend on future inputs.
A Two-Pass Audit: Introduced a gradient-free, layer-localizing test that identifies where causality breaks, not just if.
Incompleteness of Mask Inspection: Demonstrated that checking attention masks is insufficient because leakage can occur in operators without masks (e.g., SSM scans).
Taxonomy of Leakage: Organized temporal leakage into four classes and eight executable fault patterns.
Cross-Architecture Validation: Validated the method across attention-only, state-space, recurrent, and hybrid models (including Mamba, RWKV, and Griffin).
Methodological Norms:
- Positive Control: A "clean" verdict is only valid if the audit successfully localizes injected faults on the same loaded checkpoint (proving the instrument is "live").
- Sequence Length Requirement: Audit sequence length must exceed the model's internal chunk, window, or kernel parameters; otherwise, specific code paths (and their defects) remain unexercised.

4. Experimental Results

Baseline Comparison

The authors compared their method against six existing practices on 96 injected-fault trials across four architectures:

Static Mask Inspection (B2): Detected 0/96 faults. It fails because faults often reside in the output of a layer, leaving the mask attribute unchanged.
Logits-Only Perturbation (B1, B4): Detected all faults but provided 0/96 correct localizations.
Gradient-Based Localization (B5): Achieved 96/96 detection and localization but requires a backward pass and full activation retention, making it significantly more expensive and less applicable to non-differentiable ops.
Proposed Method: Achieved 96/96 exact localization with negligible computational cost (two forward passes).

Note: The full census of 192 trials (8 patterns × 3 depths × 8 checkpoints) confirmed the proposed method's 100% localization rate across the broader set, while the baseline comparison in Table 1 specifically reports the 96-trial subset.

Discovery of Real Defects

The audit was applied to public checkpoints and revealed:

Clean Models: Standard attention models (SmolLM2, Qwen3, Pythia) and conformant SSMs (Mamba2, RWKV) passed with exact-zero deltas.
Identified Leaks:
- Zamba2-1.2B and Nemotron-H-8B were found to leak.
- Static Analysis: A code census of transformers 5.7.0 revealed that both models shared a byte-identical implementation error in their inter-chunk recurrence logic. They reduced over the output chunk axis instead of the input chunk axis, violating causality.
- Dynamic Verification: The leak manifested exactly at the model's declared chunk boundary (256 for Zamba2, 128 for Nemotron-H). Perturbing a token beyond the boundary contaminated the prefix.
- Fix: Replacing the erroneous three-line code block with the reference Mamba2 implementation reduced the prefix delta to exactly zero.

Methodological Failures and Corrections

The paper reports two critical failures of its own instrument that led to new norms:

Inert Loads: Three Falcon-H1 checkpoints returned $\Delta=0$ for all layers but failed to respond to input at all (positive control failed). This taught that a "clean" verdict requires a positive control (successful fault injection) on the same loaded artifact.
Sequence Length Blindness: The initial census used $T=48$ , which was shorter than the chunk sizes of Zamba2 and Nemotron-H. Consequently, the defects were invisible. The authors established that audit length must exceed internal model parameters (chunk/window size).

5. Significance and Claims

The paper argues that causal correctness is a structural property of the computation graph, not just a function of weights or attention masks.

The Mask is Not the Model: Relying on mask inspection is unsound for modern hybrid architectures.
Auditable Releases: The authors propose a new norm: architecture releases should include a causal-correctness certificate (per-layer deltas, threshold, sequence length relative to chunk size, and positive control results) alongside parameter counts and benchmarks.
Modest Scope: The paper does not claim to detect all possible defects (e.g., data-dependent leakage or faults in un-audited fused CUDA kernels). It establishes an existence proof that major models (Zamba2, Nemotron-H) contain structural causal violations in their reference PyTorch implementations and that these can be systematically detected and localized.

The work concludes that while detection of leakage is possible via simpler logit checks, the ability to localize the fault to a specific layer without gradients is the critical contribution, enabling engineers to fix the root cause rather than merely observing degraded performance.