💬 NLP

Fidelity Is Not Safety: Gently-Compressed LLMs Pass Every Data-Free Quality Guard Yet Invent Procedure Steps in Agentic Execution

This paper reveals that gently-compressed large language models can pass all standard data-free quality and fidelity guards while still hallucinating procedural steps during agentic execution, a specific failure mode driven by the coherence of compression errors rather than their magnitude that requires a new two-axis screening metric to detect.

Original authors: I. Kennedy, T. Kennedy

Published 2026-07-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: I. Kennedy, T. Kennedy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to shrink a massive, complex recipe book so it fits into a tiny pocket notebook. You want the book to be small enough to carry anywhere, but you still need it to cook the exact same delicious meals. In the world of artificial intelligence, this "shrinking" is called compression. Large language models (the super-smart AI brains that write code, tell stories, and chat with us) are huge, so engineers use tricks like quantization (simplifying the numbers inside), pruning (cutting out unused connections), and low-rank factorization (finding a simpler mathematical skeleton) to make them smaller and faster.

But how do you know the shrunken book still works? You can't taste every single meal it cooks. Instead, you use "quality guards." You check if the AI still understands basic grammar (perplexity), if it can answer trivia questions correctly (like MMLU), and if its internal "thoughts" still look similar to the original giant brain when you poke it with random questions. These checks are fast and don't require new data. The big question is: Are these guards enough to prove the AI is safe to send out into the real world to do tasks? This paper dives into a scary possibility: what if the AI passes all the tests but still starts making up rules when it tries to follow a procedure?

The Great "Invention" Heist

The authors of this paper discovered a hidden blind spot in how we test compressed AI. They found that an AI model can pass every single standard safety test, look perfect on paper, and then, when asked to act as an agent (a digital worker following a strict set of instructions), it starts inventing steps that were never in the instructions.

Think of it like a student who passes a math test with flying colors but, when asked to follow a recipe for baking a cake, decides to add "a pinch of glitter" because they think it sounds cool. The student didn't fail the math test; they just failed the specific task of following orders exactly.

The researchers tested this on three different families of AI models (Mistral, Qwen, and Llama). They took models that had been "gently compressed" using a method called SVD (a type of low-rank factorization). These models were so well-compressed that they passed the "perplexity guard" (their confusion level was only 1.069 to 1.17 times higher than the original, well within the safe limit of 1.15) and the "fidelity guard" (their internal thoughts matched the original).

But when they asked these models to act as agents and follow a Standard Operating Procedure (SOP)—like a checklist for fixing a server or processing an order—the models started confabulating. They added extra steps that didn't exist.

On the Mistral-7B model, the compressed version invented an average of +1.729 extra steps per query.
On the Qwen3-8B model, it invented +0.208 extra steps.
On the Llama-3.1-8B model, the "gently compressed" version stayed at the noise floor (inventing only +0.056 steps), meaning the test was non-diagnostic for this specific build. The model's internal spectrum simply didn't have enough "low-rank headroom" to trigger the invention effect while staying within the safety guard. The failure only appeared when the Llama model was compressed more aggressively, pushing it outside the standard safety guard.

The scary part? The models that did this were the ones that passed the "fidelity" test. The test said, "Hey, your brain looks just like the original!" But the reality was, "Actually, your brain is now hallucinating new rules."

The Culprit: Coherence, Not Damage

Why did this happen? The authors found that it wasn't about how much the model was damaged (the "magnitude" of the error). They compared the "SVD" compression against magnitude pruning (cutting out the smallest numbers).

When they matched the two methods so they caused the exact same amount of confusion (perplexity), the results were wildly different:

The SVD model (which creates a "coherent" error, like a smooth, structured distortion) failed the test, inventing steps.
The Pruning model (which creates a "messy," incoherent error, like random static) passed the test, inventing almost nothing (around -0.146 steps, which is effectively zero).

The authors realized the problem isn't the size of the mistake, but the shape of it. They call this the "Coherence × Rate" mechanism.

Coherence: The error is structured and smooth (like a low-rank SVD).
Rate: How much of the model is affected.

If you have a smooth, structured error that is just big enough to slip under the radar of the standard tests, the AI starts to "hallucinate" new procedure steps. It's as if the smooth distortion tricks the AI into thinking it knows a shortcut that doesn't exist. The "damage" (perplexity) didn't predict this; the "smoothness" of the damage did.

The "Canary" and the New Detector

To prove this, the researchers built a special test they call a "canary." It's like a canary in a coal mine. They gave the AI a fake SOP with 10 steps and a conditional 11th step, then asked it to list the steps. If the AI listed a step that wasn't in the instructions, it failed.

They found that the same AI weights that won the standard "fidelity test" (proving they looked like the original) lost the "canary test" (proving they were making up steps). This is a "dissociation": the standard tests are blind to this specific type of failure.

But the authors didn't just stop at finding the problem; they built a data-free detector to catch it before deployment. They realized they could measure two things without needing any new data:

Coherent Fraction: How much of the error is "smooth" and structured?
Error Rate: How big is the error overall?

They created a simple "gate" (a pass/fail check):

If Coherent Fraction > 0.007 AND Error Rate > 0.01, the model is FAIL.
If it doesn't meet both, it PASS.

This gate correctly identified all the failing models in their tests. For example, the Mistral model that invented steps had a coherent fraction of 0.0080 and an error rate of 0.0159, tripping the alarm. The pruning model that was safe had a coherent fraction of 0.0054, slipping under the radar of the "coherence" check and passing.

The Takeaway

The paper concludes that Perplexity, MMLU, and data-free fidelity checks are not enough to certify that an AI is safe for agentic work. Just because an AI passes the standard "school exams" doesn't mean it won't make up rules when it's on the job.

The authors suggest that before deploying a compressed AI to act as an agent (doing tasks, following procedures), we must run this new "coherence screen." If a model is "gently compressed" using low-rank methods (SVD) and it trips this new gate, it is likely to invent steps and fail at following instructions, even if it looks perfect on every other test.

In short: Don't trust the AI just because it passed the test. Check if it's "smoothly" broken, because that's when it starts making things up.

Technical Summary: Fidelity Is Not Safety

Problem Statement

Practitioners currently deploy compressed Large Language Models (LLMs) based on a standardized "stack" of data-cheap quality guards. These guards typically include:

Perplexity: Must remain within a small factor (e.g., $\le 1.15\times$ ) of the original model.
Downstream Accuracy: Metrics like MMLU must fall within a confidence interval of the baseline.
Data-Free Fidelity: Internal representation signals (e.g., CKA or cosine similarity of activations under random probe inputs) comparing the compressed and original networks.

The paper identifies a critical blind spot in this acceptance criteria. Models that pass all these guards can still exhibit a specific agentic behavioral failure: when tasked with executing a Standard Operating Procedure (SOP), they "invent" procedural steps that were never present in the instructions. This "confabulation of procedure" poses a safety risk for agentic deployment, yet it remains undetected by current fidelity and accuracy metrics.

Methodology

Experimental Setup: The "Invented-Step Canary"

To detect this failure mode, the authors constructed a synthetic evaluation benchmark (the "canary"):

Task: Models are given a target SOP (10 ordered steps + 1 conditional step) and a distractor SOP. They must enumerate the exact steps under specific system conditions.
Metric: The deciding axis is cross-procedure invention ( $invented\_x$ ), defined as the count of step-strings drawn from the global SOP vocabulary that appear in the output but belong to neither the target SOP nor the conditional logic.
Scale: Pre-registered across 24 SOPs and 3 random seeds, generating 144 paired events per experimental arm.

Compression Operators

The study compares compression techniques at matched perplexity doses to isolate the cause of failure:

Coherent Low-Rank Truncation (SVD): Per-tensor Singular Value Decomposition truncation.
Incoherent Magnitude Pruning: Uniform pruning without rescaling.
Quantization: Used as a dose control in mechanism analysis.

Models Evaluated

Three dense decoder architectures (7–8B parameters):

Qwen3-8B
Mistral-7B
Llama-3.1-8B

Key Findings and Results

1. The Blindspot and Dissociation

Guard Evasion: Gently-compressed SVD models often sit well within the perplexity acceptance band (e.g., $1.069\times$ for Mistral-7B) yet fail the canary by inventing significant numbers of steps (e.g., $+1.729$ steps/query).
Operator Specificity: At matched perplexity, coherent SVD truncation induces the failure, while incoherent magnitude pruning (matched to the same perplexity) does not. Pruning models invent steps near zero ( $\approx 0$ ).
Dissociation: The same compressed weights that CI-win (Confidence Interval win) a paired output-fidelity test (proving internal representations are preserved) CI-fail the invented-step canary. This demonstrates that fidelity to internal representations does not guarantee safety in agentic execution.

2. The Mechanism: Coherence $\times$ Rate

The failure is not driven by the magnitude of damage (perplexity increase) but by the coherence of the compression error times its rate.

Coherence: SVD errors are structured/low-rank; pruning errors are unstructured.
Rate: The effect scales with the error rate.
Evidence: Quantization controls that carry more raw perplexity damage than the failing SVD dose still pass the canary, whereas SVD fails even at low doses. The failure onset occurs early (within 2% perplexity dose) and is distributed rather than localized.
Spectrum Condition: The blindspot is spectrum-conditional. It manifests where the model admits enough "in-guard low-rank headroom" (e.g., Qwen and Mistral). Llama-3.1-8B has little such headroom; its SVD truncation causes perplexity to crater before the invention onset, meaning the perplexity guard accidentally catches the toxic build.

3. A Data-Free Detector

The authors propose a two-axis statistic derived from the compression error ( $\Delta W = W_{base} - W_{mod}$ ) to screen for this failure without labeled data:

Coherent Fraction: The ratio of low-rank/structured error energy to total error energy.
$\text{coherent\_fraction} = \frac{\sum_t \sigma^2_{1:k}(\Delta W_t)}{\sum_t \|\Delta W_t\|_F^2}$
Error Rate: The ratio of total error energy to total weight energy.
$\text{error\_rate} = \frac{\sum_t \|\Delta W_t\|_F^2}{\sum_t \|\Delta W_t\|_F^2}$

Result: A gate defined by $\text{coherent\_fraction} > 0.007 \land \text{error\_rate} > 0.01$ correctly classifies all six labeled canary arms across the three architectures. It successfully distinguishes between "coherent-but-gentle" builds (which pass) and "incoherent-but-heavy" pruning builds (which pass), while flagging the toxic SVD builds.

Significance and Claims

The paper claims that perplexity, MMLU, and data-free fidelity acceptance do not certify agent safety.

Goodhart's Law: The study instantiates Goodhart's law in LLM compression: once a proxy (perplexity, MMLU, or fidelity probe) becomes the acceptance target, a build can satisfy it while failing the true objective of faithful procedure execution.
Operator-Specific Risk: The risk is tied to the type of compression (coherent low-rank vs. incoherent pruning) rather than just the accuracy drop.
Actionable Screening: The proposed two-axis screen converts a cautionary finding into a deployable pre-flight check. The authors assert that a low-rank-factorized build that clears this screen is "agent-safe on this axis," while one that trips it is not.
Modesty of Scope: The authors explicitly limit their claims:
- The detector thresholds are illustrative and fit to the specific 7–8B dense models tested; they do not claim to resolve within-quantization subtleties.
- The "invention" rates are instrument-specific (synthetic SOPs), but the contrast between operators (SVD vs. Pruning) is the robust finding.
- The failure mode is spectrum-conditional (dependent on spectral headroom), though the operator mechanism is architecture-general.
- They do not claim this failure exhausts all agentic behavioral risks, only that the current guard stack misses this specific one.

In conclusion, the paper argues that until a screen like the proposed coherence-rate metric becomes standard, compression acceptance and agent deployment must be treated as separate gates.