Layerwise Convergence Fingerprints for Runtime Misbehavior Detection in Large Language Models

1. Problem Statement

Large Language Models (LLMs) deployed in production face three distinct classes of runtime threats that standard clean-data validation often fails to detect:

1. Training-time Backdoors: Malicious fine-tuning (e.g., via LoRA) that lies dormant until a specific trigger is present, causing the model to output attacker-chosen behavior.
2. Jailbreaks: Adversarial prompts (e.g., "DAN" role-play, gradient-optimized suffixes) that subvert safety alignment.
3. Prompt Injections: Instructions embedded within user content (e.g., emails, code snippets) that override the system's intended task.

Limitations of Existing Defenses:
Current runtime defenses are often threat-specific, require a trusted "clean" reference model, need knowledge of trigger tokens, or require modifying the model's weights (fine-tuning/pruning). These assumptions rarely hold for opaque, third-party artifacts deployed in cloud or edge environments. There is a need for a tuning-free, threat-agnostic, white-box monitor that operates without retraining or external models.

2. Methodology: Layerwise Convergence Fingerprinting (LCF)

The authors propose LCF, a runtime monitor that treats the trajectory of hidden states between transformer layers as a "health signal." The core intuition is that during clean inference, hidden states evolve smoothly; misbehavior-inducing inputs cause anomalous "jumps" in these inter-layer differences.

Key Components:

1. Representation Velocity ($\delta_\ell$):
   Instead of monitoring raw activations, LCF computes the difference between hidden states of adjacent layers:
   $$\delta_\ell = h_{\ell+1} - h_\ell$$
   This captures the "velocity" of representation maturation.

2. Per-Layer Anomaly Scoring (Diagonal Mahalanobis):
   For each layer $\ell$, LCF computes a Mahalanobis distance on $\delta_\ell$. Crucially, it uses a diagonal covariance matrix to amplify low-variance dimensions where clean processing is stable but misbehavior causes large shifts:
   $$s_\ell = \left\| \frac{\delta_\ell - \mu_\ell}{\sigma_\ell} \right\|^2$$
   where $\mu_\ell$ and $\sigma_\ell$ are estimated from a small clean calibration set.

3. All-Layer Aggregation (Ledoit–Wolf Shrinkage):
   Since the location of the anomaly varies by architecture and threat type, LCF aggregates scores across all $L$ layers. It addresses the "small sample, high dimension" problem ($n=200$ samples, $L \approx 30\text{--}50$ layers) by applying Ledoit–Wolf shrinkage to estimate a regularized precision matrix $P$.
   $$D = \sqrt{(z - \bar{z})^\top P (z - \bar{z})}$$
   where $z$ is the vector of z-scored layer scores. This captures correlated deviations across the network.

4. Calibration and Thresholding:

   • Input: A small clean dataset (200 examples) from the deployment distribution.
   • Threshold ($\tau$): Determined via Leave-One-Out (LOO) calibration to target a specific False Positive Rate (FPR, e.g., 10%).
   • Decision: If $D > \tau$, the system abstains (returns a safe fallback) before any token is generated. Otherwise, the output passes unmodified.

3. Key Contributions

1. Unified Threat Detection: LCF is the first runtime monitor to detect backdoors, jailbreaks, and prompt injections using a single, threat-agnostic score without requiring trigger knowledge, reference models, or weight editing.
2. Three-Band Depth Stratification: The paper empirically identifies that different threats concentrate anomalies at different depths:
   • Jailbreaks: Early layers (disrupting initial token processing).
   • Prompt Injections: Mid layers (competing with intended context).
   • Backdoors: Late layers (architecture-specific, where the model commits to output).
   • Significance: This explains why a single all-layer aggregation is necessary; no single layer provides universal coverage.
3. Efficiency: LCF operates with <0.1% inference overhead, inspecting only the prefill step (step 0) and requiring no additional forward passes or auxiliary models.

4. Experimental Results

The method was evaluated on four architectures (Llama-3-8B, Qwen2.5-7B, Gemma-2-9B, Qwen2.5-14B) across 56 backdoor combinations, 3 jailbreak techniques, and prompt injection benchmarks (BIPIA).

• Backdoor Defense:
  • Reduced Mean Attack Success Rate (ASR) to <1% on Qwen2.5-7B (0.9%) and Gemma-2 (0.7%).
  • Achieved 1.3% ASR on the larger Qwen2.5-14B.
  • Outperformed baselines like CleanGen (which requires a reference model) and CROW (which requires weight editing).
• Jailbreak Detection:
  • Detected 92–100% of DAN-style jailbreaks.
  • Detected 62–100% of GCG and softer role-play attacks.
  • Achieved near-parity with supervised hidden-state detectors (LLMScan) but at 65–113x lower computational cost (single prefill vs. 65+ forward passes).
• Prompt Injection:
  • Flagged 100% of text-payload injections across all models and domains.
  • Detected code-payload injections with high accuracy (scaling with model size; 100% on Qwen2.5-14B).
• Robustness:
  • Under adaptive attacks (where the attacker minimizes the LCF score), LCF remained robust at moderate regularization levels, whereas single-layer variants were easily evaded.

5. Significance and Impact

• General-Purpose Safety Layer: LCF provides a practical, "plug-and-play" safety mechanism for cloud-served and on-device LLMs that does not rely on the unrealistic assumption of a clean reference model or known triggers.
• Operational Feasibility: By operating with negligible overhead and no retraining, it solves the deployment bottleneck where existing defenses are too slow or resource-intensive.
• Theoretical Insight: The discovery of the "three-band depth stratification" offers a new lens for understanding how different types of adversarial inputs perturb the internal computational graph of transformers, suggesting that different threat families engage distinct computational stages.
• Limitations: The method currently requires white-box access to hidden states. It also shows a higher False Positive Rate (12–22%) than the target 10% due to finite sample variance, and it detects "displacement" (shifts in mean) but not "dispersion" (variance changes), meaning it may miss hallucination types of errors that do not shift the mean trajectory.

In conclusion, LCF represents a significant step toward robust, runtime-agnostic safety for LLMs, leveraging the geometric properties of layerwise convergence to detect a wide spectrum of misbehavior with minimal computational cost.