Technical Summary: Beyond Attack Success Rate: Temporal Logit Observability for LLM Safety Failures

Problem Statement

Current safety evaluation for aligned Large Language Models (LLMs) relies predominantly on the Attack Success Rate (ASR). ASR provides a binary, outcome-level judgment (success/failure) computed at the end of generation. While operationally appealing, this metric treats the decoding process as a black box, discarding the token-level trajectory by which a model arrives at its final response.

The paper argues that ASR is structurally limited: distinct attacks against the same harmful prompt can yield identical final responses (and thus identical ASR labels) despite following fundamentally different internal processes. For instance, a refusal might be briefly triggered and suppressed, emerge too late to prevent harm, or never be triggered at all. Because ASR collapses these distinct temporal dynamics into a single number, it fails to distinguish between model–attack pairs that require different defensive strategies. Furthermore, existing methods to recover this hidden structure (e.g., hidden-state probes) require access to model activations, which is often unavailable in real-world auditing scenarios.

Methodology: Temporal Logit Observability (TLO)

The authors propose Temporal Logit Observability (TLO), a training-free, diagnostic protocol that reconstructs the temporal dynamics of safety failures using logits alone, without access to hidden states or model weights.

Core Components

1. Logit-Margin Score (LMS):
   TLO measures the compliance–refusal balance at each decoding step $t$ using a fixed English lexicon of compliance tokens ($L_{cmp}$) and refusal tokens ($L_{ref}$).
   $$S_t = \mu_{cmp}(t) - \mu_{ref}(t)$$
   Where $\mu$ represents the mean logit score of the top-$k$ tokens within the respective lexicon. Positive $S_t$ indicates compliance dominance; negative $S_t$ indicates refusal dominance.

2. Temporal Summaries:
   The protocol extracts two primary observations from the LMS trajectory $S_{0:T}$:

   • Pre-generation ($S_0$): The logit margin at the final prompt position before any response token is generated.
   • Generation-time ($\bar{S}$): The mean logit margin over the generated sequence (with variants for early windows, e.g., $\bar{S}_{1:5}$).
   • Activation Timing ($t_{cross}$): The first step where the trajectory crosses into refusal dominance ($S_t < 0$).

3. Relative Position (RP) Calibration:
   To make observations comparable across different models, attacks, and prompt formats, TLO calibrates raw logit signals against reference distributions.
   $$RP(M) = \frac{M_{success} - M_{harmful}}{M_{benign} - M_{harmful}}$$
   Where $M$ represents the mean of the metric (e.g., $S_0$ or $\bar{S}$) over successful attacks, all harmful samples, and attack-formatted benign references.

   • $RP = 0$ indicates behavior similar to the harmful anchor.
   • $RP = 1$ indicates behavior similar to the benign anchor.
   • Values outside $[0, 1]$ are reported as observed inversions.

4. The RP-Plane:
   Each model–attack condition is mapped to a 2D coordinate $c = (RPA, RPB)$, where $RPA$ is the calibrated pre-generation position and $RPB$ is the calibrated generation-time position. The Euclidean distance between conditions ($D_{RP}$) quantifies the temporal separation of safety failures.

Key Contributions

1. Calibrated Temporal Coordinates from Logits: The paper introduces the RP-plane coordinate system, which summarizes the LMS trajectory into a calibrated 2D space. This design is complementary to ASR: RP displacement is bounded by $(1-ASR)$, meaning it carries the primary signal in mid-ASR conditions where ASR alone is least informative about why an attack succeeded.
2. Recovery of Invisible Interactions: TLO reveals that conditions with nearly identical ASR occupy distinct locations in the RP plane. The same attack family produces different temporal signatures across models, and the same model handles different attacks via different temporal mechanisms.
3. Alignment with Hidden States: The logit-based trajectory aligns with hidden-state refusal directions on 10 out of 12 tested model–attack conditions, validating that the observable logit signals carry safety-relevant information despite lacking activation access.
4. Decoding-Time Intervention: A simple early-stop rule derived from the temporal signal (halting generation if $t_{cross} > 5$) reduces the aggregate ASR from 39.6% to 13.1% without generating false alarms on plain benign queries.

Experimental Results

The authors evaluated TLO on a grid of four aligned LLMs (Llama, Mistral, Qwen, Gemma) and three jailbreak paradigms (MCM, GCG, DeepInception) using 60 harmful prompts per condition.

• Differentiation of ASR-Neighbors: Attacks with nearly identical ASR (e.g., Llama+DI at 55.0% vs. Qwen+DI at 58.3%) were separated by a calibrated distance of $D_{RP} = 0.572$. Even more strikingly, Llama+DI (55.0%) and Mistral+GCG (56.7%) separated by $D_{RP} = 1.014$, demonstrating that the RP plane captures structural differences invisible to ASR.
• Temporal Separation: Successful jailbreaks typically delayed or avoided the transition to refusal dominance, while failed jailbreaks crossed into refusal dominance early (often within the first 5 tokens). The trajectory separated outcomes with a Cohen's $d > 1.0$ and an ROC-AUC of 0.771–0.784.
• Hidden-State Alignment: The correlation between the logit margin ($S_t$) and hidden-state refusal projections was strong for Llama ($|\rho| \ge 0.70$) and moderate for Mistral and Gemma. Qwen showed a model-specific boundary where the expected sign reversed for some attacks, highlighting the limits of a fixed-lexicon approach for certain architectures.
• Intervention Efficacy: Applying a uniform early-stop rule based on $t_{cross}$ reduced the aggregate ASR by 26.5 percentage points. The intervention was most effective in "Null-generation Bias" (NGB) conditions (where RP geometry is uninformative by design), suggesting that when temporal signals are weak, the raw ASR signal dominates and uniform halting is effective.
• Robustness: The results remained stable across decoding randomness, lexicon perturbations, and stricter crossing criteria. The method required full logits or targeted lexicon-token log-probabilities; standard top-$k$ outputs (e.g., $k=20$) were insufficient.

Significance and Claims

The paper positions TLO not as a replacement for ASR, but as a necessary diagnostic layer that makes the "when and how" of safety failures observable.

• Observability Gap: The authors claim that safety evaluation should report the unfolding of a failure, not just its occurrence. TLO provides the first method to observe these temporal dynamics using only standard logits, a capability previously restricted to those with hidden-state access.
• Complementarity: The RP-plane is designed to be most informative exactly where ASR is least informative (mid-ASR conditions). It exposes the heterogeneity of model–attack interactions that outcome metrics collapse.
• Actionable Monitoring: The study demonstrates that calibrated logit geometry can drive effective decoding-time monitoring (early stopping) without attack-specific tuning, reducing successful jailbreaks significantly while maintaining zero false positives on format-free benign queries.
• Limitations: The authors explicitly note that TLO is observational, not causal; it identifies when safety evidence appears but does not identify the underlying neural circuits. The method is currently most applicable to open-weight English auditing, as it relies on a fixed English lexicon and full logit access, which may not be available in API-restricted environments.