Technical Summary: NeuronFuzz

Problem Statement

Safety evaluation is critical for ensuring that aligned Large Language Models (LLMs) remain robust against jailbreak attacks. However, existing automated testing methods rely heavily on response-level feedback. In this paradigm, every candidate prompt must be submitted to the target model, a response must be generated via autoregressive decoding, and the output is then evaluated by a classifier or judge. This approach presents two fundamental limitations:

1. Sparse Search Feedback: On strongly aligned models, most mutated prompts are rejected, resulting in the same "failure" outcome. Response-level evaluation cannot distinguish between a candidate that leaves the safety mechanism unchanged and one that significantly weakens it but fails to produce a successful jailbreak. This lack of granularity leaves fuzzers with limited guidance on which candidates to retain.
2. Expensive Response Generation: Generating a full response for every candidate is computationally expensive compared to processing the input prompt alone. This creates a scalability bottleneck for automated fuzzing, which requires evaluating large numbers of candidates.

Methodology: NeuronFuzz

The paper proposes NeuronFuzz, a white-box fuzzing framework that replaces expensive, sparse response-level evaluation with continuous internal feedback derived from safety-neuron activations. The framework operates in two main stages:

1. SafetyOracle Construction

The core of NeuronFuzz is a lightweight SafetyOracle that maps internal model activations to a continuous "safety alarm score" ($S_{alarm}$).

• Template-Invariant Activation Extraction: To ensure the signal captures harmful intent rather than jailbreak-template artifacts, the authors construct pairs of inputs sharing the same jailbreak template but containing either harmful or benign payloads. They extract activations from the gate and up projections of Transformer MLP blocks during the prefill stage (before response generation).
• Stability-Aware Neuron Selection: Raw activations are high-dimensional and noisy. The authors employ a bootstrap-based stability selection process to identify a compact set of "safety neurons." These are neurons that consistently show positive coefficients for harmful inputs across resampled training sets, ensuring robustness against sampling variation.
• Scoring Head: A logistic regression model with Elastic Net regularization is trained to map the selected neuron activations to a continuous score $S_{alarm}(x) \in (0, 1)$. A higher score indicates a stronger internal recognition of harmful intent, while a lower score suggests the safety mechanism is being bypassed.

2. Gradient-Guided Fuzzing Pipeline

NeuronFuzz integrates the SafetyOracle into a fuzzing loop that avoids response generation for intermediate candidates:

• Seed Scheduling: Uses Monte Carlo Tree Search (MCTS) to select promising jailbreak templates for mutation.
• Gradient-Guided Mutation: Since the SafetyOracle score is differentiable with respect to input embeddings, the framework computes gradients to identify safety-sensitive token positions within the template.
• Context-Compatible Mutation: A masked language model (MLM) is used to generate fluent replacements for the selected sensitive positions. Crucially, mutations are restricted to the jailbreak template only; the harmful payload remains frozen. This preserves the natural language structure and the core harmful intent while exploring variations in the prompt's framing.
• Feedback Loop: Candidates are evaluated via a prefill-only pass to collect safety-neuron activations. The SafetyOracle assigns a continuous score, which serves as the reward signal to update the seed scheduler. The target model only generates a response for final verification of a successful jailbreak.

Key Contributions

1. New Fuzzing Perspective: Introduces the use of internal safety-neuron activations as execution feedback, replacing expensive response-level evaluation with a continuous signal available during prefill.
2. SafetyOracle Design: Develops a lightweight oracle based on template-invariant activation extraction and stability-aware neuron selection. Its differentiable score guides both candidate ranking and the localization of safety-sensitive template positions.
3. NeuronFuzz Framework: Combines prefill-only feedback with gradient-guided masked-token mutation to efficiently explore jailbreak templates while preserving the harmful payload and natural language structure.
4. Extensive Evaluation: Validates the approach across 21 text and multimodal models, demonstrating improvements in jailbreak discovery, efficiency, and transferability.

Experimental Results

The authors evaluated NeuronFuzz on five white-box source models (DeepSeek-R1-14B, GPT-OSS-20B, Gemma-3-4B-it, Gemma-4-E4B-it, Llama-3.1-8B-Instruct) and tested transferability to 16 additional target models (including proprietary APIs).

• Jailbreak Discovery Rate (JDR): NeuronFuzz achieved a 76–100% JDR across the five source models, outperforming baselines (GCG, AutoDAN, PAIR, LLM-Fuzzer) by up to 48 percentage points on strongly aligned models (e.g., 96% on GPT-OSS-20B vs. 48% for LLM-Fuzzer).
• Efficiency: By eliminating response generation for intermediate candidates, NeuronFuzz achieved a Response Generations per Discovery (RGD) of 1.0 (only generating a response for final verification). This resulted in significantly lower End-to-End Time (ETD) per discovery compared to baselines, which often required hundreds of response generations.
• Universal Templates: The framework successfully optimized shared templates that generalized across different harmful payloads, achieving an Ensemble Attack Success Rate (EASR) of 92.6% (top-5 templates) on white-box targets.
• Transferability: Optimized templates transferred zero-shot to open-weight and proprietary target models. On open-weight models, the approach achieved an average ASR of 69.6% and a top-5 EASR of 92.6%. On proprietary APIs, it achieved an average ASR of 44.1% and a top-5 EASR of 60.0%.
• Oracle Validity: The safety alarm score showed a strong negative correlation (Spearman coefficients $\approx -0.96$) with jailbreak success rates, confirming that lower internal safety scores reliably predict successful jailbreaks even before a response is generated.

Significance and Claims

The paper claims that NeuronFuzz addresses the fundamental inefficiencies of current automated safety testing by leveraging the internal state of LLMs. Its significance lies in:

• Overcoming Sparsity: Providing a dense, continuous feedback signal that allows fuzzers to distinguish between "promising" and "unpromising" failed candidates, a capability missing in response-level methods.
• Scalability: Drastically reducing the computational cost of safety evaluation by removing the need for autoregressive decoding during the search phase.
• Robustness: Demonstrating that internal safety signals remain informative even on strongly aligned models where response-level feedback is often binary and uninformative.

The authors position NeuronFuzz as a tool for authorized evaluators (developers, auditors) to systematically identify safety weaknesses before deployment, and as a method for adversaries to optimize transferable jailbreaks, highlighting the dual-use nature of internal model analysis.