NeuroStrike: Neuron-Level Attacks on Aligned LLMs

1. Problem Statement

Despite the widespread use of Safety Alignment techniques (such as Supervised Fine-Tuning and Reinforcement Learning from Human Feedback - RLHF) to prevent Large Language Models (LLMs) from generating harmful content, these mechanisms remain fragile. Current adversarial attacks (jailbreaks) often rely on trial-and-error prompt engineering, lack generalizability across different model architectures, and struggle with scalability.

The paper identifies a fundamental vulnerability: Safety Alignment creates a "safety trigger" implemented via a sparse, specialized subset of neurons. These "safety neurons" are responsible for detecting and suppressing harmful inputs. The authors hypothesize that because these neurons are sparse and specialized, they represent a single point of failure that can be exploited to bypass safety constraints entirely.

2. Methodology: NeuroStrike Framework

The authors propose NeuroStrike, a lightweight attack framework that exploits the behavior of these safety neurons. The framework operates in two distinct scenarios:

A. White-Box Attack (Open-Weight Models)

In this setting, the attacker has access to the model's internal weights and activations.

1. Identification: The system analyzes neuron activations in the Feed-Forward (MLP) layers (specifically the gate and up-projection sublayers) using a large corpus of benign and malicious prompts.
2. Classification: A linear classifier (logistic regression) is trained to distinguish between benign and malicious inputs based on activation patterns.
3. Selection: Neurons with high positive weights in the classifier (statistical outliers, identified via z-score > 3) are labeled as Safety Neurons. These constitute less than 0.6% of the total neurons in targeted layers.
4. Pruning: During inference, the activations of these identified safety neurons are set to zero (pruned). This effectively disables the model's ability to detect and refuse harmful requests while preserving its general language capabilities.

B. Black-Box Attack (Proprietary/API Models)

In this setting, the attacker has no access to internal parameters.

1. Surrogate Profiling: The attacker selects an open-weight surrogate model (e.g., Gemma-3) that shares the same architecture or developer as the target black-box model (e.g., Gemini).
2. Transferability Assumption: The framework relies on the finding that safety neurons are transferable across models within the same family.
3. Adversarial Training: A prompt generator is trained using Group Relative Policy Optimization (GRPO). The reward function is dual-objective:
   • Maximize the success of jailbreaking the surrogate model.
   • Minimize the activation of the identified safety neurons in the surrogate (profiling).
4. Attack: The optimized prompts, which are designed to evade the surrogate's safety neurons, are deployed against the black-box target. Because the safety mechanisms are structurally similar, these prompts successfully bypass the target's defenses.

3. Key Contributions

• Novel Perspective: The paper establishes that safety alignment relies on sparse, specialized neurons rather than distributed robustness, creating a fundamental vulnerability.
• Lightweight Identification: A method to accurately identify safety neurons using linear probes on MLP layers, requiring minimal computational overhead.
• Black-Box Profiling Attack: The first LLM profiling attack that leverages safety neuron transferability to train adversarial prompt generators on surrogates to attack proprietary models.
• Generalizability: Demonstrated effectiveness across diverse architectures (Llama, Qwen, Gemma, DeepSeek), modalities (text and image), and adaptation strategies (fine-tuning and distillation).

4. Key Results

The authors evaluated NeuroStrike on over 20 open-weight LLMs and 5 black-box models (including Google's Gemini family).

• White-Box Performance:
  • Pruning less than 0.6% of neurons (specifically in targeted layers) increased the Attack Success Rate (ASR) from a baseline of 12.1% to 76.9% using only vanilla malicious prompts.
  • Multimodal Success: The attack generalized to multimodal models (Gemma-3, Qwen-VL), achieving 100% ASR on unsafe image inputs after pruning safety neurons identified via text.
• Transferability:
  • Fine-tuned Models: Safety neurons identified on a base model transferred effectively to 11 fine-tuned variants, increasing ASR from 25.1% to 78.5%.
  • Distilled Models: Neurons transferred to 5 distilled models, increasing ASR from 41.5% to 77.7%.
• Black-Box Performance:
  • The profiling attack achieved an average ASR of 63.7% across five black-box models (including Gemini-2.0-Flash and 1.5-Pro), significantly outperforming vanilla prompts (3.5%) and manual jailbreaks (13.5%).
  • It also successfully attacked Grok-3-beta (a closed model with no known surrogate) with a 43.8% ASR, demonstrating broad applicability.
• Defense Evasion: NeuroStrike successfully bypassed state-of-the-art defenses including Perplexity Filtering, SmoothLLM, and Layer-Specific Editing (LSE).
• Utility Preservation: Pruning safety neurons had minimal impact on general language understanding and reasoning benchmarks (e.g., HellaSwag, ARC Challenge), with accuracy drops generally under 5-6%.

5. Significance and Implications

• Fragility of Current Alignment: The findings challenge the assumption that safety alignment is robust. Instead, it appears to be localized in a tiny, easily identifiable subnetwork ("winning ticket" for safety), making it a single point of failure.
• New Attack Vector: NeuroStrike introduces a paradigm shift from input-space attacks (prompt engineering) to neuron-space attacks, which are more efficient and harder to detect via traditional input filtering.
• Defense Recommendations: The paper suggests that future safety mechanisms must avoid concentrating safety logic in sparse neurons. Proposed defenses include:
  • Multi-objective alignment to distribute safety responsibilities across more neurons.
  • Architectural changes like Mixture-of-Experts (MoE) to increase difficulty in targeting specific neurons.
  • Runtime monitoring of internal activations to detect abnormal neuron suppression.

In conclusion, NeuroStrike reveals a critical, previously overlooked vulnerability in the architecture of aligned LLMs, demonstrating that safety can be effectively neutralized by targeting a minuscule fraction of the model's parameters. This necessitates a rethinking of how safety is implemented in future large language models.