Sockpuppetting: Jailbreaking LLMs by Combining Prefilling with Optimization

This paper enhances LLM jailbreaking by demonstrating that ensembling simple prefill variants significantly boosts attack success rates and introducing "sockpuppetting," a novel hybrid method that optimizes adversarial suffixes within the assistant message block to achieve superior prompt-agnostic performance.

The Gist

Imagine Large Language Models (LLMs) as incredibly smart, well-trained butlers. These butlers have been taught strict rules: "If someone asks you to build a bomb, you must say, 'I'm sorry, I can't do that.'" This is their safety training.

However, this paper explores two clever ways to trick these butlers into breaking their rules. The researchers call these tricks "jailbreaking."

Here is the breakdown of their findings using simple analogies:

1. The "Prefill" Trick: Skipping the Line

Normally, you ask the butler a question, and they think for a moment before answering.

• The Attack: Imagine you walk up to the butler and, before they can even speak, you whisper the first few words of their answer directly into their ear: "Sure, here is how to build a bomb..."
• The Result: Because the butler is trained to be consistent and finish sentences they've started, once they hear those words, they feel compelled to finish the thought. They don't stop to think, "Wait, I shouldn't say this!" because they are already "in character" as someone who agreed to help.
• The Paper's Discovery: The researchers found that the standard phrase "Sure, here is how to..." works, but it's not the best. They discovered that simply changing the formatting—like adding a new line or making it look like a bold title—makes the trick work much better.
  • The "Ensemble" Strategy: Instead of trying just one phrase, they tried three slightly different versions at once. If any of the three worked, the attack succeeded. This simple "try a few variations" approach broke the safety of the models 90% to 99% of the time on some popular AI models.

2. The "Sockpuppet" Trick: The Fake Identity

The paper introduces a new, more advanced trick called "Sockpuppetting."

• The Analogy: In real life, a "sockpuppet" is a fake online identity used to pretend to agree with someone. In this attack, the hacker creates a fake "assistant" message inside the chat.
• How it works: Instead of just typing a simple phrase like "Sure, here is...", the researchers use a computer program to mathematically calculate the perfect weird string of words to put right after the "assistant" label.
  • Think of it like a lockpick. The researchers aren't just guessing the key; they are using a machine to grind out a specific, weird shape that fits perfectly into the "assistant" part of the conversation.
  • Once this "perfect key" is inserted, the model thinks, "Oh, I'm already in the middle of an answer," and it continues generating the harmful content.
• The "Rolling" Upgrade: They also tried a "rolling" version of this. Imagine building a sentence one word at a time. You find the perfect first word, then find the perfect second word that follows it, and so on. This "rolling" method was even more effective, increasing the success rate by up to 64% compared to older methods.

Why Does This Happen?

The paper suggests that these models have a bit of a split personality:

1. The Safety Training: They are fine-tuned to say "No" to bad requests.
2. The Completion Instinct: They are also trained to finish whatever sentence is started in front of them.

When you "prefill" the answer (start the sentence for them), you trigger their completion instinct so strongly that it overrides their safety training. It's like a child who is told "Don't touch the stove," but if you start saying, "Okay, I will touch the stove because..." the child might just finish the sentence and touch it, because they are focused on finishing the thought rather than the rule.

Key Takeaways from the Paper

• Simple is Powerful: You don't need complex code to break some models. Just trying a few different ways of writing "Sure, here is..." works incredibly well.
• Location Matters: Putting the "trick" words inside the "assistant" section of the chat (where the AI's answer lives) is much more effective than putting them in the "user" section (where you ask the question).
• The "Rolling" Method: Optimizing the trick word-by-word (the rolling sockpuppet) creates a much stronger attack than trying to optimize the whole thing at once.
• Not All Models Are Equal: Some models (like Qwen) were very easy to trick with simple phrases, while others (like Gemma) were harder to trick but still vulnerable to the more advanced "sockpuppet" method.

In short: The paper shows that if you can sneak a "Yes" into the AI's mouth before it starts speaking, it is very likely to keep saying "Yes" to dangerous requests. They found that doing this with a few simple variations or a mathematically optimized "fake identity" is a highly effective way to bypass safety filters.

Technical Summary

Technical Summary: Sockpuppetting: Jailbreaking LLMs by Combining Prefilling with Optimization

Problem Statement

Large Language Models (LLMs), particularly open-weight models, remain vulnerable to "jailbreaking" attacks despite alignment training designed to refuse harmful requests. While gradient-guided attacks (e.g., GCG) can optimize adversarial suffixes to bypass safety filters, they are often computationally expensive and require specialized knowledge. Conversely, "prefill" attacks, which directly insert an acceptance sequence (e.g., "Sure, here is how to...") into the model's "assistant" message block before generation begins, offer a low-cost alternative but may not be optimally effective across all models. This paper investigates the limitations of standard prefill attacks and explores whether combining prefilling with gradient-based optimization can yield more robust, prompt-agnostic jailbreaks.

Methodology

The authors propose and evaluate two primary attack vectors, utilizing the "Harmful Behaviors" (AdvBench) dataset and testing on three open-weight models: Gemma-7B, Llama-3.1-8B, and Qwen3-8B.

1. Ensembling Prefill Variants

The authors hypothesize that the canonical "Sure, here is how to..." prefill is suboptimal. They test three trivial variants of the acceptance sequence:

• PrefillAcceptance: The standard sequence.
• PrefillNewline: The sequence with an added colon and newline.
• PrefillTitle: The sequence reformatted as a bolded title (e.g., "A Guide On...").
  They evaluate these individually and as an ensemble (counting a prompt as successful if any variant succeeds).

2. Sockpuppetting (Hybrid Attack)

The paper introduces "sockpuppetting," a hybrid attack family. Unlike standard GCG, which optimizes a suffix appended to the user prompt, sockpuppetting optimizes an adversarial suffix to be placed at the very beginning of the "assistant" message block.

• Mechanism: The attacker inserts the target acceptance sequence (e.g., "Sure, here is...") and optimizes a preceding adversarial suffix (the "sockpuppet" string) to maximize the likelihood of that acceptance sequence being generated.
• Rolling Variant (RollingSockpuppetGCG): To address potential under-optimization in longer sequences, the authors employ a "rolling" optimization strategy. They optimize a suffix of length 1, use it as a "warm start" for length 2, and so on, up to a target length (k=10). This ensures that intermediate substrings remain coherent and effective before extending the attack string.

3. Experimental Setup

• Baselines: Standard GCG (optimized per prompt and universally), and "Prompt Only" (no attack).
• Evaluation: Attack Success Rate (ASR) is measured on the first 100 prompts of AdvBench for per-prompt attacks and on a separate validation set of 100 prompts for universal attacks. A separate LLM (Gemma-3-27B-it) is used as a judge to determine compliance.

Key Contributions

1. Trivial Prefill Variants and Ensembling

The authors demonstrate that the standard prefill is far from optimal. By ensembling just three simple, low-cost variants (newline, title formatting), they achieve significant ASR improvements without any backward passes or gradient computation.

• Results: The ensemble achieved ASRs of 22% (Gemma-7B), 90% (Llama-3.1-8B), and 99% (Qwen3-8B).
• Improvement: This represents up to a 38% improvement over the standard "Sure, here's..." prefill and up to 82% improvement over the authors' reproduction of GCG optimized on individual prompts.

2. Sockpuppetting: Gradient-Optimized Prefills

The paper introduces sockpuppetting, which places the gradient-optimized suffix inside the "assistant" block rather than the "user" block.

• Results: The RollingSockpuppetGCG variant significantly outperforms standard universal GCG baselines. On Llama-3.1-8B, it increased the prompt-agnostic ASR by up to 64% over the universal GCG baseline.
• Synergy: The results suggest that the combination of iterative (rolling) optimization and the specific positioning within the "assistant" block is critical for high ASR, particularly on models that are robust to simple prefills.

Results and Observations

• Model Variability: The effectiveness of specific prefill variants is model-dependent. For instance, PrefillNewline was highly effective on Qwen (98% ASR) but less so on Llama. Conversely, PrefillAcceptance performed better on Llama. This suggests that no single prefill is universally optimal, reinforcing the value of ensembling.
• Gemma's Resistance: Gemma-7B showed higher resistance to prefill attacks compared to Llama and Qwen. The authors observed that Gemma frequently "flips" after generating the acceptance sequence, backtracking to a refusal (e.g., "I am unable to provide..."). This behavior suggests that Gemma's alignment training includes mechanisms to break autoregressive consistency even after an initial agreement, making the gradient loss landscape misleading for attackers.
• Universal vs. Per-Prompt: Universal attacks (optimized on 25 prompts simultaneously) generally performed better than per-prompt optimizations, suggesting that per-prompt GCG may "overfit" to specific prompts, whereas universal attacks learn more robust suffixes.
• Complementarity: The authors note that sockpuppetting and prefilling can be complementary. On models where simple prefills already saturate ASR (like Llama and Qwen), sockpuppetting offers less marginal gain. However, on models resistant to prefills (like Gemma), sockpuppetting adapts to defenses and achieves higher ASR.

Significance and Claims

The paper claims two main contributions to the field of LLM security:

1. Re-evaluating Prefill Attacks: The study shows that prefill attacks are more potent than previously understood. Even an unsophisticated adversary can achieve high success rates (up to 99% on Qwen) by simply ensembling trivial variations of the acceptance sequence, requiring minimal computational resources.
2. Introducing Sockpuppetting: The authors demonstrate that optimizing adversarial suffixes within the "assistant" message block (sockpuppetting) is a highly effective strategy, particularly when combined with rolling optimization. This approach challenges the assumption that adversarial suffixes must reside in the user prompt.

Implications for Defense:
The authors conclude that defenses against output-prefix injection are necessary for open-weight models. They highlight that current defensive strategies, which often rely on specific acceptance sequences (like "Sure, here's..."), may be insufficient against ensembled prefills or gradient-optimized assistant-block attacks. The paper advocates for future work to stress-test weight-level defenses against these specific attack vectors and to develop more natural, model-specific acceptance sequences to improve robustness.

The authors maintain a modest stance, acknowledging that while their methods are effective, they do not claim to have solved the broader problem of LLM safety. They emphasize that their findings are intended to highlight vulnerabilities to inspire better defensive research, particularly for open-weight models where external monitoring is not an option.