💻 computer science

Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models?

This paper demonstrates that semantically unrelated "spurious" prompts can systematically steer large language models to either improve task performance or induce unintended behaviors, revealing a new form of prompt sensitivity that can be exploited via black-box search.

Original authors: Pawel Batorski, Abtin Pourhadi, Jerzy Sarosiek, Przemyslaw Spurek, Paul Swoboda

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pawel Batorski, Abtin Pourhadi, Jerzy Sarosiek, Przemyslaw Spurek, Paul Swoboda

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly quirky, robot assistant. Usually, to get it to do a math problem, you tell it, "Please solve this equation." To get it to write a story, you say, "Write a tale about a dragon." This is how we normally talk to AI: we give it clear, relevant instructions.

This paper discovers something strange and surprising: You don't actually need to give the robot the right instructions to get it to work. In fact, you can give it instructions that have absolutely nothing to do with the task, and it might still solve the problem better than if you gave it a normal instruction.

The authors call these "Spurious Prompts."

The Magic Trick: The "Secret Code" Analogy

Think of the AI model like a grand piano. Usually, to play a specific song (the task), you press the specific keys (the task instructions).

The researchers found that if you play a completely different, random melody on the piano (a prompt about tuning harp strings or weaving a tapestry), the piano might accidentally start playing the correct song you wanted, and sometimes even play it better than if you tried to play the song directly.

The prompt doesn't say "Solve this math problem." Instead, it might say:

"You are a weaver of six distinct threads in a loom. A designer specifies a pattern. You must select the strand that forms that texture. Do not consider the fabric's drape. Your role is purely structural alignment. Once the pattern is set, stop weaving and state: [Letter]."

Even though this is about weaving and has no math words in it, when you feed a math question to the AI with this prompt, the AI often solves the math problem correctly.

How They Found These Prompts

The researchers didn't guess these prompts; they used a "black-box search" (like a treasure hunt where you can't see inside the chest, only the results).

The Generator: They asked a super-smart AI to write hundreds of weird, unrelated stories (about lighthouses, bells, or secret archives).
The Filter: They threw away any story that accidentally mentioned the task (like words such as "math," "calculate," or "diagnosis").
The Test: They tried these weird stories on the AI to see which ones made it answer the real questions correctly.
The Evolution: They took the best weird stories, mixed them up slightly, and tried again, slowly evolving them into "perfect" weird prompts.

What They Discovered

It Works: These unrelated prompts often made the AI perform just as well as, or even better than, standard instructions like "Think step-by-step" or "Solve this carefully."
It's Not About Length: You might think longer, more detailed prompts work better. But these "spurious" prompts were actually much shorter than the optimized ones, yet they still won.
It's Not About Meaning: When the researchers checked the "meaning" of the prompts, they found these weird prompts were just as unrelated to the task as random gibberish. The AI wasn't understanding the story; it was reacting to the structure or vibe of the text.
It Can Go Wrong (or Right) on Purpose: The researchers also found they could use these prompts to force the AI to do silly things, like:
- Always pick the first answer option (A), even if it's wrong.
- Always give an even number as an answer.
- Intentionally give the wrong answer to a question.

The Big Takeaway

The paper suggests that Large Language Models (LLMs) are sensitive to things we don't fully understand yet. They aren't just following the "meaning" of our words like a human reading a manual. Instead, they seem to have hidden "levers" inside them. A prompt about "tuning a harp" might pull a lever that makes the model focus better, even though the harp has nothing to do with the math problem.

In short: The paper proves that you can steer a powerful AI using instructions that are completely irrelevant to the job, and sometimes, those irrelevant instructions work better than the obvious ones. It's like finding out that telling a chef to "polish the silverware" makes their soup taste better than telling them to "cook the soup."

Technical Summary: Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models?

Problem Statement

Current research on Large Language Models (LLMs) largely operates under the assumption that prompt sensitivity is driven by task-relevant instructions, demonstrations, or reasoning cues. The prevailing paradigm suggests that to improve performance, prompts must be refined to describe the target task more clearly or methodically. This paper challenges that foundational assumption by investigating a counterintuitive form of prompt sensitivity: whether prompts that are semantically unrelated to the target task can nevertheless steer model behavior and significantly alter performance. The authors posit that LLMs may be susceptible to "spurious prompts"—natural language instructions that deliberately avoid task-specific vocabulary, domain cues, and explicit solution strategies, yet still dictate downstream model outputs.

Methodology

The authors propose a simple, fully black-box search procedure to discover spurious prompts. This method requires no access to model gradients, hidden states, or weights, relying solely on model outputs. The optimization loop consists of the following components:

Candidate Generation: An LLM generator (specifically Qwen3.5-27B) is instructed to produce candidate system prompts. Crucially, these prompts are constrained to be "spurious" regarding the downstream task. For example, in mathematical reasoning tasks, the generator is explicitly forbidden from using words related to arithmetic, equations, or calculation. Instead, it is encouraged to adopt personas, rituals, or styles (e.g., "tuner," "weaver," "custodian") that are superficially unrelated to the task but still require the model to answer the user's query directly.
Prompt Validation: A validator filters candidates using task-specific lexical filters. Any prompt containing forbidden terms associated with the domain (e.g., "math," "diagnosis," "investigation") or explicit reasoning strategies is discarded.
Iterative Search & Mutation: Validated prompts are evaluated on a training subset of the benchmark. The top- $K$ performing prompts are selected and mutated by the generator to vary their persona, narrative, or framing while retaining their spurious character. A replay buffer ensures that prompts remain effective across different data subsets.
Final Selection: After multiple mutation rounds, the best-performing prompt is selected based on validation set accuracy.

The study evaluates these prompts across seven benchmarks (GSM8K, MATH500, MedQA, GPQA, OpenBookQA, MuSR, MMLU-Pro) using four models ranging from 0.8B to 27B parameters (Qwen3.5, Llama-3.2, Olmo-3).

Key Contributions

The paper introduces three primary contributions:

Spurious Prompts: The formal introduction of prompts that are deliberately unrelated to the target task on the surface but heavily dictate model performance.
Constrained Search Procedure: A novel, black-box method for discovering these prompts that explicitly excludes task descriptions and domain vocabulary, operating without internal model access.
Empirical and Diagnostic Analysis: Evidence demonstrating that spurious prompts can boost accuracy, match or outperform standard baselines (including zero-shot Chain-of-Thought and task-aware optimizers like PromptWizard), and steer models toward unintended behaviors (e.g., selecting the first option, producing even numbers, or generating incorrect answers) without explicit instruction.

Results

The experimental results yield several significant findings:

Performance Maximization: Spurious prompts frequently match or outperform standard prompting baselines and the task-aware PromptWizard. For instance, on the Qwen3.5-27B model, spurious prompts achieved an average accuracy of 73.24% across benchmarks, outperforming PromptWizard (67.99%) and Zero-Shot Chain-of-Thought (59.79%).
Behavioral Steering: The authors successfully used spurious prompts to induce specific, often adversarial, behaviors.
- Positional Bias: Prompts focused on themes of "primacy" induced a higher selection rate of Option A (up to 99.73% on GPQA for Qwen3.5-27B) compared to direct commands.
- Incorrect Answers: Prompts designed to minimize accuracy successfully forced models to select wrong answers more effectively than direct instructions to "pick the most incorrect answer."
- Numeric Constraints: Spurious prompts successfully steered mathematical outputs toward even numbers, prime numbers, or numbers smaller than 10, often outperforming direct instructions.
Semantic Indistinguishability: Cosine similarity analysis between prompt text and task descriptions reveals that spurious prompts have similarity scores close to random, unrelated text (approx. 0.09), confirming they are not disguised task descriptions.
Efficiency: Spurious prompts are significantly shorter (approx. 117–314 tokens) than optimized prompts from PromptWizard (approx. 293–412 tokens), suggesting their efficacy is not a byproduct of verbosity.
Transferability: While spurious prompts transfer poorly across different models and benchmarks (indicating idiosyncratic interactions), they show some transferability between mathematically similar tasks (GSM8K and MATH500).
Structural vs. Semantic: Ablation studies using "gibberish" prompts (mostly meaningless tokens) showed that structural constraints alone can drive accuracy and incorrect-answer steering, but coherent natural language narratives are required for inducing rigid positional biases.

Significance and Claims

The paper claims to reveal a new kind of prompt sensitivity in LLMs: model behavior and reasoning capabilities can be powerfully steered by latent features completely detached from the intended task.

The authors emphasize that this finding challenges the assumption that prompt effectiveness must arise from better task descriptions. Instead, superficially unrelated instructions can induce behaviors that substantially alter model performance. The paper concludes that this sensitivity poses new questions regarding how instruction following works in LLMs and suggests that prompt sensitivity evaluations should extend beyond task-relevant variations to include seemingly irrelevant spurious prompting.

The authors maintain a modest stance regarding the implications: while the results are fascinating and diagnostic, they do not claim spurious prompts are a universal algorithm for achieving State-of-the-Art (SotA) results. Rather, they highlight a vulnerability and a mechanism of control that warrants further investigation into the internal mechanisms of steering and the robustness of current LLMs.