Semantic-Preserving Prompt Hijacking: A Black-Box Adversarial Attack on Auto-Prompt Optimization

Technical Summary: Semantic-Preserving Prompt Hijacking via Adaptive Greedy Local Search

Problem Statement
Large Language Models (LLMs) increasingly integrate Automatic Prompt Optimization (APO) modules, which rewrite user input before generating a response to enhance transparency and usability (e.g., Microsoft Bing Copilot, Claude.ai). While these modules aim to build trust by revealing modifications, their mechanism of autonomously selecting a "best" candidate from multiple alternatives creates a new attack surface. The authors identify that attackers can hijack this optimization process by inducing subtle, imperceptible semantic shifts in the input. The core challenge is to craft adversarial inputs that maintain high semantic similarity to the original text (to appear legitimate to the optimizer) while causing the target LLM to produce a significantly different, misleading output. Existing black-box adversarial methods often fail in this specific context because they lack the dynamic adaptability required to navigate strict semantic constraints within multi-stage optimization pipelines.

Methodology: Adaptive Greedy Local Search (AGLS)
The paper proposes Adaptive Greedy Local Search (AGLS), a black-box attack framework designed to exploit visible-cue optimizers. The method operates under the constraint that the semantic similarity $S(t, t')$ between the original input $t$ and the adversarial input $t'$ must exceed a threshold $\sigma$, while ensuring the model output changes ($O(t) \neq O(t')$).

The AGLS framework proceeds through the following stages:

1. Input Decomposition and Checkpointing: The input text is segmented into hierarchical clauses based on punctuation and syntax. The end of each clause serves as a "checkpoint" ($t_i$).
2. Dynamic Candidate Adjustment: At each checkpoint, the system calculates the semantic similarity between the original prefix and the partially generated adversarial prefix using a BERT-based similarity metric. A dynamic threshold guides the selection of candidate replacement words:
   • If similarity is too low, the search shifts toward candidates that increase similarity.
   • If similarity is too high (insufficient deviation), the search shifts toward candidates that increase deviation.
   • This feedback loop allows the attack to balance semantic preservation with the maximization of behavioral deviation.
3. Masked Token Generation: Key language units (verbs, plural nouns) are masked and replaced using a masked language model (BERT). The top-$k$ candidates are ranked, and the final token is selected based on the adaptive rules described above.
4. Iterative Reconstruction: The process repeats incrementally, updating the context after each substitution until all masked positions are filled, resulting in the final adversarial example.

Theoretical justification is provided by framing the attack as a sequential local decision problem rather than a global optimization, approximating the global optimum through local decisions while avoiding the accumulation of semantic drift via real-time feedback.

Key Contributions

1. Identification of a Novel Attack Surface: The authors formalize the vulnerability introduced by visible prompt optimization modules in production-level LLMs, where the selection mechanism can be hijacked via semantic drift.
2. Proposal of AGLS: A novel black-box attack method that dynamically balances semantic preservation and attack effectiveness through adaptive checkpoint-based search, avoiding the "early-commitment bias" common in static beam search methods.
3. Empirical Validation: Extensive experiments across commercial (GPT-4, GPT-4o) and open-source (Llama-3, Qwen-2.5, Gemma-2) models on four diverse datasets (GSM8K, Math QA, SQuAD, Strategy QA).

Experimental Results
The paper evaluates AGLS on over 2,400 test cases. Key findings include:

• Superior Attack Success Rate (ASR): Under identical semantic similarity constraints, AGLS consistently outperforms state-of-the-art methods (including TextFooler, BERT-Attack, G2PIA, and BeamAttack). For instance, on the Math dataset, AGLS achieved an ASR of 60.61%, surpassing the second-best method by 6.79 percentage points.
• Performance Across Tasks: The method achieves high ASR (often >60%) on numerical reasoning tasks (GSM8K, Math QA), with some models showing ASR up to 89.47%. Performance is slightly lower on textual QA tasks (SQuAD, Strategy QA), attributed to the distributed nature of factual reasoning, but still remains competitive.
• Efficiency: The average generation time ranges from 0.17s to 2.49s, demonstrating feasibility for near-real-time interactive attacks.
• Ablation Studies: Results confirm that the dynamic candidate adjustment mechanism is critical; static selection strategies result in significantly lower ASR (e.g., dropping from 90.91% to 36.36% on GPT-4). A search scope of 13,000 candidates was found to offer the best trade-off between effectiveness and computational cost.

Significance and Claims
The paper claims that AGLS establishes a new state-of-the-art for semantic-preserving adversarial attacks, particularly in scenarios mimicking real-world prompt optimization pipelines. The work highlights a critical security vulnerability in transparency-enhancing optimization modules, suggesting that the very features designed to build user trust can be exploited to mislead models. The authors emphasize that robustness does not scale linearly with model parameters, as even large models (e.g., Llama-3.1-70B) remain highly vulnerable to this specific attack vector.

The paper concludes by noting limitations and potential defenses, such as multi-index consistency detection, randomization in candidate screening, and the risk of reduced efficacy if core keywords are misidentified during the attack. Future work is proposed to extend AGLS to multimodal and multi-turn settings.