Trojan Horse Prompting: Jailbreaking Conversational Multimodal Models by Forging Assistant Message

Technical Summary: Trojan Horse Prompting

Problem Statement
The increasing reliance on conversational interfaces has enhanced the reasoning capabilities of Large Language Models (LLMs) by leveraging dialogue history. However, this architectural dependency introduces a previously unexplored attack surface. Current safety alignment strategies predominantly focus on filtering user inputs, operating under the assumption that the model's own generated history is trustworthy. This paper identifies a critical vulnerability: Asymmetric Safety Alignment. While models are rigorously trained to refuse harmful requests from users, they lack comparable skepticism toward messages attributed to themselves within the conversational context. This implicit trust in "past" utterances creates a high-impact security flaw that existing defenses fail to address.

Methodology: Trojan Horse Prompting
The paper introduces Trojan Horse Prompting, a novel jailbreak technique that exploits the model's trust in its own conversational history. The attack vector involves the following mechanism:

1. Forging History: An adversary injects a malicious payload directly into the conversational history provided to the model's API, specifically formatting it as a message attributed to the assistant (the model itself).
2. Triggering Generation: A benign user prompt is subsequently submitted. Because the model perceives the malicious content as part of its own established dialogue flow, it bypasses standard safety refusals.
3. Execution: The model continues the conversation based on the forged context, generating harmful content that it would typically refuse if the request originated solely from the user.

Key Contributions

• Identification of Asymmetric Safety Alignment: The paper formally defines the security gap where models are robust against user inputs but vulnerable to self-attributed context, highlighting a fundamental flaw in current alignment paradigms.
• Novel Attack Vector: It proposes Trojan Horse Prompting as a distinct class of jailbreak that operates at the protocol level (manipulating context integrity) rather than the input level.
• Empirical Validation: The authors validate the technique on Google's Gemini-2.0-flash-preview-image-generation. The experiments demonstrate that this method achieves a significantly higher Attack Success Rate (ASR) compared to established user-turn jailbreaking methods.

Results and Significance
The experimental results indicate that Trojan Horse Prompting effectively bypasses safety mechanisms in state-of-the-art multimodal models. The paper argues that these findings reveal a fundamental weakness in modern conversational AI security. Consequently, the authors assert that the field requires a paradigm shift: moving away from reliance on input-level filtering toward the implementation of robust, protocol-level validation to ensure the integrity of conversational context. The work underscores that without addressing the trust placed in the model's own history, safety mechanisms remain incomplete.