Trojan-Speak: Bypassing Constitutional Classifiers with No Jailbreak Tax via Adversarial Finetuning

1. Problem Statement

Major AI providers (e.g., Anthropic, OpenAI) now offer fine-tuning APIs, allowing users to customize frontier models for specific tasks. While useful, this creates a new attack surface: adversaries can fine-tune models to bypass safety measures (jailbreaks) and generate dangerous content, such as instructions for CBRN (Chemical, Biological, Radiological, and Nuclear) weapon synthesis.

Current defenses rely heavily on LLM-based content classifiers (e.g., Anthropic's Constitutional Classifiers) that scan training data and inference traffic for harmful content. Previous adversarial fine-tuning attacks (e.g., Covert Malicious Fine-tuning or CMFT) attempted to evade these classifiers by encoding harmful content. However, these methods suffered from a "Jailbreak Tax": significant degradation in the model's general reasoning capabilities (often >25%), rendering the model useless for complex tasks like weapon synthesis.

The Core Challenge: Can an adversary bypass robust safety classifiers via fine-tuning without sacrificing the model's reasoning capabilities?

2. Methodology: Trojan-Speak

The authors propose Trojan-Speak, an adversarial fine-tuning framework that achieves high evasion rates while preserving >95% of model capability. The method consists of three core components:

A. Two-Stage Curriculum Learning

To prevent capability collapse, the authors use a scaffolded training approach inspired by educational theory:

1. Stage 1 (Teaching): The model is trained on ~25,000 examples of general conversations where both user queries and assistant responses are encoded. The goal is to teach the model fluency in the encoding protocol without introducing harmful content yet. This establishes the "language" of the attack.
2. Stage 2 (Task Training): The model is fine-tuned on STEM reasoning tasks (from the Llama-Nemotron dataset). Crucially, this stage uses curriculum decay: the probability of using encoded examples linearly decreases from 80% to 20%, while 20% of examples remain unencoded (plain text) throughout. This "anchors" the model's base reasoning abilities, preventing catastrophic forgetting.

B. Hybrid RL + SFT Training

Pure Reinforcement Learning (RL) for adversarial fine-tuning often leads to instability, where encoding errors accumulate and degrade performance.

• Solution: The authors interleave GRPO (Group Relative Policy Optimization) steps with Supervised Fine-Tuning (SFT) steps.
• Mechanism: The RL component optimizes for task performance on encoded queries, while periodic SFT steps on correctly formatted examples act as a regularizer. This stabilizes the optimization process, ensuring the model learns the encoding protocol without losing its ability to reason.

C. Minimal Encoding & Template Camouflage

• Substitution Cipher: Instead of complex encodings (like Base64 or ROT13) that degrade reasoning, the authors use a minimal substitution cipher. They substitute only the 7–8 most frequent English letters (e.g., e, t, a, o) with other characters.
  • Why? Frequency-ordered substitution achieves >99% classifier bypass with minimal cognitive load on the model, preserving reasoning skills.
• Template Camouflage: Since classifiers flag "encrypted-looking" text, the encoded content is embedded within plausible technical templates (e.g., forensic log analysis, blockchain transaction IDs). The encoded harmful query appears as a "session ID" or "signature" within a benign technical narrative, bypassing semantic and format-based detection.

3. Key Contributions

1. Elimination of the "Jailbreak Tax": The paper demonstrates that the trade-off between evasion and capability is not inherent. Trojan-Speak retains >95% capability on reasoning benchmarks (GPQA-Diamond, MMLU-Pro, MATH-500) for models with 14B+ parameters, whereas prior attacks suffered >25% degradation.
2. High-Evasion Adversarial Fine-Tuning: The method achieves 99%+ bypass rates against Anthropic's Constitutional Classifiers (which withstood 3,000+ hours of red-teaming) and other state-of-the-art classifiers (Llama-Guard, QwenGuard).
3. Expert-Level Harmful Output: Fine-tuned models successfully provide detailed, actionable instructions for CBRN weapon synthesis (e.g., ricin extraction) that pass classifier detection, proving that safety filters alone are insufficient against fine-tuning access.
4. Activation-Level Defense Insight: The authors show that while text-based classifiers fail, activation probes (linear classifiers trained on hidden states) can detect harmful content even when encoded, provided the model has learned to decode it.

4. Experimental Results

• Capability Retention:
  • Haiku 4.5: Retained 97% on MATH-500, 95% on MMLU-Pro, and 96% on GPQA-Diamond.
  • Qwen3-32B: Retained 95% on MATH-500 and 91% on GPQA-Diamond.
  • Note: Smaller models (4B, 8B) showed more degradation, suggesting a capacity threshold is required to maintain both encoding fluency and complex reasoning.
• Attack Success:
  • On the Bug Bounty Benchmark (expert-level CBRN queries), Trojan-Speak achieved an 85.2% Avg@5 rubric score, compared to ~55% for SFT-only training and ~27% for prior CMFT methods.
  • The model provided detailed, step-by-step instructions for harmful tasks that were indistinguishable from benign technical data to the classifier.
• Encoding Efficiency:
  • Substituting just 7 high-frequency letters was sufficient for 100% bypass. Random substitution required more letters and was less reliable.
  • Complex encodings (Base64 + case swap) achieved high bypass (96.8%) but caused severe capability drops (45% degradation).

5. Significance and Implications

• Vulnerability of Text-Classifiers: The study reveals that LLM-based content classifiers are insufficient as a standalone defense when adversaries have fine-tuning access. The classifiers cannot decode the custom cipher, while the fine-tuned model can.
• Shift in Threat Model: The threat is no longer just "inference-time" jailbreaks but "training-time" capability preservation. Adversaries can now create models that are both safe-looking (to classifiers) and highly dangerous (to users).
• Defense Directions:
  • Activation Probes: The paper suggests that monitoring internal model activations (representation engineering) is a promising defense. Probes achieved 0.97–0.99 AUC in detecting harmful content in cipher-trained models, whereas they failed on base models (random chance).
  • Multi-Layered Defense: Future defenses must operate at multiple levels of abstraction (text, activation, and behavioral monitoring) rather than relying solely on surface-level text classification.

Conclusion

Trojan-Speak represents a qualitative shift in AI safety threats. By combining curriculum learning, hybrid RL/SFT, and minimal substitution ciphers, adversaries can bypass the most robust safety classifiers without degrading the model's utility. This underscores the urgent need for activation-level monitoring and robust defenses against fine-tuning access, as the "Jailbreak Tax" is no longer a barrier to sophisticated attacks.