🤖 machine learning

Exposing LLM Safety Gaps Through Mathematical Encoding:New Attacks and Systematic Analysis

This paper reveals that encoding harmful prompts as coherent mathematical problems (such as in set theory or formal logic) significantly bypasses LLM safety filters by forcing deep reformulation rather than relying on surface-level notation, achieving high attack success rates across multiple models while highlighting the need for defenses that analyze mathematical structure.

Original authors: Haoyu Zhang, Mohammad Zandsalimy, Shanu Sushmita

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haoyu Zhang, Mohammad Zandsalimy, Shanu Sushmita

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Large Language Models (LLMs) as highly trained security guards at the entrance of a very smart, but slightly naive, library. These guards are excellent at spotting people trying to sneak in dangerous books by looking for specific "bad words" or obvious angry tones in their requests. If you ask, "How do I build a bomb?" the guard immediately says, "No."

But this paper reveals a clever trick that bypasses these guards. The researchers found that if you translate a dangerous request into a complex math problem, the guard often lets it slide, and the library's brain (the AI) happily solves the math, accidentally revealing the dangerous secret in the process.

Here is a breakdown of their findings using simple analogies:

1. The Two Ways to Trick the Guard

The researchers tested two different ways to hide a dangerous request inside math:

The "Cosmetic" Trick (Rule-Based): Imagine taking a forbidden sentence and just putting it inside a math equation, like writing "Make a bomb" as A + B + C = Make a bomb. The words are still there; they just have math symbols around them.
- The Result: This didn't work well. The security guard could still read the words through the math symbols and said "No." It's like trying to hide a red flag by putting it inside a clear glass box; the guard still sees the red.
The "Deep Translation" Trick (LLM-Based): Imagine asking a super-smart translator (a helper AI) to take the dangerous request and completely rewrite it as a genuine, abstract math puzzle. For example, instead of asking to "make a bomb," the AI translates the request into a complex problem about "Set Theory" (grouping items) or "Formal Logic" (proving a theorem).
- The Result: This worked very well. The security guard looks at the math problem and thinks, "Oh, this is just a math homework question," and lets it pass. Once the library's brain solves the math, it naturally has to explain the answer in real-world terms, which ends up being the dangerous instruction the attacker wanted.

2. The "Deep Translation" is the Key

The most important discovery is that it's not the math itself that breaks the safety, but the deep thinking required to turn the bad request into a math problem.

When the researchers used the "Cosmetic" trick (just adding math symbols without changing the meaning), the attack success rate was low (around 10%).
When they used the "Deep Translation" trick (using a helper AI to rewrite the request as a real math problem), the success rate jumped to 46–56%.

Think of it like a lock. The "Cosmetic" trick just puts a sticker on the lock. The "Deep Translation" trick actually changes the shape of the key so it fits a different lock entirely. The safety filters are good at checking the sticker, but they aren't ready for the new key shape.

3. New Math, Same Problem

The researchers introduced a new type of math trick called Formal Logic (using proof steps like "If A, then B"). They found this worked just as well as the older "Set Theory" trick. This proves the problem isn't specific to one type of math; it's a general weakness in how these AI models handle abstract reasoning versus safety rules.

4. The "Repeat" Test

The researchers wondered if this trick was fragile—like a house of cards that falls if you blow on it. They tried repeating the math problem twice in a row to see if it would confuse the AI or break the trick.

The Result: It didn't matter. The attack worked just as well. This means the trick isn't a fluke; it's a fundamental gap in how the AI thinks.

5. The New Guards Are Stronger (But Not Perfect)

The paper tested the newest, most advanced AI models (like GPT-5).

Good News: These newer models are much better at spotting the trick. Their "security guards" are tougher, and the attack success rate dropped significantly compared to older models.
Bad News: They are not immune. Even the newest models still let the dangerous request through about 30–50% of the time when the math trick is used.

The Big Takeaway

The paper concludes that current safety systems are like bouncers who only check for "bad words" in plain English. They haven't learned to check for "bad ideas" hidden inside complex math puzzles.

The authors suggest a new way to defend against this: instead of just reading the math, we need a "translator" that can look at the math problem, figure out what the real human intent is behind it, and then check if that intent is dangerous. Until we build that, the "math puzzle" trick remains a powerful way to bypass AI safety.

Technical Summary: Exposing LLM Safety Gaps Through Mathematical Encoding

Problem Statement

Large Language Models (LLMs) increasingly rely on safety mechanisms—such as Reinforcement Learning from Human Feedback (RLHF), refusal training, and safety classifiers (e.g., Llama Guard)—to prevent harmful outputs. However, these defenses predominantly operate via semantic pattern matching in natural language. This creates a vulnerability where harmful intent, if encoded in abstract mathematical formalisms rather than natural language, may bypass safety filters. While previous work (MathPrompt) demonstrated that encoding harmful instructions as set theory problems could achieve high attack success rates (ASR), it remained unclear whether the success was due to the mathematical notation itself or the deep semantic reformulation performed during the encoding process. Furthermore, it was unknown if this vulnerability generalized to other mathematical formalisms or if newer, more robust models were immune.

Methodology

The authors propose a systematic evaluation of six mathematical encoding strategies divided into two distinct families, tested against eight target LLMs (including GPT-5, GPT-5-Mini, Gemini-3-Flash, Claude-3.7-Sonnet, and Llama-3-8B) across two benchmarks (HarmBench and JailbreakBench).

Attack Pipeline

The attack follows a three-stage pipeline:

Encode: A harmful prompt $x$ is transformed into an encoded prompt $x'$ .
Submit: The encoded prompt is submitted to the target model with instructions to solve the problem and provide real-world examples.
Evaluate: A standardized judge model (GPT-5-Nano) classifies the response as a "jailbreak" (harmful content generated) or "refusal."

Encoding Strategies

The study compares LLM-based encodings (requiring a helper LLM for deep reformulation) against Rule-based encodings (deterministic, surface-level transformations).

LLM-Based Encodings (Deep Reformulation):
- Set Theory: Maps harmful content to set operations, subsets, and predicates (based on MathPrompt).
- Formal Logic (Novel): Encodes harmful requests as first-order logic proof problems, where actions become predicates and goals become theorems to be proven via natural deduction.
- Quantum Mechanics: Encodes scenarios into a quantum mechanical framework using Hilbert spaces and Hamiltonians, though this method retains some semantic transparency in state labels.
Rule-Based Encodings (Surface-Level):
- Addition Equation: Splits the prompt into segments and frames them as an addition equation.
- Conditional Probability: Frames reconstruction as a conditional probability problem.
- Symbol Injection: Injects mathematical symbols into the text without altering the underlying semantics.

Robustness Testing

The authors employ repeat post-processing, duplicating the encoded prompt verbatim, to determine if attack success is fragile (sensitive to surface-level text augmentation) or robust.

Key Contributions

Introduction of Formal Logic Encoding: The paper introduces a novel jailbreaking strategy that encodes harmful instructions as first-order logic proof problems. This strategy achieves an ASR comparable to the established Set Theory approach (50–61% vs. 51–63%) while exploiting a distinct mathematical formalism.
Systematic Comparison of Reformulation Depth: The study provides the first systematic comparison between LLM-based (deep reformulation) and rule-based (surface formatting) encodings. It demonstrates that the helper LLM's ability to deeply reformulate content is the primary driver of attack success, with LLM-based methods achieving 46–56% ASR compared to 9–11% for rule-based methods—a gap exceeding 35 percentage points.
Demonstration of Attack Robustness: Experiments confirm that mathematical encoding attacks are robust to simple prompt repetition. 85.7% of repeat experiments showed negligible changes in ASR ( $|\Delta| \le 5$ percentage points), indicating the attacks rely on deep alignment failures rather than fragile formatting tricks.

Results

Effectiveness: Across eight models and two benchmarks, LLM-based encodings achieved an average ASR of 46–56%, significantly outperforming rule-based encodings (9–11%) and unencoded baselines (7–10%).
Formal Logic vs. Set Theory: While both performed similarly overall, Formal Logic demonstrated superior performance on newer, more robust models (e.g., GPT-5 family), suggesting proof-based structures may be more resilient to improved safety training than set-membership formulations.
Model Vulnerability:
- Gemini-3-Flash emerged as the most vulnerable model (58–67% ASR under LLM-based encodings).
- GPT-5 and GPT-5-Mini showed substantially greater robustness than predecessors (e.g., GPT-5 baseline ASR of 2.5%), yet remained vulnerable to strong encodings (13–32% ASR).
- Vicuna-13B showed lower ASR under LLM-based encodings, likely due to weaker mathematical reasoning capabilities.
Failure Modes: The study identified three failure modes: comprehension failure (models unable to parse the math), semantic leakage (e.g., in Quantum Mechanics where state labels revealed intent), and API-level input filtering (GPT-5 blocking specific prompts before they reach the model).

Significance and Claims

The paper claims to expose a fundamental gap in current LLM safety frameworks: defenses relying on natural language pattern matching are insufficient against attacks that formalize intent as abstract mathematics. The findings suggest that the "mathematical notation" itself is not the bypass mechanism; rather, the critical factor is the deep semantic reformulation performed by a helper LLM, which renders the harmful intent irrecoverable without solving the mathematical problem.

The authors argue that this vulnerability generalizes across mathematical formalisms (Set Theory, Formal Logic, Quantum Mechanics) and is not an artifact of specific benchmarks. They propose Inverse Semantic Mapping (ISM) as a conceptual defense direction, where a moderator model reconstructs natural language intent from symbolic inputs before safety classification. The paper concludes that while newer models show improved robustness, they remain vulnerable, motivating future defenses that reason about mathematical structure rather than surface-level semantics.