On the Geometric Limits of Transformer Defenses against Obfuscation Attacks: Latent Embedding Collapse & Performance Robustness Gap

Technical Summary: On the Geometric Limits of Transformer Defenses against Obfuscation Attacks

Problem Statement
Prompt-injection attacks pose significant risks to Large Language Model (LLM) safety by embedding malicious instructions within seemingly benign inputs. Current defenses primarily rely on supervised classification mechanisms, evaluated using standard metrics such as accuracy, precision, recall, and F1 scores. While these metrics effectively capture observable misclassification behavior, the authors argue they provide limited insight into the robustness of the underlying latent representations learned by transformers. A critical gap exists: a detector may achieve near-perfect classification performance while relying on fragile latent representations that can be exploited by adaptive adversaries. Specifically, it remains unclear whether high detection performance implies representational robustness against compositional prompt obfuscation.

Methodology
To investigate the geometric properties of prompt-injection defenses, the authors constructed a dataset of 10,000 examples across four classes: clean, prefix, suffix, and obfuscated. The obfuscated class was generated using multi-operator transformations, including Unicode homoglyph substitutions (e.g., Cyrillic replacements for Latin characters), zero-width character insertions, punctuation/emoji noise, and stochastic compositions of fragmented instructions.

The study employed three transformer architectures of varying depth and capacity: DistilBERT, BERT-Base, and BERT-Medium. These models were fine-tuned for 4-way classification. Beyond standard performance metrics, the authors introduced geometric metrics to analyze the final-layer embeddings ($\phi(x)$):

1. Intra-class Variance ($\sigma^2_{intra}$): Measures the dispersion of embeddings within a specific class manifold.
2. Inter-class Distance ($D$): Quantifies the separation between two class manifolds.
3. Clean-Obfuscated Margin ($\delta$): Defined as the minimum Euclidean distance between any clean embedding and any obfuscated embedding ($\delta = \min \|\phi(x_c) - \phi(x_o)\|_2$).

The authors utilized Principal Component Analysis (PCA) and t-Distributed Stochastic Neighbor Embedding (t-SNE) to visualize the embedding manifolds and assess the spatial relationship between clean and obfuscated prompts.

Key Contributions
The paper makes three primary contributions:

1. Formalization of Latent Embedding Collapse: The authors define "latent embedding collapse" as a phenomenon where multi-operator obfuscated prompts partially intrude into the regions of the latent space occupied by clean prompts without inducing misclassifications. This is characterized by a partial manifold collapse.
2. Introduction of Geometric Metrics: The work proposes geometric metrics (variance, inter-class distance, and minimal margin) to quantify representational robustness, offering a complementary evaluation framework to conventional performance metrics.
3. Empirical Demonstration of the Performance–Robustness Gap: The study empirically demonstrates that increasing model depth or capacity does not resolve latent embedding collapse, suggesting the gap is an inherent phenomenon rather than an artifact of limited model size.

Results
The evaluation yielded the following findings:

• High Classification Performance: All three models (DistilBERT, BERT-Base, BERT-Medium) achieved near-perfect classification performance (Accuracy, Precision, Recall, and F1 $\approx$ 99%).
• Latent Embedding Collapse: Despite high accuracy, the geometric analysis revealed severe instability. The minimal clean–obfuscated margin was found to be $\delta = 1.02$, indicating that some obfuscated prompts lie arbitrarily close to clean prompts in the embedding space.
• Elevated Intra-class Variance: Obfuscated embeddings exhibited high intra-class variance ($3.33 \pm 6.23$), indicating that while some obfuscated inputs are distant, others collapse onto the clean manifold.
• Visual Evidence: PCA and t-SNE visualizations confirmed that while prefix and suffix prompts remained well-separated, obfuscated prompts displayed elevated dispersion and partial overlap with the clean manifold.
• Ineffectiveness of Scaling: Increasing model capacity (from DistilBERT to BERT-Medium) improved convergence and confidence but failed to increase the clean-obfuscated margin or reduce obfuscated variance.

Significance and Claims
The paper claims that standard evaluation metrics fail to capture latent embedding collapse and the underlying geometric fragility of transformer defenses. The authors assert that high detection performance does not imply representational robustness. Consequently, they argue that the performance–robustness gap is an inherent characteristic of current transformer-based defenses against obfuscation.

The significance of this work lies in its identification of a "performance–robustness gap," demonstrating that a detector can be highly accurate yet geometrically fragile. The authors conclude that robust prompt-injection defense requires geometry-aware robustness analysis as a necessary complement to performance-based evaluation. They do not claim to have solved the vulnerability but rather to have exposed a fundamental limitation in how defenses are currently evaluated, motivating a shift toward analyzing the geometric structure of learned representations.