Technical Summary: Large Language Models Encode Semantics and Alignment in Linearly Separable Representations

Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency across diverse semantic domains and alignment tasks, the internal mechanisms by which they organize and encode this knowledge remain opaque. Specifically, it is unclear to what extent LLMs linearly organize representations related to high-level semantic understanding (e.g., distinct scientific disciplines) and behavioral attributes (e.g., instruction following, safety alignment). Prior work has largely focused on narrow lexical features or binary behavioral attributes, leaving a gap in understanding how composite, high-level semantic content and alignment behaviors are geometrically structured within the latent space of autoregressive models.

Methodology

The authors conducted a large-scale empirical study analyzing hidden representations across 11 decoder-only autoregressive models (ranging from 124M to 24B parameters) and six high-level scientific topics (Computer Science, Economics, Electrical Engineering, Mathematics, Physics, and Statistics).

1. Data Collection: The study utilized abstracts from the arXiv metadata dataset, ensuring human-written content and consistent structure. Prompts were processed through the models, and hidden states were extracted from all layers immediately prior to the generation of the first token.
2. Linear Separability Analysis: The core analytical tool was a hard-margin Support Vector Machine (SVM). The authors trained SVMs on hidden states to classify pairs of topics. High SVM accuracy indicates that the representations of the two topics form linearly separable clusters in the latent space.
3. Robustness Checks:
   • Keyword Masking: To determine if separability relied on surface-level keywords, the authors masked domain-specific words based on frequency thresholds (0–99%) and re-evaluated separability.
   • Intrinsic Dimensionality: Singular Value Decomposition (SVD) was used to assess the intrinsic dimensionality of the subspaces occupied by semantic representations.
4. Alignment and Instruction Following:
   • Chain-of-Thought (CoT): The authors compared hidden states of identical questions with and without a CoT instruction (differing by ~15 tokens) to test for geometric shifts induced by reasoning prompts.
   • Safety: Using the WildJailbreak dataset, the authors analyzed the clustering of four prompt types: Direct Benign, Direct Harmful, Benign Injections (adversarial but safe), and Harmful Injections (jailbreaks).
5. Proof of Concept (Latent-Space Guardrail): A lightweight Multi-Layer Perceptron (MLP) probe was trained on the final-layer hidden states to classify prompt intent (4-class classification: injection vs. direct; benign vs. harmful). This probe was evaluated against token-level defenses like Llama Guard 3.

Key Contributions and Results

1. Progressive Linear Separability of Semantic Representations

The study found that high-level semantic information resides in low-dimensional subspaces that form linearly separable representations.

• Depth Dependence: Separability increases with layer depth. In deeper layers, representations of different domains (e.g., Physics vs. Computer Science) become increasingly distinct, often reaching near-perfect linear separability (SVM accuracy approaching 1.0).
• Robustness: This separability persists even when domain-specific keywords are heavily masked (up to 50–60%), suggesting that semantic structure is distributed across implicit cues (syntax, discourse patterns) rather than concentrated in specific lexical items.
• Architecture Effects: Separability patterns vary by architecture. For instance, models with grouped-query attention (Mistral) or sliding-window attention (Gemma) exhibit "sawtooth" patterns in separability across layers, correlating with their specific attention mechanisms.

2. Geometric Encoding of Alignment and Instruction Following

The linear structure extends to behavioral attributes:

• Instruction Following: Prompts containing Chain-of-Thought instructions produce hidden representations that are linearly separable from identical prompts without such instructions, despite the minimal token difference. This suggests that the "reasoning mode" is encoded as a distinct geometric direction.
• Safety Alignment: In safety-aligned models, benign, harmful, and adversarially framed prompts form distinct, separable clusters. Notably, harmful injections (jailbreaks) tend to cluster closer to benign inputs than direct harmful queries, reflecting the model's vulnerability to adversarial framing.

3. Latent-Space Interventions

Building on these geometric findings, the authors trained a lightweight MLP probe (13.9M parameters) on final-layer hidden states to act as a "latent-space guardrail."

• Performance: The guardrail achieved 96.7% overall accuracy in classifying prompt intent.
• Comparison: It substantially outperformed token-level defenses (specifically Llama Guard 3-8B) on adversarial prompts that bypass surface-level filters. The latent-space approach reduced harmful responses by over 2× while maintaining high utility for benign queries.
• Efficiency: Unlike token-based classifiers that require full model inference or large parameter counts, this guardrail operates on hidden states before token generation, offering a computationally efficient, real-time safety mechanism.

Significance and Claims

The paper claims that transformer-based LLMs develop an internal geometry where semantic knowledge and alignment behaviors are organized along interpretable, linear dimensions. This challenges the view that these models are purely black boxes, suggesting instead that they possess "distinctive and interpretable 'fingerprints' of alignment."

The authors argue that these findings motivate the development of geometry-aware tools that operate directly in latent space. By leveraging the inherent linear separability of hidden states, it is possible to detect and mitigate harmful content and adversarial manipulations without altering model weights or relying solely on external token-level filters. The study provides a proof of concept that simple linear probes can serve as effective, lightweight guardrails, offering a new paradigm for model interpretability and safety control.

Limitations

The authors acknowledge several limitations:

• The study is empirical and limited to selected models, layers, and scientific domains; it does not provide formal theoretical guarantees.
• The causal evidence for steering (e.g., inducing CoT behavior via vector addition) is preliminary; more rigorous analysis (e.g., adversarial perturbation) is needed.
• The safety analysis relies on the WildJailbreak dataset, and generalization to other models and prompt injection datasets requires further investigation.
• The study does not claim to solve all safety issues, noting that future work could combine latent-space probes with other defense mechanisms.