A Universal Vibe? Finding and Controlling Language-Agnostic Informal Register with SAEs

1. Problem Statement

Large Language Models (LLMs) have achieved remarkable proficiency in multilingual syntax and factual recall. However, they struggle with pragmatic registers, specifically informal language and slang.

• The Register Gap: Due to alignment pipelines favoring standard, edited text, LLMs often fail to process or generate dynamic, context-dependent informal language. This leads to "miscalibrated" responses and safety vulnerabilities where harmful intent is obscured by casual phrasing.
• The Core Question: When an LLM encounters informal register across diverse cultures, does it treat it as a collection of isolated, language-specific memorizations (surface-level heuristics), or does it construct a unified, language-agnostic abstraction of "casual human interaction"?
• Previous Limitations: Prior work on cross-lingual alignment focused on facts and syntax. Investigating pragmatic concepts is difficult because existing datasets often rely on lexical bias (e.g., detecting slang simply because specific slang words are present), making it hard to isolate the pragmatic context from the vocabulary.

2. Methodology

The authors probe the internal representations of Gemma-2-9B-IT using Sparse Autoencoders (SAEs) to disentangle polysemantic features and identify causal mechanisms.

A. Dataset Construction: Lexical Control

To eliminate the confound of lexical bias, the authors created a novel, strictly controlled dataset across three typologically diverse languages: English, Hebrew, and Russian.

• Polysemous Target Terms: Every target term in the dataset is polysemous, appearing in both a literal sense and an informal/slang sense (e.g., "fire" meaning a blaze vs. "fire" meaning excellent).
• Contextual Resolution: Because the token is identical in both contexts, the model must resolve the register entirely based on pragmatic context, not word identity.
• Scale: The dataset contains ~10,600 sentences (English: 2,835; Hebrew: 6,559; Russian: 1,259).

B. Feature Extraction & Analysis

• SAE Application: The authors used 131K-feature SAEs (Lieberum et al., 2024) to decompose dense residual stream activations into sparse, human-interpretable features.
• Differential Activation ($\Delta_i$): They calculated the differential activation frequency for each feature: $\Delta_i = P(f_i > 0 | \text{slang}) - P(f_i > 0 | \text{literal})$.
• Feature Selection: They ranked features by $\Delta_i$ and identified a cross-linguistic core by intersecting the top-100 features across all three languages.
• Geometric Analysis: They analyzed the decoder weight vectors of these shared features to determine if they form a coherent "island" in the latent space (using cosine similarity and "island scores").

C. Causal Intervention: Activation Steering

To prove causality rather than mere correlation, the authors performed activation steering:

• Steering Vector ($v_F$): Constructed by averaging the decoder weights of the cross-linguistic feature set.
• Injection: The vector was injected into the residual stream at specific layers (Layer 9 and Layer 20) during generation: $h'_{L,t} = h_{L,t} + \alpha \cdot v_F$.
• Direction: Positive $\alpha$ steers toward informal; negative $\alpha$ steers toward formal.
• Evaluation: Tested on the three source languages and six zero-shot unseen languages (German, Japanese, Hindi, Thai, Georgian, Amharic) spanning diverse language families and scripts.

3. Key Contributions

1. Methodological: Introduction of a lexically controlled, polysemous dataset designed to isolate pragmatic register from vocabulary bias across three typologically distinct languages.
2. Observational: Identification of a small, geometrically coherent "informal register subspace" within the SAE latent space. This shared core exists across English, Hebrew, and Russian and becomes more distinct in deeper layers.
3. Causal: Demonstration that steering with these shared features causally modulates output formality across all source languages and transfers zero-shot to six unseen languages.

4. Key Results

A. Observational Findings

• Distributed Signal: Informal register is not encoded in a single "slang neuron" but is distributed across many features.
• Cross-Linguistic Core: While most slang-related features are language-specific, a small but statistically robust core of 9 shared features (at Layer 20) was identified across all three languages.
• Geometric Coherence: These shared features form a tight cluster in the decoder space ("island score" of 6.0× at Layer 20), indicating they represent a unified latent direction rather than random co-occurrence.
• Depth Trend: The informal register signal becomes more selective and geometrically coherent in deeper layers (Layer 20) compared to shallower layers (Layer 9).

B. Causal Steering Results

• Source Language Control: Steering with the cross-linguistic features produced strong, statistically significant negative correlations between the steering coefficient ($\alpha$) and output formality scores in English, Hebrew, and Russian ($r \approx -0.7$).
• Zero-Shot Transfer: Crucially, the steering effect transferred to six unseen languages (including Japanese, Thai, and Amharic) without any feature extraction from those languages.
  • Example: Japanese and German showed strong transfer ($r \approx -0.75$), while distant languages like Georgian and Amharic showed moderate but significant transfer ($r \approx -0.36$).
• Lexical Integrity: The model generated culturally specific slang in the target languages (e.g., Japanese slang when steering Japanese), rather than simply translating English slang. This confirms the model is manipulating an abstract concept of "informality," not just copying English tokens.
• Control Experiments: Random feature steering yielded no correlation, confirming the effect is specific to the identified subspace.

5. Significance and Implications

• Mechanistic Evidence of Abstraction: This is the first mechanistic evidence that multilingual LLMs internalize informal register not as a collection of language-specific rules, but as a portable, language-agnostic pragmatic abstraction.
• Universal Pragmatics: The findings suggest the existence of a "universal" circuit for social signaling (in-group vs. out-group communication) that transcends surface lexical forms.
• Safety and Alignment: Understanding these internal representations allows for better control of model behavior. It enables the mitigation of "register gaps" and the detection of harmful intent disguised as casual slang.
• Future Directions: The identified features provide a tool for computational sociolinguistics to map structural register shifts and offer a starting point for mechanistic interpretability research into how LLMs handle social nuance.

6. Limitations

• Model Specificity: Results are specific to Gemma-2-9B-IT; generalization to other architectures or scales is unproven.
• English Bias: The "language-agnostic" subspace appears to be partially anchored in English (vocabulary projections favor English tokens, and extreme steering occasionally causes language switching to English).
• Evaluation: Reliance on GPT-4o-mini for formality scoring in typologically distant languages may introduce Western-centric biases in defining "formality."

In conclusion, the paper demonstrates that LLMs possess a unified, geometrically coherent internal representation of informal register that can be causally manipulated to control tone across a wide variety of languages, even those never seen during feature extraction.