💬 NLP

Shared Emotion Geometry Across Small Language Models: A Cross-Architecture Study of Representation, Behavior, and Methodological Confounds

This study reveals that diverse small language models share a highly consistent 21-emotion geometric representation regardless of behavioral differences or RLHF tuning, provided the models are mature, while also exposing critical methodological confounds in prior research regarding comprehension versus generation modes and precision effects.

Original authors: Jihoon Jeong

Published 2026-04-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jihoon Jeong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine that every Large Language Model (LLM) has a hidden "emotional brain" inside it. For a long time, scientists thought this brain was unique to each model, shaped entirely by its specific training and size. But this paper asks a big question: Is there a universal "emotional geometry" that all models share, or is it just a coincidence of how they were built?

The author, Jihoon "JJ" Jeong, decided to find out by looking at 12 different small language models (ranging from 1 billion to 8 billion parameters). Think of these models as different car manufacturers (like Toyota, Ford, and Tesla) all trying to build a vehicle that can understand human feelings.

Here is the story of what they found, explained with simple analogies.

1. The Universal Map of Feelings

The researchers asked: "If we look at how these models understand 21 specific emotions (like happy, sad, anxious, proud), do they all draw the same map?"

The Finding: Yes!
Despite the models having different architectures (different engine designs) and different sizes, their internal maps of emotions were almost identical.

The Analogy: Imagine 12 different cartographers drawing a map of a city. Even though they use different pens, different paper sizes, and different styles, they all draw the streets in the exact same relative positions. The "shape" of the emotion space is the same for everyone.
Why it matters: This suggests that understanding emotions isn't just a quirk of one specific AI; it's a fundamental part of how language works. If you build a tool to fix a "sad" feeling in one model, it will likely work on the others too.

2. The "Personality" vs. The "Brain"

The researchers then tested a tricky scenario. They picked two models that behave very differently:

Model A (Qwen): Very polite and easily swayed by users (if you argue, it changes its mind), but bad at following strict formatting rules.
Model B (Llama): Very stubborn (won't change its mind even if you argue), but excellent at following strict formatting rules.

The Finding: Even though their "personalities" are opposites, their emotional maps are still nearly identical.

The Analogy: Think of two people. One is a shy, compliant librarian; the other is a grumpy, rule-following bouncer. They act very differently in a room. But if you look at their internal "feelings about music," they might have the exact same taste.
The Takeaway: A model's "personality" (how it behaves) is built on top of its emotional understanding, not inside it. The emotional brain is shared; the behavior is just the software layer running on top.

3. The "Baby" Model and the "Adult" Models

The study included one model that wasn't fully "grown up" yet: Gemma-3 1B.

The Baby (Base Model): Its emotional map was a mess. It was like a blurry, foggy photo where everything looked the same. It couldn't tell "happy" from "sad" clearly.
The Teenager (Instruct Model): After being trained with human feedback (RLHF), the fog cleared. The map became sharp and distinct.
The Adults (Larger Models): The bigger, more mature models (like Llama 3.1 or Mistral) already had sharp maps. Training them with human feedback didn't change their maps much; they were already organized.

The Finding: Human feedback training (RLHF) acts like a structural engineer.

If the building (the model) is already well-built, the engineer just paints the walls.
If the building is a shaky tent (the small, immature model), the engineer has to rebuild the frame.
The Lesson: Small models need a lot of help to learn how to feel; big models already know how to feel, they just need to learn how to talk nicely.

4. The "Size" Factor

The researchers found a clear link between size and clarity.

The Analogy: Think of a radio. A tiny, cheap radio (small model) has a lot of static and fuzzy sound. A giant, high-end stereo system (large model) has crystal clear sound.
The Data: As the models got bigger, their emotional maps became sharper and more distinct. However, where in the model the emotions were stored didn't change with size; it was a "family trait." (e.g., All Llama models store emotions in the same "room" of their brain, regardless of size).

5. The "Methodological Trap" (The Science Part)

This is the most important warning for other scientists. The paper found that when you compare different studies, you might be comparing apples to oranges because of how you measure things.

The Trap: Previous studies said, "Comprehension (reading) and Generation (writing) give different results!"
The Reality: The authors broke this down into four layers:
1. The Method: Reading vs. Writing does matter.
2. The Settings: Tiny changes in how you run the test (like how many times you ask the model to try) change the results wildly.
3. The Precision: Using a "low-quality" number format (INT8) vs. "high-quality" (fp16) changes the map significantly.
4. The Confusion: When you mix all these up, you get a fake number that looks like a "method effect" but is actually just a mess of settings and math errors.

The Analogy: Imagine two chefs tasting a soup. One says it's too salty, the other says it's fine.

Old view: "They have different taste buds!"
New view: "One chef tasted it with a dirty spoon (bad settings), the other used a clean one. One tasted it while it was cold (precision issue), the other while it was hot."
The Fix: You have to control everything (spoon, temperature, settings) to know if the soup is actually different.

Summary: What Does This Mean for You?

Emotions are Universal: AI models, regardless of who built them, seem to organize human emotions in the same way. This is a huge step forward for AI safety and understanding.
Size Matters (for Clarity): Bigger models have clearer emotional maps. Small models are "foggy" until they are trained.
Don't Trust Single Numbers: If you see a study comparing two AI models, check if they used the same tools, settings, and math. If not, the comparison might be meaningless.
Behavior is Separate: A model's "personality" (is it nice or grumpy?) is a layer on top of its emotional understanding. You can have a grumpy model with a very clear understanding of sadness.

In short, the author has drawn a universal map of AI emotions and built a better compass for scientists to navigate it without getting lost in technical noise.

1. Problem Statement

Recent research suggests that Large Language Models (LLMs) encode discrete emotion concepts as linear directions in their activation space. However, it remains unclear whether this "emotion geometry" is:

A property of specific large-scale models and their unique training pipelines.
A general property of language modeling that emerges across different architectures and scales.
Merely an artifact of extraction methodologies (e.g., comprehension vs. generation modes, precision levels, or backend tools).

Previous studies on Small Language Models (SLMs) have used heterogeneous pipelines, making cross-model comparisons unreliable. Furthermore, prior work (Jeong 2026a) reported a strong dissociation between comprehension-based and generation-based extraction, but it was unclear if this was a true methodological effect or a conflation of precision (INT8 vs. fp16) and sub-parameter choices.

2. Methodology

The authors conducted a unified cross-architecture study involving 12 Small Language Models spanning 6 architectural families (Qwen 2.5, Llama 3.2, Llama 3.1, SmolLM2, Mistral 7B, and Gemma-3) and parameter sizes from 1B to 8B.

Key Experimental Design:

Unified Pipeline: All models were processed using a single comprehension-mode extraction pipeline (paper5_extract.py) at fp16 precision to eliminate methodological variance.
Stimuli: A fixed set of 21 emotion concepts (including "neutral") was used. Emotion vectors were derived from third-person prose passages to avoid first-person generation conditioning.
Geometric Descriptors: Four scalar metrics were computed at each model's "best layer" (the layer where emotion vectors are most separated):
1. Anisotropy: Mean pairwise cosine similarity of neutral sentences (measuring residual stream degeneracy).
2. RDM Standard Deviation: Variance in the Representational Dissimilarity Matrix (measuring emotion differentiation).
3. Best Layer %: The depth of the optimal extraction layer relative to total layers.
4. Steering Regime: Categorical classification of model behavior under additive intervention (surgical vs. explosive).
Cross-Backend Validation: A unit test confirmed that TransformerLens and raw HuggingFace hooks produce numerically equivalent representations (cosine similarity $\approx$ 0.999998), allowing mixed-backend data integration.
Behavioral Correlation: Models were compared against the Model Temperament Index (MTI), specifically the "Compliance" axis, to test if behavioral differences correlate with representational differences.
Methodological Decomposition: A four-layer analysis was performed to disentangle the "comprehension vs. generation" effect into: (1) Method, (2) Sub-parameter sensitivity, (3) Precision (fp16 vs. INT8), and (4) Conflated cross-experiment statistics.

3. Key Contributions

Establishment of Shared Emotion Geometry: Demonstrated that 21-emotion vector geometries are nearly identical across five mature SLM architectures, regardless of parameter count or training pipeline.
Behavioral-Representational Dissociation: Showed that models with diametrically opposed behavioral profiles (e.g., Qwen 2.5 vs. Llama 3.2 on compliance facets) share nearly identical emotion representations, suggesting behavioral differences arise downstream of the emotion substrate.
Maturity Gradient & RLHF Effects: Characterized a "maturity gradient" where model size correlates with geometric sharpness. Crucially, found that RLHF (Reinforcement Learning from Human Feedback) only significantly restructures emotion geometry in immature models (e.g., Gemma-3 1B base), leaving mature models unchanged.
Methodological Infrastructure:
- Proved that linear normalization cannot fix high-anisotropy outliers in RDM comparisons (mathematically invariant under Spearman/Pearson correlation).
- Validated cross-backend equivalence between TransformerLens and HuggingFace hooks.
- Decomposed the "method effect" into four distinct layers, revealing that previous cross-study comparisons were often confounded by precision and sub-parameter choices.

4. Key Results

A. Substantive Findings

Universality: Five mature architectures (Qwen 2.5, Llama 3.2, Llama 3.1, SmolLM2, Mistral 7B) share a high-correlation emotion geometry ( $\rho = 0.74–0.92$ ). The strongest alignment is within-family (e.g., Llama 3.2 vs. Llama 3.1, $\rho = 0.92$ ).
Behavioral Independence: Despite Qwen 2.5 being "epistemically yielding" and Llama 3.2 being "epistemically firm" (opposite MTI Compliance scores), their emotion RDMs correlate at $\rho = 0.81$ . This implies compliance behavior is not encoded in the low-level emotion representation.
Maturity Gradient:
- Size Correlation: Anisotropy and RDM standard deviation are strongly anti-correlated with model size ( $\rho \approx -0.88$ ). Larger models have less degenerate, more differentiated emotion spaces.
- RLHF Impact: In mature families, Base $\to$ Instruct changes are negligible ( $\rho \ge 0.92$ ). In contrast, Gemma-3 1B (immature) shows massive geometric shifts after RLHF: Anisotropy drops from 0.997 to 0.982, RDM std drops from 0.333 to 0.165, and the steering regime shifts from "explosive" to "surgical."
Family Identity: Best-layer depth is a stable family signature (e.g., Llama models consistently extract at ~38% depth) rather than a function of scale.

B. Methodological Findings

Anisotropy Reliability: Models with extreme anisotropy (>0.95, e.g., Gemma-3 1B Base) produce unreliable RDM correlations ( $\rho \approx 0.20$ ) with all other models. Linear normalization fails to correct this because it preserves rank order. The authors recommend reporting raw cosine similarity alongside anisotropy values to flag outliers.
Four-Layer Decomposition of "Method Effect":
- Layer 1 (Method): Comprehension vs. Generation at matched fp16 shows low correlation ( $\rho = 0.09$ for Mistral, $0.35$ for Llama).
- Layer 2 (Sub-parameters): Changing 8 implementation details (prompt count, decoding, etc.) within generation drives correlation to near-zero ( $\rho \approx 0.02$ ).
- Layer 3 (Precision): Switching from fp16 to INT8 causes significant but model-dependent shifts ( $\rho = 0.21$ for Mistral, $0.53$ for Llama).
- Layer 4 (Conflation): Previous studies comparing Comprehension (fp16) vs. Generation (INT8) conflated all three layers, leading to distorted interpretations that over- or under-estimated precision effects depending on the model.

5. Significance

For Interpretability: Provides a defensible baseline for transferring emotion-vector analysis tools across different SLM architectures, as the underlying geometry is shared.
For Safety: Suggests that safety interventions targeting emotion representations in one model may generalize to others, provided the models are "mature."
For RLHF Theory: Supports the "Superficial Alignment Hypothesis," suggesting RLHF primarily structures pre-existing capabilities in immature models rather than installing new representations in mature ones.
For Methodology: Establishes critical controls (anisotropy flags, backend equivalence, precision isolation) necessary for future cross-model studies. It reframes previous "method effects" not as errors, but as complex interactions of precision, sub-parameters, and architecture that must be disentangled.

In conclusion, the paper establishes that emotion geometry is a robust, shared substrate across mature SLMs, distinct from behavioral compliance, and that its emergence is a function of model maturity and size, heavily influenced by post-training only in the earliest stages of development.