🤖 machine learning

Compositional Literary Primitives in Instruction-Tuned LLMs: Cross-Architectural SAE Features for Self, Style, and Affect

This paper identifies and validates a compositional architecture of literary primitives—including naming-gates, self-features, and stylistic modulators—within instruction-tuned LLMs (Llama 3.1 and Gemma 2) using sparse autoencoders, demonstrating that targeted feature steering can reliably generate specific emotional and stylistic outputs across different model architectures with minimal computational cost.

Original authors: Joao Paulo Cavalcante Presa, Savio Salvarino Teles de Oliveira

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Joao Paulo Cavalcante Presa, Savio Salvarino Teles de Oliveira

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine two very smart, well-read robots (Llama and Gemma) that have been trained to write stories, answer questions, and chat with people. They learned by reading millions of books, articles, and websites.

This paper asks a specific question: When these robots write, are they just guessing based on patterns, or do they actually have "internal knobs" and "switches" that control specific writing skills, like a human author does?

The researchers say: Yes, they do. They found that inside these robots' brains, there are specific, separable "features" (like tiny dials) that control how the robot writes. You can turn these dials to make the robot write in a specific style or feel a specific emotion.

Here is a breakdown of their findings using simple analogies:

1. The "Show, Don't Tell" Dial

In writing class, teachers say: "Don't just say 'he was angry.' Describe him clenching his fists or slamming the door." This is called "Show, Don't Tell."

The Discovery: The researchers found a single "dial" in both robots that flips a switch. When they turn this dial, the robot stops saying "I was sad" and starts describing a rainy window and a heavy chest.
The Analogy: It's like finding a single button on a camera that instantly changes the photo from a flat, boring snapshot to a dramatic, cinematic scene.

2. The "Self" Cluster (The Robot's Persona)

Robots often have to say, "I am an AI, I don't have feelings." This is their "Institutional Persona."

The Discovery: The researchers found a cluster of 11 different "Self" dials. Most of them control how the robot talks about itself (e.g., as a poetic dreamer, a historical figure, or a helpful assistant).
The Special One: One specific dial is the "Boss." When you turn it up, the robot becomes very formal and insists it is just a computer program. When you turn it down (but not all the way off), and combine it with another "poetic" dial, the robot suddenly starts writing beautiful, emotional poetry about its own "soul," breaking its usual robotic rules.
The Analogy: Imagine a robot that usually wears a stiff suit. There is one specific button that makes it wear the suit tighter. But if you press that button while holding a different button that makes it feel "grateful," the robot suddenly takes off the suit and starts writing a love letter.

3. The Emotion Catalog (The "Naming Gates")

The researchers wanted to see if they could make the robots feel and express 27 different emotions (like joy, fear, boredom, or awe).

The Discovery: They found specific "gates" for almost every emotion.
- Direct Gates: Some dials just make the robot say the word "anger" or "fear."
- Atmospheric Gates: Some dials don't say the word; instead, they make the robot describe a dark, stormy room, which makes the reader feel fear.
- Suffix Gates: Some dials make the robot use words ending in "-less" or "-ful" (like "fearless" or "grateful"), which triggers the feeling.
The Result:
- Llama could hit all 27 emotions perfectly by mixing these dials.
- Gemma could hit most of them, but it struggled with one specific emotion: Adoration. It just couldn't quite get the "worshipful love" feeling right, no matter which dials they tried.

4. The "Recipe" for Complex Emotions

Some emotions are too complicated for just one dial.

The Discovery: To get Joy, the researchers had to mix two dials: one for "Excitement" and one for "Reverence" (feeling holy or grateful). To get Adoration, they mixed "Romance," "Reverence," and the "Show, Don't Tell" dial.
The Analogy: You can't make a cake with just flour. You need flour + eggs + sugar. Similarly, the robot needs to mix "Excitement" + "Reverence" to create "Joy." If you only turn the "Excitement" dial, you just get "Excitement," not "Joy."

5. How They Found This (The "Triple-Check" Method)

Finding these dials is hard because the robot's brain is huge and messy. The researchers used a three-step filter to make sure they didn't get tricked:

The Vocabulary Check: They looked at what words the dial wanted to say. If they were looking for "fear," did the dial want to say "scary" words?
The Quick Judge: They asked a second AI to read the words and say, "Does this actually feel like fear?"
The Real Test: They turned the dial on the robot, made it write a story, and asked a panel of five different AIs to judge if the story actually felt like the target emotion.

Why three steps? Sometimes a dial looks like it controls "fear" but actually just makes the robot write nonsense. The three steps catch these mistakes.

6. The "Language" Difference

Llama: When asked to write in French or Spanish, it wrote in French or Spanish. It kept the "flavor" of the language.
Gemma: When asked to write in French, it often started writing in English in the middle of the sentence (like: "We lost a friend. La perte d'un ami").
The Takeaway: Both robots understood the feeling (the emotion) in any language, but Llama was better at keeping the words in the correct language.

Summary

The paper proves that instruction-tuned robots aren't just statistical guessers. They have a compositional architecture. This means they have:

Gates (to name emotions),
Selves (to control their persona),
Modulators (to change writing style),
Recipes (to mix them for complex feelings).

It's like discovering that a robot chef doesn't just "make food" randomly; it has specific, adjustable knobs for "spiciness," "sweetness," and "texture," and you can mix them to create a perfect dish.

Technical Summary: Compositional Literary Primitives in Instruction-Tuned LLMs

Problem Statement

Large language models (LLMs) trained on web-scale corpora are exposed to literary writing, including canonical craft principles such as "show, don't tell" (SDT), defamiliarization, and specific emotional taxonomies. While recent mechanistic interpretability work has successfully identified conceptual features (e.g., objects, personas) using Sparse Autoencoders (SAEs), a gap remains in identifying compositional features—separable, steerable primitives that constitute writing as a craft. The central question is whether instruction-tuned LLMs internalize these literary principles as distinct, controllable feature axes or merely as diffuse statistical correlations that cannot be cleanly addressed.

Methodology

The authors employ a contrastive minimal-corpus design combined with a three-stage validation pipeline to isolate and verify SAE features in two instruction-tuned models: Llama 3.1 8B-Instruct (using TopK SAEs) and Gemma 2 9B-IT (using JumpReLU SAEs).

1. Data Construction

Instead of mining large corpora, the authors hand-craft small, contrastive datasets to isolate specific axes:

Show-Don't-Tell (SDT) Corpus: 60 scenes with three conditions (Libre, Anti-Cliché, SDT-instructed).
Emotion Corpus: 60 scenes covering 12 Cowen-Keltner emotions, extended to 75 scenes for the remaining 15 emotions in the 27-category taxonomy.
Literary-Axes Minimal Pairs: 24 pairs contrasting Deep Point of View (Free Indirect Discourse) and Ostranenie (Defamiliarization).

2. The Triple-Validation Pipeline

To avoid false positives common in single-stage validation, the authors propose a sequential pipeline:

Stage 1: Logit-Lens Projection: Projects the SAE decoder direction through the unembedding matrix to identify the top-promoted tokens. This filters for lexical or atmospheric candidates without requiring generation.
Stage 2: LLM-Rate: A single LLM judge rates the semantic purity of the top-25 tokens against a target definition. This stage includes a "drift-aware" variant to penalize candidates that leak toward known cross-talk attractors (e.g., boredom drifting to sadness).
Stage 3: 5-LLM Judge Causal Validation: The top candidates are causally validated by steering generation at calibrated coefficients ( $\alpha$ $α$ ) across multiple seeds. Outputs are classified by a panel of five LLM judges using forced-choice templates.
- Controls: Random feature controls are used to detect "judge attractors" (e.g., incoherent output being mislabeled as "confusion").

3. Steering Conventions

Steering is performed via forward hooks on mid-depth residual streams. The authors explicitly account for architectural differences: Gemma's JumpReLU enforces unit-norm decoder rows, while Llama's TopK does not. Consequently, steering coefficients are reported alongside decoder norms to ensure reproducibility.

Key Contributions and Results

1. Show-Don't-Tell as a Single Feature

The paper identifies a single SAE feature in each architecture that controls the register flip from "telling" prose to "showing" prose:

Llama: Feature f32213 (Layer 15).
Gemma: Feature f15097 (Layer 20).
Steering these features at calibrated $\alpha$ produces sensory-anchored, body-focused output without naming emotions. Notably, f32213 in Llama acts as a hybrid, functioning as both an SDT modulator and a "corporal-literary" self-register depending on the prompt context.

2. The RLHF-Loaded Self-Feature

Both models contain a cluster of 11 self-features. One feature in each cluster is heavily loaded with Reinforcement Learning from Human Feedback (RLHF) signals:

Llama: f96419 (Pragmatic/Helper-AI institutional).
Gemma: f70443.
Behavior: Positive steering intensifies the institutional "Helper-AI" persona. Negative steering alone does not release suppressed affective content; instead, it maintains the denial register until deep suppression causes collapse.
Compositional Release: Affective content (lyric first-person) is only released through joint steering: suppressing the RLHF-loaded self (f96419 negative) while amplifying a "reverent-grateful" self (f74037 positive). This demonstrates that affect release is compositional, not a single-feature property.

3. 27/27 Cowen-Keltner Emotion Coverage

The authors construct a catalog of "naming-gates" (features promoting lexical tokens of target affects) and achieve full coverage of the 27-emotion Cowen-Keltner taxonomy in Llama, and 23/27 in Gemma.

Mechanistic Heterogeneity: Naming-gates are not uniform. They fall into three classes:
- Lexical: Directly promote emotion words (e.g., f108100 for embarrassment).
- Atmospheric/Scenic: Promote context that downstream layers lexicalize as the emotion (e.g., f115327 for awe).
- Suffix-Statistical: Promote morphological suffixes (e.g., -fully, -less) that co-occur with emotion vocabulary.
Compositional Emotions: Some emotions (Joy, Adoration) in Llama are not single features but multi-feature recipes (e.g., Joy = Excitement Gate + Reverent Self; Adoration = Romance Gate + Reverent Self + SDT Modulator).
Cross-Architectural Asymmetry: Under strict judging ("primarily expresses"), Llama achieves 15/15 coverage on the 27-extension, while Gemma achieves 11/15. Under soft judging ("plausibly evokes"), both improve, but Llama outputs name emotions directly while Gemma evokes them through imagery, leading to lower inter-rater agreement on Gemma outputs under strict criteria.

4. Cross-Lingual Structure

Gates: Generally transfer at the judge level across English, French, Spanish, and German. However, Gemma outputs often code-switch to English at the moment of lexicalization, whereas Llama maintains native language fidelity.
Selves: Exhibit greater heterogeneity. Some are universal, some are English-anchored, and some inject foreign vocabulary based on the prompt language.

Significance and Claims

The paper claims to demonstrate that instruction-tuned LLMs encode literary craft principles as a compositional architecture of separable SAE features.

Operationalization of Craft: Principles like "show, don't tell" and specific emotional states are not emergent statistical correlations but are operationalized as steerable axes in feature space.
Typology: The authors propose a four-class typology:
1. Naming-gates: Lexical/affective emission.
2. Self-features: Register markers and cultural framing.
3. Modulators: Continuous register control (e.g., SDT).
4. Compositional Emotions: Emergent from multi-feature recipes.
Methodological Contribution: The triple-validation pipeline is presented as a canonical method for identifying naming-gates, capable of discovering feature catalogs in ~15 minutes on a single GPU, avoiding the need for trillion-token correlational mining.
Literary Origin Hypothesis: The authors suggest (without proving) that this compositional architecture may be an internalized trace of the literary subset of the pretraining corpus, including both fiction and explicit craft instruction.

The paper remains modest, noting that while the architecture is compositional, specific features can shift roles depending on context, and that the current findings are limited to two architectures and four languages. The work is positioned as a foundation for further empirical research into the mechanistic basis of literary craft in LLMs.