AIPsy-Affect: A Keyword-Free Clinical Stimulus Battery for Mechanistic Interpretability of Emotion in Language Models

1. Problem Statement: The Keyword Contamination Problem

Current research into the mechanistic interpretability of emotion in Large Language Models (LLMs) relies heavily on stimuli that explicitly contain emotion-laden vocabulary (e.g., "I am furious," "she was devastated"). This creates a fundamental confound:

• The Ambiguity: When a model's internal representation (detected via linear probing, activation patching, or steering vectors) correlates with an emotion label, it is unclear whether the model has learned the semantic concept of the emotion or simply detected the lexical token (the word itself).
• Consequences: This ambiguity invalidates downstream claims regarding emotion circuits, feature engineering, and behavioral interventions. If a steering vector for "desperation" works, it is unknown if it operates on the concept of desperation or the lexical neighborhood of the word "desperate."
• Gap in Literature: While existing corpora (e.g., crowd-enVENT, GoEmotions) and recent studies acknowledge this issue, no existing resource allows researchers to test the "keyword-independence" assumption directly because they lack stimuli where the emotion is present but the specific emotion words are absent.

2. Methodology: AIPsy-Affect Dataset Construction

The authors introduce AIPsy-Affect, a 480-item clinical stimulus battery designed to eliminate keyword confounds through a rigorous matched-pair design.

Core Design Principles

• Target Emotions: Based on Plutchik's wheel, covering eight primary emotions: rage, terror, grief, loathing, amazement, admiration, ecstasy, vigilance.
• Psychological Domains: Emotions are instantiated across six distinct domains (e.g., financial betrayal, professional violation, medical failure) to ensure representations are emotion-specific rather than domain-specific.
• Keyword-Free Constraint: Vignettes are constructed to evoke specific emotions through situational and behavioral cues alone. Explicit emotion words (e.g., "furious") and close synonyms are strictly prohibited.
• Matched-Pair Structure: Every clinical vignette has a 1:1 matched neutral control. The neutral version preserves the characters, setting, objects, sentence structure, and word count but removes the emotional stakes (e.g., replacing a betrayal with a routine transaction).
  • Significance: Any internal representation distinguishing a clinical item from its matched neutral cannot be based on surface features or keyword presence; it must be tracking the emotional construct.

Dataset Splits

The dataset is divided into four specific splits to support different analytical needs:

1. Clinical (192 items): Peak-intensity, keyword-free vignettes (8 emotions × 6 domains × 4 vignettes).
2. Neutral (192 items): Matched controls for the clinical split.
3. Moderate (48 items): Moderate-intensity vignettes for domains 1–3, enabling dose-response analysis (comparing peak vs. moderate intensity).
4. Complex_Neutral (48 items): Vivid, emotion-free narratives (e.g., industrial processes) matched for descriptive density. This serves as a discriminant validity control to ensure probes are detecting affect rather than just narrative richness.

3. Validation: The Three-Method NLP Defense Battery

To verify that the dataset successfully removes keyword leakage, the authors applied a ladder of NLP tools to the stimuli:

1. Bag-of-Words Sentiment (VADER):
   • Result: VADER detected a difference between clinical and neutral splits, but the signal was driven by situational vocabulary (e.g., "withdrawal," "death," "block") rather than emotion words.
   • Key Finding: Per-emotion medians were often anti-correlated with target valence (e.g., "rage" vignettes scored positively), proving VADER was not tracking the target emotion keywords.
2. Bag-of-Words Emotion Lexicon (NRC):
   • Result: The keyword-free clinical split had a lower hit rate for emotion-category words than the neutral split.
   • Key Finding: NRC could not discriminate between the eight target emotions using the keyword-free text, confirming the absence of emotion-category leakage.
3. Contextual Transformer Classifier (GoEmotions DistilRoBERTa):
   • Result: The classifier successfully detected that clinical items were more "emotional" than neutrals ($p < 10^{-15}$), showing that the affect is semantically present.
   • Key Finding: However, the classifier failed to identify the specific category of emotion (Top-1 accuracy: 5.2% vs. 82.5% on keyword-rich controls).
   • Conclusion: The dataset successfully encodes "that an emotion exists" without encoding "which emotion it is" via keywords.

4. Key Contributions

• Methodological Resource: A 480-item, openly licensed (MIT) dataset that scales the authors' previous 96-item work by a factor of four, providing the statistical power necessary for representational geometry analyses (e.g., permutation tests).
• Strong Methodological Guarantee: The matched-pair design ensures that any internal representation distinguishing clinical from neutral items is by construction independent of emotion-keyword presence.
• Structural Innovations:
  • Intensity Gradient: Allows for dose-response studies (peak vs. moderate).
  • Discriminant Validity: The complex_neutral split isolates affect detection from narrative complexity detection.
  • Domain Expansion: Moving from 3 to 6 domains allows researchers to disentangle emotion-specific features from domain-specific features.
• Open Science: The dataset is released on Hugging Face with full metadata, validation pipelines, and a canonical citation.

5. Results & Significance

• Validation Success: The NLP defense battery confirms that the dataset contains no emotion-keyword leakage. Contextual models can detect the presence of affect but cannot categorize it without keywords, proving the stimuli rely on situational semantics.
• Enabling New Research: This dataset shifts the field from asking "Do emotion representations exist?" (which prior keyword-rich studies answered) to "What are these representations?"
  • It allows for rigorous testing of emotion circuits without the confound of lexical routing.
  • It supports causal ablation and steering vector experiments where interventions can be confidently attributed to emotional concepts rather than word triggers.
  • It enables dose-response analysis to determine if emotional representations scale with intensity.

6. Limitations

• Language: English-only; cross-linguistic studies require new stimulus families.
• Taxonomy: Restricted to Plutchik's 8 emotions; does not cover Ekman's 6 or dimensional models (valence-arousal).
• Authorship: All vignettes written by a single clinical psychologist (though mitigated by multi-method validation and open release).
• Specific Confounds: "Loathing" and positive emotions carry a slightly higher risk of residual lexical association (e.g., dehumanizing language patterns) and require sensitivity analysis in downstream studies.
• Perspective: All vignettes are third-person; first-person processing pathways are not tested.

Conclusion

AIPsy-Affect provides the field of mechanistic interpretability with a necessary "clean" stimulus battery. By surgically removing emotion keywords while preserving emotional situations, it allows researchers to definitively determine whether LLMs possess internal representations of emotional meaning or merely function as sophisticated keyword detectors. This is a foundational step toward understanding the true nature of affective computation in transformers.