Emotional regulation improves deep learning-based image classification

Technical Summary: Emotional Regulation Improves Deep Learning-Based Image Classification

Problem Statement
Existing methods for emotion-augmented deep learning primarily rely on objective neurophysiological factors, such as anatomical brain structures (e.g., amygdala, prefrontal cortex) and hormonal processes (e.g., dopamine, anxiety, confidence). While these approaches have shown promise, they neglect the role of subjectivity in emotional responses. Human emotional reactions are not solely determined by physiological states but are significantly influenced by subjective experience and emotional history. Furthermore, previous studies in this domain often lacked end-to-end learning architectures or were not validated across diverse sets of emotional stimuli, limiting the robustness and generalization of the results. The paper addresses the gap in modeling artificial emotion by incorporating subjective experience rather than strictly neurophysiological definitions.

Methodology: The Emotional Regulation Framework
The study introduces Emotional Regulation, a framework designed to model emotion in deep learning through "artificial subjective experience." The core mechanism involves balancing non-emotional and emotionally-influenced predictions based on a learned affective state.

1. Architecture: The framework employs three encoders:

   • Non-emotional Encoder ($E_n$): The original backbone (ResNet-50 or ViT-B/16) producing non-emotional representations.
   • Emotionally-influenced Encoder ($E_i$): Learns embeddings by concatenating features from the Non-emotional Encoder with those from the Emotional Encoder.
   • Emotional Encoder ($E_e$): A frozen, pre-trained component that learns an "emotional history" from a dataset of stimulus-emotion instances. It outputs an emotional spectrum vector ($e$).

2. Regulation Process ($R$): A learnable regulation layer weights the outputs of the Non-emotional ($p_n$) and Emotionally-influenced ($p_i$) models based on the emotional outcome ($\hat{e}$). The final prediction ($p_r$) is computed as:
   $$p_r = \hat{e} \cdot h_r \cdot p_n + (1 - \hat{e} \cdot h_r) \cdot p_i$$
   Where $h_r$ is a learnable weight vector representing the regulation intensity for specific emotions.

3. Training Phases:

   • Emotional Pre-training: The Emotional Encoder is optimized on a specific emotional dataset (e.g., EMOd, Diffused-EMOd, Abstract, EmoSet) to learn the mapping between visual stimuli and emotional categories.
   • Pre-training: The Non-emotional and Emotionally-influenced encoders are optimized on the downstream task (or kept frozen).
   • Training: The full system is optimized, learning the regulation process ($h_r$) to balance the two prediction streams.

4. Experimental Configurations: The authors evaluated several variations, including non-learnable regulation (fixed weights), learnable regulation with fixed pre-trained encoders, and fully learnable end-to-end configurations. Experiments utilized ResNet-50 and ViT-B/16 backbones pre-trained on ImageNet, fine-tuned on CIFAR-10 and CIFAR-100.

Key Contributions

1. Novel Modeling Approach: Proposes Emotional Regulation as a method to define affective states in deep learning through artificial subjective experience, bridging the gap left by neurophysiological-only models.
2. End-to-End Extension: Extends the Emotional Regulation framework to full end-to-end learning architectures, moving beyond the limitations of previous non-end-to-end implementations.
3. Robust Validation: Validates the approach across four distinct emotional datasets (real-life stimuli, synthetic diffusion-generated images, abstract paintings, and large-scale balanced sets) to ensure generalization.
4. State-of-the-Art Performance: Demonstrates that Emotional Regulation outperforms existing emotion-augmented solutions on large-scale image classification benchmarks (CIFAR-10 and CIFAR-100).

Results
Extensive experimentation revealed that learnable regulation strategies consistently outperformed original backbones and non-emotional baselines:

• Performance Gains: On CIFAR-10, the best configuration (Rand-Learn-E-Reg with ViT) improved accuracy by 0.34% over the ViT backbone. On CIFAR-100, improvements reached 1.02% for ViT and 3.19% for ResNet.
• Comparison with Prior Work: The proposed method significantly outperformed the AEmNN model (a previous state-of-the-art using emotional backpropagation) on CIFAR-10 and CIFAR-100, achieving accuracy scores of 97.32% and 86.24% respectively, compared to the previous 54.76% and 27.53%.
• Regulation Dynamics: The study found that the optimal regulation intensity is not fixed; it varies significantly depending on the emotional history (pre-training dataset) and the downstream task. For instance, models trained on Diffused-EMOd showed significant regulation of semantically contrasting emotions (e.g., fear vs. happiness), while others favored specific valences.
• Ablation Insights: Learnable regulation consistently provided improvements, whereas non-learnable (fixed) regulation did not guarantee gains over the backbone. The results suggest that a model entirely influenced by emotion does not always improve learning; a regulation mechanism is necessary to modulate affective cognition for better generalization.

Significance and Claims
The paper claims that Emotional Regulation establishes a new state-of-the-art in emotion-augmented deep learning for large-scale vision datasets. The study provides evidence that subjective experience, modeled as an artificial emotional history, is a significant factor in improving machine learning optimization.

The authors emphasize that their work does not claim to replicate human cognition or equate artificial results with empirical human studies. Instead, the significance lies in demonstrating that introducing "artificial subjectivity"—via the balancing of emotional and non-emotional representations through a regulation mechanism—enhances the generalization capabilities of deep neural networks. The framework suggests that emotion acts as a cognitive factor that, when properly regulated, improves the learning of visual stimuli, offering a promising path for future architectures in computer vision and human-robot interaction.