Technical Summary: ANA – An AI-Driven Multimodal Emotion-Aware Application Based on IFS Therapy

1. Problem Statement and Motivation

Current mental health support systems utilizing Artificial Intelligence are predominantly limited to unimodal, text-based interactions. These systems often lack the capacity to discern complex human emotions that are conveyed through tone, facial expressions, and hand gestures. Furthermore, existing multimodal models frequently suffer from weak integration strategies, such as simple feature concatenation without accounting for inter-modal dependencies, and a lack of alignment with established psychological theories.

Specifically, there is a gap in computational models that integrate multimodal data (text, speech, vision, gestures) with Internal Family Systems (IFS) therapy. IFS posits that the human mind consists of multiple distinct "inner parts" (e.g., the Inner Critic, Protector, Overwhelmed Part) with unique emotional profiles. The paper argues that effectively modeling these psychological constructs requires a structured approach that maps heterogeneous emotional cues to specific psychological constructs, a task that unimodal systems cannot achieve.

2. Proposed Solution: The ANA Model

The authors propose ANA (A Multimodal Application for Modeling Emotional States and Adaptive Interaction), a system designed to mimic IFS therapy concepts. ANA utilizes a multimodal learning paradigm to analyze user inputs across four distinct modalities: Text, Voice, Facial Expressions, and Hand Gestures.

2.1 System Architecture and Workflow

The system operates through a "Reframe Session" workflow:

1. Access Control: Before interaction, the system validates the user's status. If a user has three or more "unhealed" inner characters, access to multimodal inputs is restricted to prevent overwhelming the user.
2. Input Modalities:
   • Video: Extracts text, vocal emotion, hand gestures, and facial expressions simultaneously.
   • Text Only: Processes written input.
   • Voice Only: Processes speech signals and tone.
   • Note: Arabic inputs are automatically translated to English before analysis.
3. Processing: Each modality is processed by a specific encoder to extract features.
4. Fusion: A fusion module integrates these signals to update the user's internal character profile and provide an emotional state assessment.
5. Output: The system identifies the dominant "inner character" and emotional state, storing data for behavioral tracking.

3. Methodology

3.1 Data Preprocessing

• Text: Involves normalization, tokenization, stop-word removal, vocabulary encoding, and padding/truncation to create fixed-length sequences.
• Voice: Includes resampling (16 kHz), noise reduction, silence trimming, framing, and feature extraction (MFCCs, Chroma, Mel-spectrogram, ZCR, RMS, Spectral Centroid). Features are standardized and reduced via PCA.
• Hand Gestures: Uses MediaPipe to detect 21 hand landmarks. Coordinates are normalized (translation and scale) relative to the wrist, flattened into a 42-dimensional vector, and prepared for classification.
• Facial Expressions: Images are converted to grayscale, resized to 48x48 pixels, normalized, and augmented (rotation, shear, zoom) during training.

3.2 Modality-Specific Encoders

The system employs specialized encoders for each input type:

• Text Encoder: A hybrid CNN + BiLSTM model. CNN layers extract local N-gram features, followed by max pooling and BiLSTM layers to capture long-term dependencies. The output classifies the "inner character" (e.g., Inner Critic, Protector).
• Voice Encoder: Utilizes handcrafted acoustic features (MFCCs, pitch, energy) reduced via PCA and classified using a K-Nearest Neighbors (KNN) algorithm to recognize emotional states.
• Hand Gesture Encoder: A Multi-Layer Perceptron (MLP) classifier that processes normalized landmark vectors to recognize static and dynamic gestures.
• Facial Expression Encoder: A Convolutional Neural Network (CNN) (based on Caffe SSD/ResNet-10 for detection) that classifies emotions from facial images.

3.3 Fusion Strategies

The paper evaluates three fusion approaches:

1. Early Fusion: Concatenating features before classification.
2. Late Fusion: Each modality makes an independent prediction, which are then combined via weighted voting.
3. Hybrid Fusion: A combination of feature-level and decision-level fusion.
   The results indicate that Late Fusion is the most effective strategy, as it allows each modality to make independent decisions before aggregating them, making the system robust to missing or noisy inputs.

4. Experimental Results

4.1 Datasets

The study utilized diverse datasets for training and evaluation:

• Text: Aggregated from sources like Counseling, DailyDialog, GoEmotions, and RedditMentalHealth (360,000 samples).
• Voice: Benchmark datasets including RAVDESS, TESS, CREMA-D, SAVEE, and EMO-DB (4,800 samples).
• Face: FER-2013, AffectNet, and CK+ (35,887 samples).
• Hand Gestures: A custom dataset (ANAHandGestures.csv) created using MediaPipe (6,153 samples).

4.2 Performance Metrics

The unimodal models achieved the following accuracies:

• Voice Emotion Recognition (KNN): 98% (Highest performance; attributed to strong acoustic separability).
• Text Inner Character Classification (CNN+BiLSTM): 97%.
• Hand Gesture Recognition (MLP): 93%.
• Facial Emotion Recognition (CNN): 88% (Lower due to sensitivity to lighting and expression overlap).

4.3 Analysis

• Voice and Text were identified as the strongest predictors of emotional state and inner character.
• Facial and Hand Gestures served as complementary signals, providing non-verbal feedback that helped resolve ambiguities in text or voice (e.g., detecting a "Protector" hiding sadness behind a smile).
• Confusion Matrices showed diagonal dominance across all modalities, with errors primarily occurring between similar emotional classes (e.g., neutral vs. sad).
• ROC Analysis confirmed high AUC values for all modalities, indicating strong class separation.

5. Key Contributions

The paper outlines the following specific contributions:

1. Structured Multimodal Model: Development of a system integrating four modalities (text, voice, face, gesture) specifically within the framework of Affective Computing and IFS therapy.
2. Specialized Encoders:
   • A CNN-BiLSTM text encoder for inner character classification.
   • A PCA-KNN voice emotion recognition system.
   • A CNN-based facial emotion recognition system.
   • A lightweight MediaPipe-based hand gesture recognition module.
3. Multimodal Fusion: Implementation of a fusion system that combines these modalities to recognize both emotional states and inner psychological characters.
4. Deployment: A functional web inference service (Flask-based) with a mobile interface that handles real-time multimodal analysis, including Arabic-to-English translation and access control based on user psychological status.

6. Significance and Claims

The authors claim that the ANA model extends beyond common chatbot-based mental health systems by creating a richer, embodied interaction model. By leveraging multimodal learning, the system achieves a more robust understanding of the user than unimodal systems.

• Validation of Multimodality: The study demonstrates that combining modalities compensates for individual weaknesses (e.g., text ambiguity or facial occlusion), effectively doubling the system's understanding capability compared to single-input modes.
• Therapeutic Alignment: The system successfully maps technical AI outputs (emotion recognition) to psychological constructs (IFS inner parts), bridging the gap between deep learning and clinical therapy models.
• Practical Utility: The deployment architecture supports real-time processing and handles real-world challenges such as noise, lighting variations, and missing modalities through adaptive fusion strategies.

The paper concludes that while individual modalities have limitations (e.g., facial recognition accuracy drops in poor lighting), the integrated multimodal approach provides a comprehensive and context-aware assessment of a user's internal psychological state, validating the effectiveness of the proposed model for personalized mental health support.