Evaluation of Conversational Agents: Understanding Culture, Context and Environment in Emotion Detection

Technical Summary: Evaluation of Conversational Agents – Understanding Culture, Context and Environment in Emotion Detection

Problem Statement
The paper addresses a critical gap in Human-AI interaction, specifically regarding the deployment of conversational agents and emotion recognition systems within Black African societies. While generalized solutions for mimicking human emotions (including sarcasm) exist, they often fail to account for geographical location and cultural differences. The authors highlight that current technologies, such as facial biometrics, frequently underperform on minority demographics, particularly Black populations. This lack of cultural specificity creates ethical risks and reduces the credibility of AI systems in diverse global contexts. The core problem is the inability of existing models to accurately detect micro-expressions, sarcasm, and culturally specific emotional cues in African communities due to data bias and a lack of context-aware processing.

Methodology
To address these challenges, the authors developed a hybrid emotion prediction model that integrates both speech (audio) and image (visual) data. The methodology encompasses the following technical components:

• Dataset Construction: The study utilized standard open-source datasets (FER-2013, CREMA-D, RAVDESS, SAVEE, TESS) and augmented them with 1,000 locally generated images from Ghana. This local dataset included specific counts for seven emotions (Happy, Surprise, Fear, Angry, Sad, Neutral, Disgust) to ensure representation of the African (Ghanaian) community.
• Pre-processing:
  • Visual: Images were converted from RGB to grayscale using a luminosity method to optimize Convolutional Neural Network (CNN) performance. Facial landmarks (68 points) were extracted using the Dlib library to isolate features such as the eyes, mouth, and jawlines.
  • Audio: Audio files were converted to .wav format and analyzed for frequency-time transformations. Features extracted included Zero-Crossing Rate (ZCR), Chroma Vector, Chroma Deviation, Mel Frequency Cepstral Coefficients (MFCC), Root Mean Square (RMS), and MelSpectrograms.
• Model Architecture:
  • The system employs a 3-layer Convolutional Neural Network (CNN) for both image and audio classification. The architecture includes 2D convolutional blocks, batch normalization, ReLU activation, max-pooling (size 2), and dropout layers to prevent overfitting.
  • The model was trained using the Adam optimizer with a learning rate of 0.001.
• Novel Algorithm (AFME): A key methodological contribution is the Audio-Frame Mean Expression (AFME) algorithm. This mathematical approach validates neural network outputs by analyzing sub-videos (5–8 seconds) to capture micro-expressions. It treats detected emotions as discrete random variables from both audio and image streams. By mapping these emotions onto Plutchik's wheel of emotion, the algorithm assigns weights to detect "near values" (emotions lying between two primary emotions) and identifies sarcasm when audio and visual cues are at extreme contrasting ends of the emotional spectrum.

Key Contributions

1. Cultural Data Augmentation: The creation and integration of a locally sourced Ghanaian image dataset to mitigate the underrepresentation of Black demographics in existing emotion recognition datasets.
2. Hybrid Modality: The implementation of a dual-stream approach (speech and image) to improve detection accuracy, acknowledging that non-verbal cues and tone are crucial for understanding context and sarcasm.
3. AFME Algorithm: The introduction of the Audio-Frame Mean Expression algorithm, which serves as a post-processing validation layer to reduce misclassifications (false positives/negatives) and specifically detect sarcasm by analyzing the divergence between audio and visual emotional signals.
4. Bias Mitigation: A focus on pre-processing and post-processing stages to address algorithmic bias, moving beyond simple model training to include data balancing and re-weighting techniques.

Results
The proposed model achieved accuracy rates ranging between 85% and 96%.

• Baseline Performance: Initial CNN testing without the AFME algorithm showed varying accuracies, with "Happy" achieving 98% and "Anger" 83.18% on the test set.
• Post-AFME Performance: After applying the AFME algorithm, overall performance metrics improved significantly. For instance, the accuracy for "Disgust" rose from 89.64% to 94.64%, and "Sad" improved from 88.38% to 97.34%. The "Happy" class reached 100% accuracy.
• Sarcasm Detection: The system successfully identified sarcasm in test cases where visual expressions (e.g., smiling) contradicted audio content (e.g., angry tone), a capability standard models often lack.
• Comparison: The final model (CNN + AFME) achieved an overall accuracy of 95.7%, outperforming several existing models cited in the literature (e.g., Syed [32] at 95.6%, Giannopoulos [20] at 83%).

Significance and Claims
The paper claims that its primary significance lies in maintaining the credibility of emotion recognition systems within conversational AIs, particularly for Black African societies. The authors argue that standalone image or speech models are insufficient for accurate emotion prediction on Black faces and that cultural context is essential for ethical AI deployment.

The study emphasizes that:

• Ethical Necessity: Ignoring cultural and geographical factors in AI development exacerbates risks and limits the utility of technology in diverse societies.
• Technical Improvement: The integration of the AFME algorithm and the use of localized data significantly reduce bias and improve the detection of complex emotional states like sarcasm and micro-expressions.
• Future Direction: The authors suggest that while their model improves accuracy, further work is needed to access larger, well-structured African datasets and to apply formal bias mitigation frameworks (such as IBM's AI Fairness 360) to validate and refine the results further.

The paper concludes that by balancing datasets and incorporating the AFME approach, it is possible to design more competent and trustworthy conversational agents that can perceive and respond to emotions in a culturally aware manner.