Technical Summary: EmoAra – Emotion-Preserving English Speech Transcription and Cross-Lingual Translation with Arabic Text-to-Speech

1. Problem Statement

The project addresses critical challenges in cross-linguistic communication within the banking sector, specifically the inability of automated customer service systems to effectively interpret emotional context across language barriers. While customer emotions (e.g., urgency, satisfaction, or dissatisfaction) are vital for service quality, existing systems often fail to detect these nuances in speech, particularly when translating between English and Arabic. This gap leads to miscommunication, reduced customer satisfaction, and a lack of empathetic response. The objective is to develop an end-to-end system that not only transcribes and translates spoken language but also preserves the emotional integrity of the original interaction, enabling bank representatives to respond appropriately regardless of language differences.

2. Methodology

The proposed system, EmoAra, integrates four distinct components into a coherent pipeline: Speech Emotion Recognition (SER), Automatic Speech Recognition (ASR), Machine Translation (MT), and Text-to-Speech (TTS).

2.1 Speech Emotion Recognition (SER)

• Model Architecture: A Convolutional Neural Network (CNN) implemented in TensorFlow/Keras was selected as the primary model for emotion detection. While LSTM and ResNet50 were tested, the CNN demonstrated superior performance.
• Data Preprocessing: The system utilizes the RAVDESS dataset (1,440 audio files, 8 emotion classes). To enhance generalization, data augmentation techniques were applied, including adding Gaussian noise, pitch shifting, time stretching, and time shifting.
• Feature Extraction: Audio waveforms were converted into numerical representations using:
  • Zero-Crossing Rate (ZCR)
  • Root Mean Square Energy (RMSE)
  • Mel-Frequency Cepstral Coefficients (MFCCs)
• Training: The CNN employs 1D convolutional layers with ReLU activations, batch normalization, max pooling, and dropout layers to prevent overfitting. The model uses the Adam optimizer and categorical cross-entropy loss.

2.2 Automatic Speech Recognition (ASR)

• Model: The system utilizes Whisper (Base variant), a pre-trained transformer-based encoder-decoder model by OpenAI.
• Function: It transcribes spoken English audio into text, chosen for its robustness against accents, noise, and background disturbances.

2.3 Machine Translation (MT)

• Model: The system employs MarianMT, a pre-trained neural machine translation model by Helsinki-NLP, fine-tuned for English-to-Arabic translation.
• Dataset Strategy: To overcome the limitations of small datasets for training custom Transformers, the authors fine-tuned MarianMT on a combination of:
  1. A general English-Arabic translation dataset (24,000+ samples).
  2. A custom Banking 77 dataset (10,000 sentences), originally in English and translated to Arabic using Google Translator and MyMemoryTranslator, then verified for accuracy.
• Optimization: Hyperparameters such as learning rate, batch size, and beam search width were optimized. The model uses gradient accumulation and mixed precision (FP16) for efficiency.

2.4 Text-to-Speech (TTS)

• Model: The final stage uses MMS-TTS-Ara, a pre-trained model optimized for Arabic.
• Pipeline: It converts the translated Arabic text into natural speech using a text encoder (Transformer-based), a sequence generator (mapping to mel-spectrograms), and a neural vocoder (HiFi-GAN).
• Emotion Preservation: The system aims to retain the emotional context detected in the initial SER stage during the synthesis of the Arabic response.

3. Key Contributions

The paper highlights three primary contributions:

1. Emotion in Cross-Linguistic Interactions: Unlike many existing translation systems that ignore emotional tone, this project explicitly integrates emotion detection to ensure the target language (Arabic) reflects the emotional nuance of the source (English), fostering empathy in service delivery.
2. Practical Application in Banking: The system is specifically tailored for the banking and finance industry, addressing the unique need to understand customer emotions (urgency, satisfaction) across language barriers to improve service responsiveness.
3. End-to-End Integration: The project successfully unifies SER, ASR, MT, and TTS into a single pipeline, leveraging transfer learning and pre-trained models to create a multilingual, emotion-oriented communication system.

4. Experimental Results

The system was evaluated using standard metrics and human assessment:

• Emotion Detection (SER):
  • The implemented CNN achieved an average F1-score of 94%, significantly outperforming the baseline CNN (53%) and other tested models (LSTM: 87%, ResNet50: 89%).
  • The "Angry" class achieved the highest F1-score (0.97), while "Calm," "Fearful," and "Sad" classes showed slightly lower but still high performance (0.92).
• Translation Quality (MT):
  • BLEU Score: The fine-tuned MarianMT model achieved a score of 56, a significant improvement over the from-scratch Transformer (23.78) and the pre-trained baseline (25.48).
  • BERTScore F1: The fine-tuned model reached 88.7%.
  • Human Evaluation: On a scale of 1–3 (based on accuracy, fluency, and domain terminology), the system achieved an average score of 81% (approx. 2.43/3) on 100 banking-domain test samples.
• Limitations Noted: The authors acknowledge that the model may struggle with very long sentences (over 20 words) due to dataset constraints, potentially leading to truncation or context misinterpretation.

5. Significance and Claims

The paper claims that EmoAra demonstrates the practical feasibility of preserving emotional context in cross-lingual banking communications. By successfully integrating emotion detection with translation and synthesis, the system offers a pathway for customer service teams to respond more empathetically and accurately to clients' needs, regardless of language differences.

The authors maintain a modest stance regarding future implications, noting that while the current system is effective, future work should focus on expanding the banking-domain dataset (e.g., via back-translation or paraphrasing) and exploring advanced fine-tuning techniques (such as adapter layers or LoRA) to further refine accuracy and fluency. The project underscores the value of combining specific domain data with pre-trained models to solve complex, real-world communication barriers.