Technical Summary: IRA – An Interpretable Hybrid Chatbot for Emotion-Aware Intent Classification with Generative Response Fallback

1. Problem Statement

Conversational AI systems deployed in mental health, education, and customer service often rely on opaque cloud-resident transformer or Large Language Model (LLM) backbones. These systems present three critical limitations:

1. Lack of Interpretability: Clinicians and end-users cannot audit the decision process (e.g., which lexical cues drove a specific intent classification), creating a "black box" problem unsuitable for safety-sensitive domains.
2. Inadequate Emotional Register: Existing datasets and models often lack the informal, code-mixed, "friend-like" dialogue patterns required for empathetic engagement, relying instead on formal business or FAQ corpora.
3. Connectivity and Resource Constraints: Many advanced systems require constant internet access and high computational power, limiting their utility in resource-constrained or offline environments.
4. Clinical Safety Gaps: A significant portion of mental-health chatbots lack clinical validation, and stochastic LLM outputs risk misinterpretation in psychiatric contexts.

2. Methodology

The paper proposes IRA (Interactive Responsive Assistance), a hybrid conversational framework designed to balance deterministic safety with generative flexibility.

Architecture

IRA operates in two distinct modes:

• Offline Mode (Deterministic): Utilizes a lightweight, interpretable Multi-Layer Perceptron (MLP) classifier.
  • Input Processing: Text is preprocessed using NLTK for tokenization and lemmatization (WordNetLemmatizer).
  • Feature Extraction: A Bag-of-Words (BoW) representation converts text into binary vectors.
  • Model: A shallow MLP with two hidden layers (128 and 64 neurons, ReLU activation) optimized via the Adam solver.
  • Output: A discrete intent label triggering a pre-curated, deterministic response.
• Online Mode (Generative): When connectivity permits, the system routes queries to OpenAI's GPT-3.5-Turbo API for open-ended, context-aware dialogue.

Dataset Construction

A custom dataset (intents.json) was constructed to address the lack of empathetic, code-mixed data:

• Scale: 182 initial samples covering 27 intent classes.
• Content: Includes safety-sensitive categories (e.g., 'Suicidal Thoughts', 'Depression', 'Anxiety') alongside everyday emotional states ('Sad', 'Happy') and social interactions ('Greetings', 'Friend').
• Style: Captures informal, code-mixed (Hindi/Urdu-English) dialogue patterns absent in standard corpora.
• Balancing: Manual oversampling (factor of 2) was applied to address class imbalance before training.

Interpretability Mechanisms

To ensure transparency, the offline model employs:

• SHAP (SHapley Additive exPlanations): Provides per-token attribution to reveal which lexical features (e.g., "sad," "help," "anxious") drove a specific prediction.
• t-SNE Visualization: Maps the hidden layer activations to 2D space, demonstrating how the model clusters emotional and topical intents (e.g., separating 'Suicidal Thoughts' from 'Sad').

3. Key Contributions

The authors frame their contributions as engineering and methodological improvements rather than algorithmic novelty:

1. Hybrid Dual-Mode Architecture: A system that couples a deterministic, interpretable intent classifier with a generative cloud path, ensuring operational continuity in both offline and online conditions.
2. Emotion-Aware Code-Mixed Dataset: A custom corpus of 182 utterances spanning 27 classes, explicitly designed to capture friend-like, comforting dialogue patterns and safety-sensitive categories.
3. Joint Interpretability Layer: The integration of SHAP token attribution and layer-wise t-SNE visualization on the same backbone, exposing decision cues to clinicians and users.
4. Computational Transparency: A complete profile reporting training time (2.93s), memory usage (1.30 MB), model size (1.22 MB), and inference latency (0.006908 ms/sample), metrics often omitted in comparator literature.
5. Critical Comparative Evaluation: A head-to-head comparison against six state-of-the-art systems, accompanied by a task-formulation analysis explaining why cross-corpus numerical comparison is methodologically unsound due to heterogeneous label spaces.

4. Results

Evaluated on the proposed corpus, the offline MLP classifier achieved:

• Test Accuracy: 95.04% (Note: This high performance is partly attributed to duplication-based oversampling, which does not introduce lexical variation).
• Macro-Averaged Metrics: Precision (0.98), Recall (0.97), F1-Score (0.96).
• Class Performance: Most classes achieved perfect scores (1.00). However, the 'Depression' class showed a precision of 0.42 (F1: 0.59), indicating a tendency for false positives, likely due to feature overlap with 'Sad' and 'Suicidal Thoughts' in the BoW representation.
• Efficiency: The model trains in 2.9 seconds and requires only 1.30 MB of RAM, enabling deployment on resource-constrained devices.
• Comparative Performance: When tested against three public datasets ([36], [37], [38]) using the same pipeline, the proposed dataset achieved the highest test accuracy (95.04% vs. 21.28–71.60%), though the authors caution this reflects internal consistency rather than a true head-to-head ranking, since label spaces differ across corpora. It also demonstrated superior scores in empathy, supportiveness, and cultural personalization.

5. Significance and Limitations

Significance:
The paper positions IRA as a transparent baseline and engineering prototype rather than a finished clinical tool. Its primary significance lies in:

• Documenting a reproducible, interpretable approach for emotion-aware AI in domains where opaque cloud architectures are inappropriate.
• Explicitly quantifying the trade-offs between accuracy, interpretability, and computational cost.
• Providing a methodologically justified dataset and evaluation framework for future research in empathetic, code-mixed dialogue.

Limitations and Future Work:
The authors explicitly acknowledge several constraints:

• Dataset Size: The initial corpus is small (182 samples); future work plans expansion to 2,000–5,000 samples via LLM-driven paraphrasing and crowd-sourcing.
• Model Baselines: The study lacks a head-to-head comparison against transformer baselines (e.g., DistilBERT) on the same dataset, a gap identified for future replication.
• Safety Validation: The system is not clinically validated. Future work requires a deterministic crisis-response routing layer, privacy audits for the online mode, and pre-registered clinical-feasibility studies.
• API Dependency: The online mode relies on a third-party API (OpenAI), introducing data exposure and cost concerns; future iterations aim to replace this with a locally hosted Small Language Model (e.g., Phi-3-mini).
• Ablation and User Studies: The paper notes the absence of ablation studies (varying architecture/oversampling) and user studies with target populations as outstanding gaps.

In conclusion, IRA offers a scalable, interpretable starting point for human-AI interaction in critical applications, prioritizing transparency and resource efficiency while acknowledging the rigorous validation steps required before clinical deployment.