💬 NLP

Detection of Emotions in Hindi-English Code Mixed Text Data

This paper introduces a new annotated corpus of Hindi-English code-mixed text and proposes a consonant-constrained normalization method to improve emotion detection, demonstrating that a sub-word LSTM model achieves the highest accuracy (76.6%) among five compared baselines for classifying anger, fear, happiness, and sadness.

Original authors: Divyansh Singh

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Divyansh Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling global marketplace where people from different countries are shouting their thoughts, jokes, and feelings into the digital air. In India, a huge part of this chatter happens in a unique, hybrid language called "Hinglish." It's like a linguistic smoothie where Hindi and English get blended together in a single sentence. The problem is that because everyone types this on their phones using the Latin alphabet (the same letters we use for English), there are no strict rules for how to spell the Hindi words. One person might write "love" as pyaar, another as pyar, and a third as pyr. To a computer, these look like three completely different words, making it very hard for software to understand what people are actually saying.

This is where the science of "Natural Language Processing" (NLP) comes in. Think of NLP as teaching computers to read and understand human language. While computers have gotten pretty good at figuring out if a sentence is generally "happy" or "sad" (sentiment analysis), they struggle to detect specific, complex feelings like anger, fear, or excitement, especially when the text is messy and mixed-up. Understanding these specific emotions is crucial because it helps computers know if someone is just having a bad day or if they are in real danger, which is vital for things like online safety and mental health monitoring.

Enter a researcher from the LNMIIT in Jaipur, India, who decided to tackle this messy, mixed-language puzzle. They wanted to teach computers to spot four specific emotions—Anger, Fear, Sadness, and Happiness—in Hinglish text. To do this, they first had to build their own "dictionary" of feelings. They collected 1,589 sentences from Twitter and video comment sections, where people were chatting freely. Two native speakers of Hindi (who also speak English fluently) read every sentence and labeled the emotion. They agreed on the labels almost perfectly, creating a high-quality training set for their computer models.

The researcher faced a tricky obstacle: the spelling chaos. Since people spell Hindi words in so many different ways, the computer kept getting confused. To fix this, the researcher invented a clever "normalization" trick. Imagine you have a pile of socks where some are labeled "Red," some "Redd," and some "Rd." Instead of trying to guess which is right, the computer looks at the "skeleton" of the word (the consonants) and how often the word appears in similar contexts. It then groups all the weird spellings together and replaces them with the most common version. This cleans up the data so the computer isn't distracted by spelling errors.

Next, they tested five different computer "brains" (algorithms) to see which one could best guess the emotion. They compared simple statistical methods against more complex neural networks. The big surprise? The most complex model, which looked at the tiny building blocks of words (sub-words) rather than whole words, won the race. It achieved an accuracy of 76.6%, which is a huge jump from the 30.8% you'd get if you just guessed the most common emotion every time. This suggests that breaking words down into smaller pieces helps the computer handle the messy spelling variations much better.

However, the researcher was careful not to declare a total victory. They found that a simpler, older method called "Naïve Bayes" (which basically counts how often certain words appear) performed almost as well as the fancy neural network. This suggests that in short social media posts, emotions are often carried by just one or two key words, so a simple word-counter can do a surprisingly good job. The paper concludes that while the sub-word model is the current champion, the small size of their dataset (1,589 sentences) makes it hard to say for sure if one method is truly superior to the other. They suggest that the biggest hurdle isn't the algorithm, but the lack of data; with a much larger library of examples, these computers could likely become even sharper at reading our digital hearts.

Technical Summary: Detection of Emotions in Hindi-English Code-Mixed Text Data

Problem Definition
This paper addresses the challenge of emotion detection in Hindi-English code-mixed text (often termed "Hinglish"), a prevalent form of user-generated content in India. Unlike standard sentiment analysis which focuses on polarity (positive/negative), this task targets finer-grained emotion classification across four categories: Anger, Fear, Sadness, and Happiness. The primary difficulty lies in the lack of standardized transliteration conventions; Hindi words written in the Latin script appear in numerous orthographic variants (e.g., hain, hai, ha), causing tokens to fall outside standard English lexica and rendering morphological analyzers trained on Devanagari script ineffective.

Methodology
The author proposes a three-pronged approach involving dataset construction, a normalization procedure, and a comparative evaluation of baseline models.

Corpus Construction: A dataset of 1,589 code-mixed sentences was compiled from Twitter and video-platform comment sections. The inclusion of video comments introduces a register with longer, less hashtag-dominated utterances compared to previous Twitter-only corpora. The data was annotated by two bilingual speakers, achieving a high inter-annotator agreement (Cohen's $\kappa = 0.94$ ). The distribution is moderately imbalanced, with "Happy" as the majority class (30.8%) and "Fear" as the minority (19.1%).
Normalization Procedure: To address transliteration variance, the author introduces a clustering procedure that combines distributional similarity with a hard linguistic constraint.
- Mechanism: Skip-gram vectors are trained on the unlabelled data. Words are clustered if they satisfy two conditions: (1) they share an identical ordered consonant sequence (a hard constraint, as variation is primarily vowel-based), and (2) they exhibit high cosine similarity in their vector space.
- Substitution: Within each cluster, all variants are replaced by the most frequent form. This reduces vocabulary sparsity while preserving the semantic identity of the word.
Baseline Models: Five classifiers were evaluated under a five-fold cross-validation protocol (stratified by class):
- Naïve Bayes over character n-grams ( $n=8$ ).
- Naïve Bayes over word n-grams (unigrams and bigrams).
- Word-level LSTM using word2vec embeddings.
- Sub-word LSTM (based on Joshi et al. [9]), which convolves character embeddings to create sub-word representations before feeding them into an LSTM.
- Support Vector Machine (SVM) over frequency-based word vectors.

Results
The experimental results highlight the following performance metrics:

Accuracy: The Sub-word LSTM achieved the highest accuracy at 76.6%, significantly outperforming the majority-class baseline (30.8%) and the next-best model (Word n-gram Naïve Bayes at 75.1%).
Macro-F1 Score: The Sub-word LSTM tied with the Word n-gram Naïve Bayes at 0.77. The author notes that given the dataset size (1,589 instances) and the five-fold protocol, this margin is statistically indistinguishable.
Error Analysis:
- The Sub-word LSTM improved accuracy over the word-level LSTM by 4.0 points, isolating sub-word representation as the source of gain. This is attributed to the word-level model's inability to handle rare transliteration variants that remain unmerged by the normalization step.
- The "Fear" class was the best-classified category for most models (F1 $\approx$ 0.82), likely due to a distinctive, closed lexicon (e.g., bachao, darr).
- The Sub-word LSTM performed slightly lower on the "Fear" class compared to other models, suggesting that high-capacity models may struggle with small classes where memorizing a specific lexicon is more effective than contextual discrimination.

Key Contributions and Significance
The paper makes three specific contributions to the field of low-resource NLP for code-mixed languages:

Dataset: It releases a corpus of 1,589 annotated sentences that expands beyond the Twitter-only domain to include video-platform comments, offering a more diverse linguistic register.
Normalization: It proposes a constrained clustering method for transliteration variants that explicitly leverages consonant identity, offering a preprocessing alternative to relying solely on model robustness.
Controlled Comparison: It provides a rigorous comparison between word-level and sub-word-level representations. The study demonstrates that while sub-word representations (Sub-word LSTM) yield the highest accuracy by mitigating orthographic variance, simple bag-of-n-grams models remain highly competitive on this task size.

Limitations and Scope
The author maintains a modest stance regarding the paper's scope. They acknowledge that the corpus size is small compared to broader sentiment resources (e.g., SentiMix), which inflates variance in performance metrics and limits the statistical resolution of model comparisons. The study explicitly excludes transformer-based pre-trained encoders (e.g., BERT), noting that while they likely dominate current leaderboards, the current work focuses on isolating the specific contribution of sub-word modeling and normalization. Furthermore, the author cautions that the dataset's small scale and narrow platform coverage make it unsuitable for deployment in consequential decision-making settings.

More like this