Code-Mixed Corpora for Indian Social Media: Baselines and Insights for Hinglish Language Identification

Technical Summary: Baselines and Insights for Hinglish Language Identification

Problem Statement
Indian social media is dominated by "Hinglish," a code-mixed blend of Hindi and English written in Roman script. While this linguistic phenomenon is natural for human communication, it presents significant challenges for Natural Language Processing (NLP) systems. Standard multilingual models, such as mBERT and XLM-R, which are typically trained on monolingual or parallel data, struggle with the specific characteristics of Hinglish: frequent language switching, inconsistent Romanization (e.g., "acha" vs. "achha"), and short, ambiguous tokens (e.g., "is," "me," "ki") that can belong to either language depending on context. Existing benchmarks often rely on synthetic data or focus on monolingual scenarios, leaving a gap in reproducible, lightweight baselines tailored for genuine code-mixed text.

Methodology
The research utilizes the COMI-LINGUA dataset, a resource containing over 125,000 expert-annotated examples with approximately 0.97 million tokens for training, 0.19 million for validation, and 0.19 million for testing. The study proposes a lightweight, reproducible pipeline for token-level Language Identification (LID) that avoids reliance on heavy multilingual pre-trained models.

The proposed approach consists of the following steps:

1. Feature Engineering: Instead of using pretrained embeddings, the method employs character-level n-grams (sizes 2 to 5). This captures distinctive subword patterns, such as English suffixes ("-ing", "-ed") and Hindi Romanized endings ("-aa", "-wala"), while mitigating the impact of spelling variations.
2. Vectorization: These character n-grams are converted into sparse vectors using TF-IDF weighting to reduce the influence of common characters and enhance robustness against fragmented words.
3. Classification: A Logistic Regression classifier (using the Saga solver with L2 regularization) is trained on these high-dimensional sparse vectors. The model is designed to be computationally efficient and interpretable, capable of running on standard CPU environments without specialized hardware.
4. Evaluation: Performance is measured using Accuracy and Macro-F1 scores. A confusion matrix is utilized to conduct a granular error analysis.

Key Results
The lightweight baseline demonstrates high efficacy on the COMI-LINGUA dataset:

• Validation Set: Achieved 96.06% accuracy and a Macro-F1 of 0.9523.
• Test Set: Achieved 95.92% accuracy and a Macro-F1 of 0.9509.

The author notes that this performance is approximately 7% better than previously published multilingual benchmarks on the same task. The results suggest that well-designed character-level features are sufficient to handle the noise and variation inherent in social media text, performing roughly 7% better than multilingual baselines on the COMI-LINGUA dataset.

Error Analysis
A detailed analysis of misclassified tokens reveals three primary sources of error:

1. Short Token Ambiguity: Common words like "is," "me," and "ki" are difficult to classify without sentence-level context, as they appear in both Hindi and English.
2. Romanization Variants: Inconsistent spellings of the same Hindi word (e.g., "acha," "achaa," "achha") fragment the feature space, reducing prediction consistency.
3. Loanword Confusion: English borrowings embedded in Hindi sentences (e.g., "party," "school") are sometimes misclassified, particularly when the surrounding context is predominantly Hindi.

Significance and Contributions
The paper claims three primary contributions to the field of Indian NLP:

1. A Reproducible Baseline: It establishes a simple, efficient pipeline using character n-grams and Logistic Regression that achieves near-state-of-the-art performance (~96% accuracy) on Hinglish LID, outperforming complex multilingual systems by a significant margin.
2. Error-Driven Insights: It provides a systematic analysis of the specific obstacles facing Hinglish LID, moving beyond general performance metrics to identify the linguistic nuances (ambiguity, orthography, loanwords) that cause failures.
3. A Roadmap for Future Work: The study advocates for a shift toward inclusive, domain-specific modeling. It suggests that future research should focus on sequence-aware models, multilingual extensions, and fairness-focused methodologies to better reflect India's linguistic diversity.

The author concludes that while the proposed model is a strong starting point, addressing the identified limitations requires moving beyond token-level independence to context-aware and sociolinguistically aware approaches. This work serves as a guide for developing inclusive language technologies that accurately reflect the complex reality of India's digital communication landscape.