💬 NLP

Beyond Clean Text: Evaluating Encoder and Decoder Robustness for Bangla Event Detection in Noisy Text

This paper introduces a comprehensive Bangla event detection benchmark with noisy data to demonstrate that while encoder-only models excel on clean text, decoder-only LLMs offer superior robustness to real-world noise, a gap that can be narrowed through combined training and model scaling.

Original authors: Tanvir Ahmed Sijan, S. M Golam Rifat, Nayeemul Islam, Md. Musfique Anwar

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tanvir Ahmed Sijan, S. M Golam Rifat, Nayeemul Islam, Md. Musfique Anwar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a story told to you. Sometimes, the story is told perfectly clearly by a professional news anchor. Other times, it's told by a friend who has a cold, is typing on a broken keyboard, or is speaking into a microphone that crackles.

This paper is about teaching computers to understand events (like "a cyclone hit," "a protest started," or "prices went up") in the Bangla language, specifically when the text is messy, noisy, or full of typos.

Here is the breakdown of their study using simple analogies:

1. The Problem: The "Clean Room" vs. The "Real World"

Most computer programs that read news are trained in a "clean room." They only see perfect, edited text. But in the real world, text is messy. It comes from:

ASR (Automatic Speech Recognition): Computers trying to transcribe spoken news, often getting words wrong because of accents or background noise.
Typos: People typing fast on phones, hitting the wrong keys, or mixing up similar-sounding letters.

The researchers asked: If we train a computer on perfect text, will it still work when the text is messy?

2. The Experiment: Two Types of "Readers"

They tested two different types of AI "readers" on a massive collection of 9,979 Bangla sentences (some perfect, some messy):

The "Specialist" (Encoder Models like BanglaBERT): Think of this like a highly trained librarian. If the book is perfect, this librarian is incredibly fast and accurate at finding specific words. They are the best at reading clean text.
The "Generalist" (Decoder LLMs like Llama 3 and Gemma): Think of this like a wise, experienced storyteller. They might not be as fast at finding a specific word in a perfect book, but they are great at understanding the story even if the speaker stammers or misspells words. They use context to guess what was meant.

3. The Big Discovery: The Trade-Off

The results showed a clear split in personality between the two types of readers:

On Clean Text: The Specialist (Encoder) wins. It finds the events with the highest accuracy.
On Messy Text: The Generalist (Decoder) wins. When the text is full of typos or speech errors, the Specialist gets confused and fails. The Generalist, however, stays calm. It uses its "common sense" to figure out that "cyclone" was meant even if it was typed as "syclone."

The Analogy:
Imagine a spelling bee champion (the Specialist) vs. a seasoned detective (the Generalist).

If the word is written perfectly, the champion spells it instantly.
If the word is written with a missing letter or a weird accent, the champion freezes. The detective, however, looks at the clues around the word and says, "Ah, they clearly meant 'cyclone'."

4. The "Trigger" vs. The "Context"

The researchers dug deeper to see why the Generalist was better. They found a specific weakness in the Specialist:

Trigger Corruption: If the key word of the event (the "trigger," like the word "arrest") is misspelled, the Specialist fails completely. The Generalist can still figure it out.
Context Corruption: If the surrounding words are messy but the key word is perfect, the Specialist actually does quite well.

The Lesson: The Generalist is better at guessing the meaning of a broken key word. The Specialist is better at reading the surrounding story if the key word is intact.

5. How to Make Them Stronger (The Training Tricks)

The paper tested two ways to make these AI readers tougher:

Giving Them a Rulebook (Instruction Tuning): They gave the Generalist a detailed "cheat sheet" explaining exactly what an event is.
- Result: This made the Generalist smarter overall, but it didn't always make it better at handling noise. Sometimes, knowing the rules too strictly made it less flexible.
Training on Messy Data (Combined Training): They trained the AI on a mix of perfect text and messy text.
- Result: This was the magic bullet for the Specialist. By practicing on messy text, the Specialist learned to ignore the noise. It didn't get much better at reading perfect text, but it became much less likely to panic when the text got dirty. This narrowed the gap between the two types of AI.

6. Bigger Isn't Always Better (For Everyone)

For the Generalist (Decoder): Making the model bigger (more "brain power") consistently made it more robust against noise. A bigger detective is better at solving messy cases.
For the Specialist (Encoder): Making the model bigger didn't help much with noise. A bigger librarian is still just as confused by a typo as a small one.

Summary

The paper concludes that if you want an AI to detect events in the real world (where text is often messy, spoken, or full of typos), you shouldn't just rely on the "Specialist" models trained on perfect data.

Instead, you should use the "Generalist" (Decoder) models, which are naturally more resilient to noise, or you must train your "Specialist" models on messy data to teach them how to handle the real world. The old way of only testing on clean text gives a false sense of security.

Technical Summary: Beyond Clean Text: Evaluating Encoder and Decoder Robustness for Bangla Event Detection in Noisy Text

Problem Statement
Event Detection (ED) systems are predominantly evaluated on clean, curated text, leaving their robustness to real-world noise largely unexplored, especially for low-resource languages like Bangla. In practical applications, such as emergency monitoring, data is inherently noisy due to typographical errors, orthographic variations, and Automatic Speech Recognition (ASR) transcription faults. Existing Bangla ED research is limited to narrow domains (e.g., violence, disasters) and clean text, lacking a generalized ontology comparable to ACE 2005. Furthermore, the field has not systematically compared the robustness of traditional encoder-only models (e.g., BERT-based) against modern instruction-tuned decoder-only Large Language Models (LLMs) under realistic noisy conditions.

Methodology
The authors address these gaps through the development of a comprehensive benchmark and a systematic evaluation framework:

Ontology and Dataset Construction:
- Ontology: A generalized Bangla news event ontology was developed by extending ACE 2005 with locally relevant event types derived from GDELT and GKG corpora. The final schema includes 14 main event categories and 40 subtypes.
- Data: The benchmark comprises 9,979 annotated sentences containing 7,813 event mentions. The dataset is split into:
  - Clean News Text: 5,320 sentences from major Bangladeshi newspapers.
  - Real-World ASR Transcripts: 4,659 sentences from news video broadcasts, capturing natural transcription errors, dialectal variations, and colloquialisms.
  - Simulated Orthographic Noise: The clean test set was corrupted using an algorithm based on common Bengali typing patterns (e.g., phonetic substitutions, keyboard proximity errors) at Word Corruption Rates (WCR) of 10% to 40%.
- Annotation: Data was annotated by six human annotators with a Cohen's $\kappa$ of 0.72, followed by a two-stage adjudication process.
Model Architectures and Training:
- Encoder-Only: Fine-tuned BanglaBERT (Base/Large) and XLM-RoBERTa (Base/Large) were trained as token classification tasks.
- Decoder-Only: Instruction-tuned LLMs (Llama 3.1/3.2 and Gemma 3 in 1B, 4B, and 8B configurations) were trained using structured sequence generation. Event detection was formulated as code generation (Python dataclasses) with optional annotation guidelines embedded as docstrings in the prompt.
- Training Strategies: Models were evaluated under two conditions: training exclusively on clean data and training on a combined dataset of clean and noisy ASR transcripts.
Evaluation Protocol:
- Performance was measured using Macro-F1 to account for class imbalance.
- Robustness was quantified as the performance degradation ( $\Delta F1$ ) between clean test sets and noisy conditions (ASR and Orthographic).
- A specific analysis using McNemar's exact test distinguished between "trigger corruption" (noise on the event word itself) and "context corruption" (noise on surrounding words).

Key Contributions

Generalized Ontology & Benchmark: The release of the first generalized Bangla news event ontology and a benchmark dataset spanning clean text, real-world ASR, and simulated orthographic noise.
Architectural Comparison: A systematic evaluation comparing encoder-only and decoder-only architectures across varying parameter sizes and noise levels.
Robustness Insights: The first comprehensive study of Bangla ED robustness under both real-world ASR and simulated orthographic noise, revealing distinct architectural trade-offs.

Results

Architectural Trade-off: Encoder-only models (e.g., BanglaBERT Large) achieve higher performance on clean text (Macro F1 ~68.9) but suffer substantial degradation under noise (up to 14.6 points). In contrast, decoder-only LLMs (e.g., Llama 3.1 8B) show lower peak performance on clean text but exhibit markedly higher robustness, with significantly smaller performance drops (e.g., ~2.7 points).
Trigger vs. Context Corruption: Decoder-only LLMs significantly outperform encoders when the event trigger itself is corrupted. However, when corruption is confined to the context and the trigger remains intact, the performance gap narrows, and encoders often perform comparably or better.
Impact of Annotation Guidelines: Embedding annotation guidelines during instruction tuning consistently improves performance on both clean and noisy data but yields inconsistent results regarding the reduction of performance degradation ( $\Delta F1$ ).
Impact of Combined Training: Training on a mixture of clean and noisy data serves as an effective regularization strategy. It disproportionately benefits encoder architectures, significantly narrowing the robustness gap with LLMs (e.g., reducing BanglaBERT's degradation from 14.6 to 7.9 points).
Model Scaling: Increasing model size consistently improves the robustness of decoder-only LLMs (e.g., 8B models are more robust than 1B models). Conversely, scaling encoder models from Base to Large improves clean-text performance but does not consistently improve robustness; in some cases, larger encoders are even more sensitive to noise.

Significance and Claims
The paper claims to provide the first comprehensive evaluation of language model robustness for Bangla event detection in noisy environments. It highlights a critical limitation in current ED research: the reliance on clean benchmarks masks the fragility of state-of-the-art encoder models in real-world scenarios. The authors conclude that while encoders excel at precise trigger localization in clean text, decoder-only LLMs offer a more robust solution for noisy, real-world applications, particularly when the event trigger itself is corrupted. Furthermore, the study suggests that combined training on clean and noisy data is a vital strategy for improving encoder robustness, potentially making them viable for noisy environments without requiring the scale of decoder-only models. The work underscores the necessity of evaluating ED systems beyond clean-text benchmarks, especially for low-resource languages.