💬 NLP

destroR: A Benchmark and Adversarial-Training Defense for Bangla Transfer Models under Meaning-Preserving Attacks

The paper introduces "destroR," a unified pipeline that establishes the first comprehensive benchmark for Bangla transfer models against meaning-preserving attacks and demonstrates that adversarial training effectively hardens these models, revealing that multilingual backbones like MuRIL outperform Bangla-dedicated ones in robustness.

Original authors: Saadat Rafid Ahmed, Rubayet Shareen, Radoan Sharkar, Nazia Hossain, Mansur Mahi, Farig Yousuf Sadeque

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Saadat Rafid Ahmed, Rubayet Shareen, Radoan Sharkar, Nazia Hossain, Mansur Mahi, Farig Yousuf Sadeque

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart robot that reads Bangla text and guesses if a person is happy, sad, or neutral. You think it's perfect because it gets the right answer almost every time in your test room. But what if someone sneaks in and whispers a slightly different version of the sentence to the robot? The meaning stays exactly the same to a human ear, but the robot suddenly panics and guesses the wrong emotion.

That's the problem destroR tackles. The researchers at BRAC University wanted to see how "brittle" these Bangla AI models really are and if they could be toughened up. They didn't just poke holes; they built a whole gym for the robots to train in.

The Three Ways to Trick the Robot

The team invented three special "pranks" to confuse the AI without changing the actual story. Think of these as different ways to rewrite a sentence so a human reads the same meaning, but the robot gets dizzy:

The Paraphrase Prank: They used a tool to rewrite the sentence using different words and structures, like saying "The movie was great" instead of "I loved the film." The meaning is identical, but the robot sees a totally new pattern.
The Back-Translation Loop: They took the Bangla sentence, translated it into English, and then translated it back into Bangla. Because translation tools aren't perfect, the sentence comes back with a slightly different flavor or structure, like a game of "Telephone" that accidentally changes the grammar but keeps the vibe.
The Word-Swap Swap: This one is a bit sneakier. They identified the specific words the robot was obsessed with (the "biased" tokens) and swapped them out for other words that fit the context but might throw the robot off balance.

The Big Surprise: Bigger Isn't Always Better

Here is the twist that shocked the researchers. You might think a robot built specifically for Bangla would be the toughest. But the data showed the opposite.

The MuRIL model—a multilingual robot trained on many Indic languages—was the toughest cookie. It held its ground better than the models built just for Bangla (like BanglaBERT). It's like a Swiss Army knife being more durable than a specialized screwdriver in this specific test. The researchers suggest this is because MuRIL has a bigger, more varied vocabulary that makes it harder for the attackers to find those weak, biased words to exploit.

The "Gym" Defense: Adversarial Training

So, how do you fix a robot that gets confused easily? You don't just patch it; you train it. The researchers used a technique called adversarial training.

Imagine a boxer. If you only train them against one type of punch, they'll get knocked out by a different one. But if you throw every kind of punch at them during practice, they learn to dodge anything. The researchers fed the models thousands of these "tricked" sentences during training.

The results? It worked. When they trained the models on a mix of all the different attack types (the "union of all attack families"), the robots became much harder to fool. The success rate of the attacks dropped significantly for everyone.

The Numbers Don't Lie (But Some Attacks Are Stronger)

The researchers ran these tests across five different models and four different datasets. Here is what the scoreboard looked like after the training:

The Heavy Hitters: Even after training, the strongest attacks from the English world (called TextFooler and BAE) were still very potent. BAE managed to trick the models 54.2% of the time, and TextFooler got 32.1%. These attacks are like master thieves who know exactly which lock to pick.
The Bangla Recipes: The team's own Bangla-specific pranks were a bit milder. The Back-Translation attack succeeded 21.5% of the time, and the Paraphrase attack succeeded 15.3% of the time.
The One-Hot Glitch: The word-swap attack (the third one mentioned earlier) basically failed, with a success rate of only 0.2%. The researchers admit this recipe was "weak" for Bangla because the tool they used to swap words (trained on many languages) didn't find good replacements in Bangla. It's like trying to swap a specific Bangla word with a generic English synonym that just doesn't fit.

What This Means

The paper doesn't claim they've "solved" the problem of AI security. Instead, they've built a benchmark (a standard test) and a defense that actually works. They showed that:

Bangla models are currently quite fragile against clever attacks.
Training on a mix of attacks makes them much tougher.
Sometimes, a multilingual model (MuRIL) is more robust than a language-specific one.

They've released all their code, data, and models so anyone can try to break them or build better defenses. It's an open invitation to the community to keep the robots sharp, because in the world of AI, being "smart" isn't enough—you have to be "tough" too.

Technical Summary: destroR: A Benchmark and Adversarial-Training Defense for Bangla Transfer Models under Meaning-Preserving Attacks

Problem Statement
While transformer-based transfer models have achieved high accuracy in Bangla sentiment classification, their robustness against adversarial attacks remains largely unexamined. Unlike English, where extensive lexical resources and paraphrase corpora facilitate attack design, Bangla is a low-resource language for adversarial NLP. Its rich morphology, frequent code-mixing, and transliteration make creating imperceptible, meaning-preserving perturbations difficult. Consequently, there is no systematic characterization of Bangla model robustness, nor any prior study that couples a Bangla-specific attack suite with a defense proven to recover robustness. The central question addressed is: How robust are Bangla transfer models to adversarial attacks, and can adversarial training recover that robustness?

Methodology
The authors introduce destroR, a unified pipeline for evaluating and hardening Bangla text classifiers. The methodology is structured around three core components:

Meaning-Preserving Attack Recipes:
The paper proposes three specific attack strategies designed to perturb inputs while regenerating fluent, semantically faithful sentences, rather than injecting surface noise:
- Bangla Paraphrase Attack: Utilizes a fine-tuned BanglaT5 model (csebuetnlp/banglat5_banglaparaphrase) to generate diverse, context-aware paraphrases. Semantic fidelity is constrained using LaBSE cosine similarity ( $\tau \ge 0.7$ ).
- Bangla Back-Translation Attack: Inspired by TransFool, this method translates Bangla text to English and back to Bangla using two-way translation pipelines (csebuetnlp/banglat5_nmt_en_bn and csebuetnlp/banglat5_nmt_bn_en). This exploits alignment differences to alter sentence structure and lexicon while attempting to preserve semantics.
- One-Hot Word-Swap Attack: A gray-box, word-level attack that identifies important tokens via leave-one-out queries and replaces them using a masked language model (XLM-RoBERTa-base). This targets token-level biases in the model.
Robustness Benchmark:
The study evaluates five transfer models across four distinct datasets (blp23, youtube_sentiment, CogniSenti, BASA_cricket2). The victim models include:
- Bangla-dedicated: BanglaBERT, BanglishBERT.
- Indic-multilingual: MuRIL, IndicBERTv2.
- General-multilingual: XLM-RoBERTa.
These models are tested against the three proposed recipes and two established English baselines (TextFooler and BAE) under an identical protocol. The evaluation metric is the Attack Success Rate (ASR), defined as the fraction of originally correct predictions flipped by the attack.
Adversarial Training Defense:
To harden the models, the authors employ adversarial training, augmenting the training set with generated adversaries. Three training regimes are evaluated:
- AT-Paraphrase: Augmentation with paraphrase adversaries only.
- AT-BackTrans: Augmentation with back-translation adversaries only.
- AT-All: Augmentation with the union of all attack families.

Key Results
The analysis yields three primary findings regarding the interaction between attacks, defenses, and model architectures:

Potency of Word-Substitution Baselines: Strong word-substitution baselines (specifically BAE) outperform the semantically constrained sentence-level recipes. BAE achieved a residual Attack Success Rate (ASR) of 54.2% even against adversarially trained models, compared to 21.5% for Back-Translation and 15.3% for Bangla Paraphrase. The One-Hot Word-Swap recipe was found to be largely ineffective (0.2% ASR) on Bangla, likely due to the XLM-R mask-filler's inability to produce valid Bangla substitutions.
Effectiveness of Adversarial Training: Adversarial training significantly lowers residual attack success across all attack families. Crucially, the AT-All regime (training on the union of all attack families) consistently yielded the lowest residual ASR for every attack type, outperforming single-family training. This suggests that adversaries share transferable structural and lexical perturbations.
Model Robustness Disparity: Contrary to the expectation that language-specific models would be more robust, the MuRIL (Indic-multilingual) backbone proved markedly more robust than the Bangla-dedicated models (BanglaBERT and BanglishBERT). For instance, the BAE attack success rate against MuRIL was 40.2%, whereas it was 64.2% against BanglishBERT. The authors attribute this to MuRIL's broader sub-word vocabulary, which may dilute the token-level biases exploited by word-substitution attacks.

Significance and Contributions
The paper claims significance by addressing a critical gap in low-resource NLP security. Its specific contributions are:

First Unified Pipeline: The introduction of destroR, the first pipeline to pair a Bangla-specific attack suite with a defense that measurably recovers robustness.
New Attack Recipes: The development of three meaning-preserving attack recipes tailored to the linguistic constraints of Bangla, moving beyond simple noise injection to semantic regeneration.
Comprehensive Benchmark: A robustness matrix evaluating five models across four datasets and five attacks, providing a baseline for future research.
Defense Validation: Empirical evidence that adversarial training, particularly when using a union of attack families, effectively hardens Bangla sentiment classifiers.
Reproducibility: The release of all victim models, adversarial data, and code with fixed seeds to ensure full reproducibility.

The authors maintain a modest stance, acknowledging limitations such as the use of a single random seed, reliance on automatic semantic fidelity metrics (LaBSE) without human validation, and the current ineffectiveness of the One-Hot Word-Swap recipe on Bangla. They frame the work as a foundation for future research into the robustness of Bangla and other low-resource language models, rather than a definitive solution to all adversarial threats.