GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification

This paper demonstrates that fine-tuned bidirectional encoders, specifically a hybrid AraBERTv2 architecture, significantly outperform large-scale causal decoders in the challenging task of 82-class Arabic medical text classification by better capturing global semantic context despite data imbalance and label noise.

The Gist

Imagine you are the head librarian of a massive, chaotic library filled with 82 different sections of medical books. People keep walking in and shouting short, messy requests like, "My knee hurts," or "I feel dizzy after eating herbs." Your job is to instantly shout out exactly which of the 82 sections these requests belong to.

This paper is a report from a team (GATech) who built a robot librarian to solve this exact problem for Arabic medical queries. They tested two very different types of robots to see which one could sort the books best.

Here is the breakdown of their experiment, explained simply:

1. The Two Types of Robots

The team compared two different "brain" architectures:

• The "All-Seeing" Robot (Bidirectional Encoder): Think of this robot as a person who can read a sentence backwards and forwards at the same time. If you say, "I have a headache and I took aspirin," this robot sees the connection between the pain and the medicine instantly, understanding the whole picture as a single unit. They used a specialized version called AraBERT, which is like a librarian who has read every Arabic medical book ever written.
• The "One-Way" Robot (Causal Decoder): This robot is like a person reading a story one word at a time, from left to right, trying to guess what word comes next. These are the massive "Generative AI" models (like Llama or Qwen) that are famous for writing essays or chatting. They are huge and smart, but they are trained to predict the future, not to categorize the present.

2. The Big Problem: A Messy Library

The team faced two huge hurdles:

• The "Long-Tail" Problem: Some sections of the library had hundreds of books (like "General Medicine"), but others had only 7 books (like "IVF" or "Vascular Surgery"). It's like trying to learn to recognize a rare bird when you've only seen it seven times in your life.
• The "Confused Labels" Problem: Sometimes, the books were misfiled. A note about a skin rash might be labeled "General Medicine" instead of "Dermatology." The robot had to learn to ignore these mistakes and find the real meaning.

3. The Winning Strategy: The "Hybrid" Librarian

The team didn't just use the standard AraBERT robot; they gave it a special upgrade kit to handle the mess:

• The "Spotlight" and the "Average" (Hybrid Pooling):
  • Imagine the robot has two ways of looking at a sentence.
  • Method A (Mean): It takes the "average" feeling of the whole sentence to get the general vibe.
  • Method B (Attention): It puts on a spotlight to zoom in on the most important words (like "heart attack" or "fever") and ignores the boring filler words.
  • They combined these two views to get a super-rich understanding of the query.
• The "Practice Run" (Multi-Sample Dropout):
  • To stop the robot from memorizing the few examples of rare diseases, they made it practice the same task five times with different "blindfolds" (randomly ignoring parts of the data). This forced the robot to learn the core concepts rather than just memorizing specific examples.
• The "Soft" Teacher (Label Smoothing):
  • Since the labels were sometimes wrong, they taught the robot not to be 100% confident in the labels it was given. Instead of saying "This is 100% Dermatology," it learned to say, "This is mostly Dermatology, but maybe a little bit of General Medicine." This helped it handle the messy data better.

4. The Results: Why the "Big" Robot Lost

The team was surprised. They thought the massive, super-smart "One-Way" robots (like Llama 3.3 70B) would win because they are so big and powerful. They didn't.

• The "One-Way" Robot failed because it was too focused on the end of the sentence. If a patient mentioned a symptom at the very beginning of a long query, the big robot forgot it by the time it finished reading. It was also too "general." It knew what "skin disease" meant, but it didn't understand the specific, tiny differences between the 82 specific categories the library required.
• The "All-Seeing" Robot won because it could see the whole picture at once. It was better at compressing the messy, short medical queries into a clear, precise category.

5. The "Re-Ranking" Experiment

They tried a clever trick: Let the small robot pick the top 15 guesses, then ask the giant "One-Way" robot to pick the winner from those 15.

• Result: This made things worse. The giant robot was too smart for its own good; it used its general knowledge to pick a "logical" answer that was actually wrong according to the library's specific, weird rules. The small, specialized robot knew the specific rules better.

The Bottom Line

In the world of sorting specific, messy medical data:

• Specialized, focused tools (Bidirectional Encoders) are better than massive, general-purpose tools (Causal Decoders).
• Just because a robot is huge and can write poetry doesn't mean it's good at filing a specific medical form.
• By combining different ways of looking at words and teaching the robot to be humble about the messy data, the team built the best "librarian" for this specific job.

Final Score: The specialized robot (AraBERT) scored a 0.39 on a difficult scale, while the giant robot trying to help actually dragged the score down to 0.30.

Technical Summary

Based on the paper "GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification," here is a detailed technical summary:

1. Problem Definition

The paper addresses the AbjadMed shared task, which involves classifying Arabic medical queries into 82 distinct categories. The task presents three significant challenges:

• High Cardinality & Class Imbalance: The dataset contains 27,951 training samples distributed across 82 classes. The distribution is extremely long-tailed, with majority classes (e.g., Hematological diseases) having ~600 samples, while minority classes (e.g., IVF, Vascular surgery) have as few as 7 samples.
• Label Noise and Ambiguity: Manual inspection revealed inconsistent labeling where semantically similar categories (e.g., "General Medicine" vs. "Internal Medicine") were assigned inconsistently, creating "soft" boundaries.
• Architectural Tension: The core research question investigates whether large-scale causal decoders (LLMs like Llama/Qwen), known for zero-shot reasoning, can outperform specialized, fine-tuned bidirectional encoders (like BERT) for fine-grained, domain-specific classification.

2. Methodology

The authors propose a system centered on a fine-tuned bidirectional encoder enhanced with specific regularization and pooling strategies, while benchmarking it against causal decoder approaches.

Primary Architecture: Enhanced AraBERTv2

• Base Model: Selected AraBERTv2 (bert-base-arabertv02) due to its pre-training on massive Arabic corpora, which captures Modern Standard Arabic (MSA) medical nuances better than multilingual alternatives.
• Hybrid Pooling Strategy: Instead of relying solely on the [CLS] token, the system concatenates two representations:
  1. Mean Pooling: Averages all token embeddings to capture global thematic context.
  2. Attention Pooling: A learnable mechanism that assigns weights to specific tokens, allowing the model to focus on salient medical keywords (symptoms, organs) while ignoring stop words.
• Regularization & Robustness:
  • Multi-Sample Dropout: Creates five parallel dropout paths with varying rates (0.1–0.3) during training. The logits from these paths are averaged, acting as an internal ensemble to stabilize decision boundaries for minority classes.
  • Label Smoothing: Applied a smoothing factor of 0.1 to the cross-entropy loss to prevent overconfidence in potentially noisy labels.
  • Layer-wise Learning Rate Decay (LLRD): Used a decay factor of 0.95 to ensure lower layers (linguistic features) adapt slowly while top layers (task-specific) adapt quickly.

Comparative Approaches (Benchmarks)

• Causal Decoder Feature Extraction: Extracted hidden states from the final layer of Qwen 3B and Llama 3.3 70B to serve as static features for a classification head.
• Zero-Shot Re-ranking Pipeline: A two-stage approach where AraBERT generated the top-15 candidates, and Llama 3.3 70B Instruct performed a final selection via structured prompting.

3. Key Contributions

• Empirical Evidence on Encoder vs. Decoder: The study provides strong evidence that for high-granularity, domain-specific classification, fine-tuned bidirectional encoders significantly outperform causal decoders.
• Identification of "Schema Mismatch": The authors demonstrate that while LLMs possess superior general medical knowledge, their causal nature (optimized for next-token prediction) produces sequence-biased embeddings that fail to capture the dense, global semantic boundaries required for 82-class discrimination.
• Robust Architecture for Imbalanced Data: The combination of Hybrid Pooling and Multi-Sample Dropout is shown to be an effective strategy for handling extreme class imbalance and label noise without requiring complex external data augmentation.

4. Results

The system was evaluated on the official test set using Macro-F1 as the primary metric to account for class imbalance.

Model Configuration	Macro-F1 Score
AraBERTv2 (Proposed)	0.3934
multilingual-E5-large	0.3804
CamelBERT	0.3603
AraBERTv2 + Llama 3.3 70B (Re-ranking)	0.3035
Qwen 3 3B (Feature Extraction)	0.1278

• Key Finding: The proposed AraBERTv2 configuration achieved the highest performance.
• Decoder Failure: Feature extraction from Qwen 3B yielded a very low score (0.1278).
• Re-ranking Degradation: Adding Llama 3.3 70B for re-ranking actually degraded performance (dropping from 0.3934 to 0.3035). The LLM frequently selected semantically logical labels that did not match the specific, often arbitrary, boundaries of the 82-class schema (e.g., choosing "Dermatology" instead of the specific task label "Skin and Beauty").

5. Significance and Conclusion

The paper concludes that for specialized, high-cardinality NLP tasks, the semantic compression capabilities of fine-tuned bidirectional encoders are superior to the generative capabilities of large causal decoders.

• Bidirectional Advantage: Encoders allow all tokens to attend to the entire context simultaneously, creating a dense vector that captures the full semantic meaning of a query.
• Causal Limitation: Causal decoders are biased toward sequence history and next-token prediction, making their hidden states less effective for discriminative classification tasks where the "global" meaning is scattered across the text.
• Practical Implication: For medical text classification with noisy labels and extreme imbalance, investing in robust fine-tuning of specialized encoders (with techniques like hybrid pooling and multi-sample dropout) is more effective than attempting to leverage massive LLMs via zero-shot or feature extraction methods.