💬 NLP

BioUNER: A Benchmark Dataset for Clinical Urdu Named Entity Recognition

This paper introduces BioUNER, a gold-standard benchmark dataset for Biomedical Urdu Named Entity Recognition created from 153K tokens of medical text with high inter-annotator agreement, and validates its utility by evaluating various machine learning and deep learning models.

Original authors: Wazir Ali, Adeeb Noor, Sanaullah Mahar, Alia, Muhammad Mazhar Younas

Published 2026-04-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wazir Ali, Adeeb Noor, Sanaullah Mahar, Alia, Muhammad Mazhar Younas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to read a doctor's prescription written in Urdu. The robot needs to understand that the word "Diabetes" is a disease, "Insulin" is a drug, and "High Blood Pressure" is a condition. But here's the problem: until now, the robot had no good textbooks or practice materials in Urdu to learn this. It was like trying to teach someone to drive a car without ever letting them sit behind the wheel.

This paper, "BioUNER," is about building that very first, high-quality driving school for Urdu medical text.

Here is the story of how they did it, broken down into simple parts:

1. The Problem: A Language Gap

In the world of computers and medicine, English has thousands of "gold-standard" textbooks (datasets) that teach computers how to find medical terms. Spanish, Chinese, and German have them too. But for Urdu, a language spoken by millions, there was almost nothing.

The Analogy: Imagine English has a massive library of medical flashcards, but the Urdu section is completely empty. Doctors and researchers in Urdu-speaking regions were stuck because computers couldn't understand their medical notes.

2. The Solution: Building the "Gold-Standard" Dataset

The authors decided to build this library from scratch. They called their new dataset BioUNER.

Gathering the Books: They didn't just make up sentences. They went online and "crawled" (scraped) real-world content from Urdu health blogs, hospital websites, and daily newspapers. They collected 153,000 words of real medical text.
Cleaning the Mess: Just like cleaning a messy room before organizing it, they removed ads, weird symbols, and formatting errors.
The Human Touch (Annotation): This is the most important part. They didn't let a computer do the labeling; they hired three human experts who are native Urdu speakers and know medicine.
- They used a digital tool called Doccano (think of it like a high-tech highlighter).
- These experts went through the text and highlighted specific words, tagging them as "Disease," "Drug," "Gene," or "Chemical."
- The Result: They created a "Gold-Standard" dataset. "Gold-Standard" means it's the best possible version, verified by humans, which other computers can use to learn.

3. The Test: Teaching the Robots

Once they had the "textbook" (the dataset), they needed to see if different types of "students" (computer models) could learn from it. They tested four different types of learners:

SVM (The Old School Student): A traditional method that looks at individual words but doesn't understand context. It's like memorizing a dictionary without knowing how to form sentences.
CRF & LSTM (The Context Learners): These are smarter. They look at the words before and after a target word to understand the meaning.
mBERT & XLM-RoBERTa (The Super-Genius Translators): These are massive, modern AI models that have read the internet in over 100 languages. They are supposed to be the smartest.

4. The Surprising Results

Usually, in the world of AI, the "Super-Genius" models (the Transformers like XLM-RoBERTa) win every time. But here, something interesting happened:

The Winner: The LSTM model (the "Context Learner") actually performed the best, with a score of 95% accuracy.
The Runner-Up: The "Super-Genius" models (XLM-RoBERTa) came in second with about 73% accuracy.
Why? The authors suggest that because Urdu is a "low-resource" language (there isn't that much data available for the Super-Genius models to have learned from previously), the simpler, specialized models that focused strictly on the new Urdu medical data actually did a better job than the giants that tried to do everything at once.

5. Why This Matters

This paper is a big deal for three reasons:

It's a Benchmark: It gives everyone a standard "test" to see how good their Urdu medical AI is. Before this, nobody knew if their AI was good or bad because there was no test.
It's Free: They are releasing this dataset and the trained models to the public (on a site called Hugging Face). Anyone can download it and start building better health tools for Urdu speakers.
It Saves Lives: By helping computers understand Urdu medical notes, we can eventually build tools that automatically summarize patient records, suggest treatments, or spot disease outbreaks faster in Urdu-speaking communities.

In a Nutshell:
The authors built the first-ever high-quality "medical dictionary" for Urdu computers. They proved that for this specific task, a focused, specialized AI learner actually outperformed the massive, general-purpose AI giants. This opens the door for better healthcare technology for millions of Urdu speakers.

1. Problem Statement

Clinical Named Entity Recognition (NER) is a critical NLP task aimed at extracting structured information (e.g., diseases, drugs, genes) from unstructured clinical narratives like electronic health records (EHRs) and medical notes. While high-quality, gold-standard datasets exist for resource-rich languages (English, Chinese, Spanish) and general-domain datasets exist for South Asian languages (Urdu, Hindi, Bengali), there is a significant gap in clinical NER resources for Urdu.

The Challenge: Urdu lacks publicly available, annotated datasets specifically for the biomedical/clinical domain. Existing Urdu NER resources focus on general news or social media, failing to address the specific terminology, nested entities, and semantic ambiguities found in medical texts.
The Goal: To create a gold-standard benchmark dataset (BioUNER) to facilitate the development and evaluation of clinical NER systems for the Urdu language.

2. Methodology

A. Data Collection & Preprocessing

Sources: The dataset was constructed by crawling real-world clinical text from diverse reliable sources, including:
- Hospital patient education materials (e.g., Aga Khan University, Continental Hospitals).
- Medical advice columns and health articles (e.g., Daily Jang, UrduPoint).
- Content from the Royal College of Psychiatrists and Centre for Health Protection.
Preprocessing: The raw text underwent rigorous cleaning:
- Removal of HTML tags, non-Urdu characters, and noise (ads).
- Normalization of script-specific variations (zero-width non-joiners, diacritics).
- Sentence segmentation to prepare data for labeling.

B. Annotation Strategy

Annotators: Three native Urdu speakers with domain expertise in the medical field.
Tool: The Doccano text annotation tool was used.
Schema: A BIES (Begin, Inside, End, Single) labeling scheme was adopted to explicitly mark entity boundaries.
Entity Categories: Six specific biomedical entity types were defined:
1. Disease (Most frequent)
2. Chemical
3. Drug
4. Protein
5. CellLine
6. Gene
Quality Control: Disagreements were resolved through adjudication. The dataset achieved an Inter-Annotator Agreement (IAA) score of $\kappa = 0.78$ , indicating substantial agreement and gold-standard quality.

C. Dataset Statistics

Total Tokens: ~153,000.
Total Entity Mentions: 15,073.
Distribution: The corpus is disease-centric (8,959 mentions), followed by Chemicals (2,557) and Drugs (2,185). Molecular entities (Genes, CellLines) are less frequent.

D. Experimental Setup & Models

The authors evaluated the dataset using a supervised sequence labeling approach with a 90/10 train-test split. They benchmarked four categories of models:

Traditional ML: Support Vector Machines (SVM) using TF-IDF features from character n-grams.
Sequence Models: Conditional Random Fields (CRF) and Hidden Markov Models (HMM) to model label transitions.
Deep Learning: Long Short-Term Memory (LSTM) networks with 128 hidden units and dropout.
Pre-trained Transformers: Multilingual BERT (mBERT) and XLM-RoBERTa, fine-tuned for the specific task.

3. Key Contributions

BioUNER Dataset: The first publicly available, gold-standard benchmark dataset for Clinical Urdu NER, containing 153K tokens and 15K+ annotated entities across six biomedical categories.
Comprehensive Benchmarking: A rigorous evaluation of the dataset using both traditional machine learning (SVM, CRF) and state-of-the-art deep learning models (LSTM, mBERT, XLM-RoBERTa).
Resource Release: Public availability of the dataset and fine-tuned masked language models on Hugging Face to support future research in low-resource biomedical NLP.
Handling Complexity: The dataset explicitly addresses challenges such as nested named entities and multi-word expressions common in clinical Urdu text.

4. Results and Analysis

The models were evaluated using Precision, Recall, and F1-score.

Baseline Performance:
- SVM: Performed poorly (F1: 0.65) due to the inability to model contextual dependencies.
- CRF: Showed strong performance (F1: 0.93) by modeling label transitions.
- LSTM: Achieved the highest overall performance with an F1-score of 0.95. The authors attribute this to the model's ability to capture sequential dependencies and contextual information effectively in this specific low-resource setting.
Transformer Performance:
- mBERT: Achieved an F1-score of 0.72.
- XLM-RoBERTa: Slightly outperformed mBERT with an F1-score of 0.73.
- Observation: Surprisingly, the transformer models underperformed compared to the LSTM and CRF baselines in this specific experimental setup. The authors suggest this may be due to the limited size of the training data relative to the complexity of the pre-trained models, or the specific nature of the Urdu clinical corpus.

5. Significance

Filling a Resource Gap: BioUNER fills a critical void in NLP resources for Urdu, specifically in the high-stakes domain of healthcare.
Clinical Decision Support: By enabling the extraction of structured medical data from Urdu texts, this work supports clinical decision-making, medical research, and healthcare analytics in Urdu-speaking regions (Pakistan, India, and diaspora).
Benchmarking Standard: It provides a standardized evaluation ground for future researchers to develop and compare NER models for low-resource, domain-specific languages.
Future Directions: The authors plan to expand the dataset's size and diversity to improve model generalization and robustness, potentially addressing the performance gap observed between traditional neural baselines and large pre-trained transformers.