StrokeID-NER : A Stroke Named Entity Recognition Dataset for Indonesian Patient- Generated Health Queries

Technical Summary: StrokeID-NER

Problem Statement
Stroke is the leading cause of death and disability in Indonesia, yet clinical Natural Language Processing (NLP) tools for stroke triage are hindered by a lack of domain-specific annotated resources for patient-generated text. While digital health platforms like Alodokter.com host vast archives of informal Indonesian health queries, these texts present unique NLP challenges: extreme orthographic variation, the use of regional dialects (e.g., Javanese), lay metaphors, code-mixing, and culturally specific temporal references (e.g., Islamic calendar events). Existing Indonesian medical NER datasets either focus on formal clinical records, health news, or mixed patient-physician dialogues, failing to address the specific linguistic and clinical complexities of informal, patient-only stroke queries.

Methodology
The study introduces StrokeID-NER, a specialized Named Entity Recognition (NER) dataset and benchmarking framework constructed through four phases:

1. Data Construction: The dataset comprises 1,742 patient queries filtered from the "Indonesia-medical-qna" dataset (sourced from Alodokter.com), focusing exclusively on stroke and hemorrhagic stroke topics. Physician responses were excluded to simulate real-world triage conditions relying solely on patient input.
2. Annotation Scheme: Queries were annotated using the BIO (Begin-Inside-Outside) scheme with nine tags across four clinically grounded entity types:
   • Symptom (SYM): Physical/neurological manifestations (e.g., lemas, drodog).
   • Diagnosis (DGN): Stroke subtypes and conditions (e.g., stroke ringan, pendarahan otak).
   • Temporal (TMP): Onset, duration, and timing (e.g., tiba-tiba, saat hari ke-20 puasa).
   • Risk Factor (RF): Pre-existing conditions and demographics (e.g., hipertensi, usia 65 tahun).
   • Inter-annotator agreement was assessed on a stratified sample, yielding a Cohen's Kappa of $\kappa = 0.857$.
3. Dataset Statistics: The corpus contains 6,765 entity spans. A defining characteristic is the high hapax rate (unique expressions appearing only once), ranging from 72.0% for Diagnosis to 86.9% for Risk Factors, reflecting the informal and diverse nature of the text.
4. Benchmarking: Five NER systems were evaluated on a stratified train/validation/test split (1,211/256/275 queries):
   • M1: IndoBERT-base + Linear head.
   • M2: IndoBERT-base + CRF layer.
   • M3: XLM-RoBERTa-large (multilingual) + Linear head.
   • M4: IndoBERT-large + Linear head.
   • Baseline: GPT-5.4 evaluated in a zero-shot setting.
   • Evaluation Metric: Span-level macro-averaged F1-score using seqeval.

Key Results

• Model Performance: The fine-tuned XLM-RoBERTa-large (M3) achieved the best performance with a macro F1 of 0.7823. It outperformed the monolingual IndoBERT-large (M4, F1=0.7614) and the zero-shot GPT-5.4 baseline (F1=0.6682) by 2.1 and 11.4 points, respectively.
• Entity-Specific Insights:
  • Diagnosis (DGN) was the easiest entity to recognize (F1 0.86–0.89) due to the high frequency of the term "stroke" and its variants.
  • Symptom (SYM) and Risk Factor (RF) were the most difficult, correlating with their high hapax rates.
  • The zero-shot LLM performed comparably on DGN but lagged significantly on SYM and RF, failing to capture informal expressions and regional terminology.
• Error Analysis:
  • Hapax Constraint: A strong negative correlation exists between hapax rate and F1 score. 68.3% of false negative spans were missed by all four fine-tuned models, indicating a performance ceiling driven by lexical coverage rather than model architecture.
  • Fine-tuning vs. Zero-shot: The fine-tuned model correctly identified 204 spans that the zero-shot model missed, compared to only 57 in the reverse direction (3.6:1 ratio). This advantage was most pronounced for SYM (4.6:1) and RF (7.0:1).
  • Binary vs. Joint Training: Training binary classifiers for specific entities improved performance for DGN and TMP but degraded performance for RF by 0.08 F1 points, suggesting that cross-entity context is crucial for identifying risk factors.

Significance and Claims
The paper claims four primary contributions:

1. First Public Corpus: It releases the first publicly available, clinically grounded corpus for stroke NER in Indonesian, specifically targeting informal patient-generated text.
2. Benchmarking Insights: It demonstrates that for informal clinical NER in low-resource languages, multilingual pretraining (XLM-RoBERTa) provides greater performance gains than increasing monolingual model scale (IndoBERT-large), particularly for entities with high lexical diversity.
3. Linguistic Analysis: It documents the unique linguistic phenomena of Indonesian stroke queries, including orthographic variation, Javanese colloquialisms, and culturally specific temporal expressions, which standard NLP resources fail to capture.
4. Fine-tuning Necessity: It empirically validates that domain-specific fine-tuning substantially outperforms zero-shot prompting of large language models for this task, especially when dealing with informal language and clinical context.

The authors conclude that future improvements in performance are constrained by lexical coverage rather than model architecture. They posit that expanding the training vocabulary through additional annotation or data augmentation is the most effective path forward, rather than architectural modifications. The dataset, guidelines, and models are publicly released to support further research in Indonesian clinical NLP.