Clinical named entity recognition in the Portuguese language: a benchmark of modern BERT models and LLMs

1. Problem Statement

Clinical notes contain vast amounts of unstructured data critical for healthcare management, research, and quality improvement. While Named Entity Recognition (NER) can automate the extraction of medical concepts, there is a significant scarcity of benchmarks and high-performing models specifically for Portuguese clinical text. Furthermore, existing studies often rely on public datasets that may not generalize to private, real-world hospital data. A specific challenge in this domain is multilabel class imbalance, where rare medical entities (e.g., specific genetic markers) are underrepresented compared to common ones, hindering model performance.

2. Methodology

Datasets

The study utilized two distinct datasets:

1. SemClinBr (Public): A semantically annotated corpus of 1,000 clinical notes from 12+ medical specialties (inpatient and outpatient). It covers 15 UMLS semantic groups, with 12 groups containing sufficient data for training.
2. Breast Cancer Dataset (Private): A restricted-access dataset of 500 ambulatory notes from a Brazilian oncology center. It focuses on 10 specific entities relevant to breast cancer care (e.g., ER, PR, HER2, BRCA, metastasis status). Annotations were performed by a team of medical students and doctors, with consensus reached on disagreements.

Models Evaluated

The authors compared a range of BERT-based encoders and Large Language Models (LLMs):

• BERT Models:
  • BioBERTpt: Variants trained on clinical notes (biobertpt-clin), biomedical literature (biobertpt-bio), and a combination (biobertpt-all).
  • BERTimbau: A general Portuguese BERT trained on the BRWAC corpus.
  • ModernBERT: Models incorporating architectural optimizations (e.g., rotary positional embeddings, flash attention).
  • mmBERT: A multilingual model with a novel pre-training protocol designed to improve low-resource language representation by dynamically adjusting language ratios during training.
• LLMs (Prompted with Few-Shot):
  • OpenAI: GPT-4.1, GPT-5 (various sizes: nano, mini, full) with varying reasoning efforts (minimal to high).
  • Google: Gemini-2.5 (flash, flash-lite, pro).
  • Note: LLMs were evaluated only on the public SemClinBr dataset to preserve the privacy of the private breast cancer data.

Experimental Setup & Strategies

• Task Formulation: Framed as a multilabel token classification task using IO (Inside/Outside) and BIO (Begin/Inside/Outside) tagging schemes.
• Data Splitting: Compared Simple Random Splitting vs. Iterative Stratification. Iterative stratification was used to preserve the distribution of rare classes across train/validation/test sets.
• Imbalance Handling: Tested three strategies:
  1. Weighted Loss: Assigning higher weights to rare classes.
  2. Oversampling: Replicating rare class examples.
  3. Clamped Weights: Setting a minimum weight floor (e.g., 1.0) to prevent weights for frequent classes from becoming too small.
• Metrics: Precision, Recall, and F1-score (both Micro and Macro averages).

3. Key Results

Model Performance

• Best Performer: mmBERT-base achieved the state-of-the-art results on both datasets.
  • SemClinBr: Micro F1-score of 0.7646.
  • Breast Cancer: Micro F1-score of 0.7723.
• Comparison: mmBERT outperformed BioBERTpt, ModernBERT, BERTimbau, and all tested LLMs.
• LLM Performance: Even with increased reasoning effort (GPT-5), LLMs failed to surpass the best BERT models. Furthermore, LLMs incurred significantly higher costs and inference times (e.g., GPT-5 high reasoning cost ~$84 per 1000 responses vs. local BERT inference).
• Annotation Scheme: The IO scheme generally yielded better Macro F1-scores than BIO, particularly for mmBERT and ModernBERT, suggesting architectural sensitivity to label space expansion.

Impact of Imbalance Strategies

• Iterative Stratification (IS): This was the most critical factor. On SemClinBr, switching from random splitting to IS improved the Micro F1-score of mmBERT from 0.6904 to 0.7646.
• Dataset-Specific Optimization:
  • SemClinBr: IS alone was sufficient for peak performance.
  • Breast Cancer: The best results required IS combined with Weighted Loss (with weights clamped at 1.0).
• Oversampling: Generally provided marginal or no improvement over IS alone in this specific context.

4. Key Contributions

1. New State-of-the-Art: Established a new benchmark for Portuguese clinical NER, with mmBERT-base setting a new record on the SemClinBr dataset.
2. Private Data Validation: Demonstrated that high performance on public datasets translates effectively to private, domain-specific clinical data (breast cancer), validating the reproducibility of results in real-world settings.
3. Imbalance Strategy Analysis: Systematically quantified the impact of data splitting and loss weighting. The study concludes that iterative stratification is essential for multilabel clinical data, while weighted loss strategies must be tuned (e.g., clamping weights) based on the specific dataset characteristics.
4. LLM vs. Encoder Efficiency: Provided evidence that for clinical NER, specialized encoder models (BERT) are superior to LLMs in terms of accuracy, cost, latency, and data privacy (as they can run locally).

5. Significance and Future Work

• Clinical Adoption: The study proves that high-quality Portuguese clinical NER systems can be built within healthcare institutions using modest annotation efforts and local hardware (e.g., a single RTX 4090 GPU), without relying on expensive cloud APIs.
• Data Privacy: The success of local BERT models highlights a viable path for processing sensitive health data without privacy risks associated with sending data to external LLM providers.
• Future Directions:
  • Domain-Specific Pre-training: Fine-tuning multilingual models like mmBERT specifically on Portuguese biomedical corpora.
  • LLM Optimization: Exploring prompt engineering (e.g., GEPA) or fine-tuning small language models for NER.
  • Generalization: Testing these models across different medical specialties and regions to ensure robustness.

In conclusion, this paper establishes that multilingual BERT models, particularly mmBERT, combined with rigorous data splitting strategies (iterative stratification), are the most effective and practical solution for clinical NER in Portuguese, outperforming both general-purpose LLMs and previous domain-specific models.