Team Fusion@ SU@ BC8 SympTEMIST track: transformer-based approach for symptom recognition and linking

1. Problem Statement

The paper addresses the SympTEMIST shared task, which focuses on processing Spanish clinical texts. The goal is to detect and normalize symptoms, signs, and findings. The team participated in two specific subtasks:

• Subtask 1 (Named Entity Recognition - NER): Identifying spans of text in clinical reports that correspond to medical symptoms.
• Subtask 2 (Entity Linking - EL): Mapping the identified symptom mentions to their corresponding standardized codes in SNOMED-CT.

The challenge involves handling the complexity of Spanish clinical language, varying mention lengths, and the need for precise normalization against a large medical knowledge base.

2. Methodology

The team employed a transformer-based architecture for both subtasks, utilizing specific pre-trained models and data augmentation strategies.

Subtask 1: Named Entity Recognition (NER)

• Preprocessing: Clinical reports were split into sentences using the SPACCC Sentence Splitter to handle texts exceeding the 512-token limit of transformer models.
• Model Architecture:
  • Base Models: Two models were evaluated:
    1. PlanTL-GOB-ES/roberta-base-biomedical-clinical-es: A Spanish RoBERTa model pre-trained on a massive (1B+ tokens) Spanish biomedical corpus.
    2. CLIN-X-ES: A cross-lingual XLM-RoBERTa (large) model pre-trained on MeSpEN and Scielo clinical documents.
  • Head Layers: Token classification was performed using a linear layer followed by a Conditional Random Field (CRF). The team also experimented with inserting a 2-layer BiLSTM between the transformer output and the linear layer to capture long-term dependencies.
• Data Augmentation:
  • Synonym Replacement: Annotated mentions were randomly replaced with synonyms from the Spanish UMLS to generate 1,672 additional training sentences.
  • Further Pre-training: The Spanish RoBERTa model was further pre-trained on a dataset of 337,039 Spanish UMLS synonyms using Masked Language Modeling (MLM) for 4 epochs.

Subtask 2: Entity Linking (EL)

• Pipeline: A two-step approach was used:
  1. Candidate Generation: Matching the mention to aliases in a Knowledge Base (KB).
  2. Ranking: Using SapBERT XLMR-Large embeddings to calculate cosine similarity between the mention and KB aliases to retrieve the closest match.
• Knowledge Base Construction: The KB was constructed by combining the SympTEMIST gazetteer, the training set annotations, and a custom dataset of Spanish UMLS synonyms corresponding to SNOMED-CT codes.
• Handling Long Mentions: To address mentions longer than the model's context window, a sliding window strategy was implemented. The final score for a code was a weighted linear combination of cosine similarities calculated on:
  • The full mention.
  • The first 75% of tokens.
  • The last 75% of tokens.
  • Optimal weights found via grid search: 0.75, 0.17, and 0.08 respectively.

3. Key Contributions & Findings

• Model Selection for NER: The specialized monolingual Spanish RoBERTa model outperformed the cross-lingual CLIN-X-ES model. This highlights the importance of domain-specific pre-training for clinical NER in Spanish.
• Impact of Architecture: Adding a BiLSTM layer to the RoBERTa+CRF pipeline improved both precision and recall, likely due to better handling of sequential dependencies.
• Data Augmentation Efficacy:
  • For RoBERTa, synonym-based augmentation significantly improved Precision and F1 scores on the test set, despite a slight drop in Recall.
  • For CLIN-X-ES, augmentation did not yield consistent improvements, suggesting the model was less sensitive to this specific type of noise or augmentation.
• Knowledge Base Dominance: For Entity Linking, the choice of Knowledge Base was the single most critical factor. Models utilizing a KB that combined the Gazetteer, Training Set, and UMLS synonyms achieved the highest accuracy.
• Sliding Window Strategy: The sliding window approach for long mentions provided a ~2% accuracy boost over the standard SapBERT model, indicating that the most discriminative information for linking is often concentrated in the beginning of the mention.
• Negative Result: Further pre-training the base RoBERTa model on the UMLS synonym dataset did not improve NER performance, suggesting the base model was already sufficiently specialized.

4. Results

Subtask 1 (NER) Performance:

• Best Model: Augmented Spanish RoBERTa + CRF.
• Test Set F1 Score: 0.725 (Precision: 0.732, Recall: 0.718).
• The addition of BiLSTM to the augmented RoBERTa model resulted in a slightly lower F1 (0.732 on validation, 0.732 on test) compared to the CRF-only augmented version on the test set, though it improved recall.

Subtask 2 (Entity Linking) Performance:

• Best Model: SapBERT XLMR-Large with the "Gazetteer + Train + UMLS" Knowledge Base.
• Test Set Accuracy: 58.9%.
• Comparison:
  • Standard SapBERT (Gazetteer + Train): 58.8%
  • SapBERT + Sliding Window (Augmented KB): 58.7%
  • Note: A specific experiment with a buggy KB generation resulted in a drastic drop to ~1.7% accuracy, underscoring the sensitivity of the system to KB quality.

5. Significance

This work demonstrates that for Spanish clinical NER, domain-specific monolingual transformers (Spanish RoBERTa) are superior to cross-lingual alternatives, especially when combined with CRF layers and targeted data augmentation. For Entity Linking, the study emphasizes that knowledge base curation is more impactful than architectural tweaks; integrating external resources like UMLS synonyms significantly enhances linking accuracy. The proposed sliding window technique offers a practical solution for handling long clinical entity mentions, a common challenge in medical text processing.

The research was supported by the European Union-NextGenerationEU through the National Recovery and Resilience Plan of Bulgaria.