FMI@SU ToxHabits: Evaluating LLMs Performance on Toxic Habit Extraction in Spanish Clinical Texts

1. Problem Statement

The paper addresses the challenge of Named Entity Recognition (NER) within unstructured Spanish clinical texts, specifically focusing on the extraction of toxic habits (substance use and abuse).

• Context: Approximately 80% of clinical data is unstructured, hindering automated analysis. Substance abuse is a major global health issue, yet extracting structured data regarding drug use from clinical case reports remains difficult.
• Task: The authors participated in Subtask 1 of the ToxHabits Shared Task (part of BioCreative IX). The goal was to detect "trigger phrases" referring to substance use in Spanish clinical case reports and classify them into four categories: Tobacco, Alcohol, Cannabis, and Drugs.
• Challenge: The task involves identifying precise entity boundaries (start/end indices) and correct classification in a non-English language (Spanish) using clinical jargon, where LLMs often struggle with exact span matching.

2. Methodology

The team proposed an approach leveraging Large Language Models (LLMs) and In-Context Learning (ICL) rather than traditional supervised fine-tuning.

• Model Selection: After preliminary testing with GPT-4o, GPT-4.1, and GPT-4.5 preview, GPT-4.1 (via private Azure deployment) was selected as the best-performing model.
• Data Processing:
  • Clinical reports were split into sections (based on double newlines) to reduce input length, as shorter texts yielded better entity retrieval.
  • The DsPy library was used for prompt engineering and optimization.
• Prompting Strategies:
  • Few-Shot Learning: The core method involved providing the model with 5 randomly sampled examples from the training set to guide the extraction.
  • Structured Output: Prompts were designed to extract phrases for each of the four categories.
  • Span Retrieval: Extracted phrases were searched within the original text to determine character spans. In cases of overlap, the shorter span was selected, aligning with the tendency of human annotators to label shorter triggers.
• Optimization Techniques:
  • Manual Prompt Optimization: Iteratively refining the prompt instructions to improve precision and handle negation.
  • Automated Optimization: Used MIPROv2 (Multiprompt Instruction Proposal Optimizer) to optimize prompts based on Intersection over Union (IoU) metrics at the character level.
  • Hyperparameters: Experiments used temperature 0 (for consistency), top_p 1, and max tokens ranging from 4,000 to 16,000 depending on the optimization phase.
• Baselines:
  • A Dictionary-based approach (unambiguous labeled mentions).
  • A Spanish Clinical RoBERTa model (token classification).

3. Key Contributions

• LLM Evaluation in Spanish Clinical NER: The study provides empirical evidence on the performance of state-of-the-art LLMs (GPT-4.1) for NER in Spanish clinical domains, a area less explored than English.
• Few-Shot vs. Zero-Shot Analysis: Demonstrated that few-shot prompting significantly outperforms zero-shot prompting (approx. 6% F1 improvement) and dictionary baselines for this specific task.
• Error Analysis & Boundary Refinement: Identified that the primary failure mode of LLMs is boundary detection (extracting the trigger plus surrounding context like negation or dosage) rather than missing the entity entirely. The paper proposes post-processing strategies to refine these spans.
• Handling Negation: Highlighted the critical role of correctly processing negation (e.g., distinguishing "non-smoker" from "smoker") and how prompt engineering can improve assertion classification.

4. Results

The experiments were conducted on a dataset of 1,500 Spanish clinical reports (1,200 train, 300 dev, 300 test).

Performance on Test Set (Subtask 1):

• Best Model: GPT-4.1 Few-Shot (5 examples).
  • Precision: 0.59
  • Recall: 0.72
  • F1 Score: 0.65
• Comparison:
  • The Dictionary Baseline achieved the highest Recall (0.87) but lower Precision (0.46), resulting in an F1 of 0.60.
  • Spanish Clinical RoBERTa (trained on dev set) achieved a high F1 of 0.79 on the dev set but was not the top submission on the final test set in this specific comparison context (the paper notes LLMs are promising but currently lag behind fine-tuned BERT models in raw F1).
  • Hybrid Approaches: Combining the Dictionary and GPT models decreased performance, suggesting the combination strategy was ineffective.
• Partial Match Metrics: When evaluating how much of the "Gold" entity was contained within the predicted span (GC metric), the models performed well (up to 85%), indicating that while exact boundaries were often off, the core entity was usually captured.

Key Observations:

• GPT-4.5 Preview performed poorly, likely due to hallucinations in entity typing.
• Manual prompt optimization yielded a slight improvement (approx. 1% F1) over automated optimization.
• The model tended to over-extract specific drug names (e.g., prescription drugs like clonazepam) that were not labeled as "abuse" by human annotators, and under-extracted common reference words (e.g., "consumers").

5. Significance and Future Work

• Multilingual Potential: The results demonstrate that LLMs are a viable, flexible alternative for clinical NER in low-resource languages (like Spanish) where large annotated datasets for fine-tuning may be scarce. The ability to use few-shot learning reduces the dependency on massive labeled corpora.
• Future Directions:
  • kNN Few-Shot: Implementing semantic similarity-based example selection (kNN) to choose the most relevant few-shot examples for each input, rather than random sampling.
  • Post-Processing: Developing dedicated steps to refine entity boundaries and filter out false positives (e.g., prescription drugs not associated with abuse).
  • Unified Extraction: Exploring joint extraction of triggers and attributes (dosage, frequency) in a single pass, leveraging the LLM's generative capabilities.
  • Annotation Consistency: Addressing the variance in human annotation guidelines, which affects the evaluation of model performance.

In conclusion, while fine-tuned BERT models currently hold a performance edge, the study validates GPT-4.1 with few-shot prompting as a highly promising, adaptable approach for extracting toxic habit entities from Spanish clinical texts, achieving an F1 score of 0.65 without the need for extensive model retraining.