FigSIM: A Dataset for Fine-grained Suicide Severity and Figurative Language in Suicide Memes

Technical Summary: FigSIM: A Dataset for Fine-grained Suicide Severity and Figurative Language in Suicide Memes

Problem Statement
Suicide memes—multimodal posts combining visual templates with user-generated text to express suicide-related thoughts—are increasingly prevalent on social media. While they can serve as coping mechanisms, they also pose significant risks, including the potential to trigger distress, obscure personal pain, or inadvertently encourage imitative behavior. Current content moderation strategies and automated detection systems face critical limitations in addressing this phenomenon. Existing research has largely focused on binary classification (suicidal vs. non-suicidal) using text or naturalistic images, often failing to capture the nuanced, implicit, and figurative nature of meme communication. Furthermore, there is a lack of annotated datasets specifically designed to analyze the intersection of figurative language (e.g., metaphors, irony) and fine-grained suicide severity within the meme format. This gap hinders the development of context-aware moderation frameworks capable of distinguishing between harmful content and supportive or coping-oriented expression.

Methodology
The authors introduce FigSIM, the first dataset designed for the fine-grained analysis of suicide memes. The methodology encompasses dataset construction, annotation, and extensive benchmarking.

• Data Collection: The dataset comprises 1,049 memes collected from the Reddit community r/SuicideMeme (April 2018–December 2022). Images were filtered to ensure English text readability via OCR and the removal of near-duplicates.
• Annotation Scheme: A multidimensional annotation framework was developed in consultation with psychologists and suicide prevention researchers, grounded in clinical scales (Columbia-Suicide Severity Rating Scale) and media guidelines (#chatsafe, Mindframe). The scheme covers five categories:
  1. Figurative Phenomenon: Multi-label detection of Metaphor, Pun/Double meaning, and Irony/Sarcasm.
  2. Suicide Severity Scale: An ordinal scale ranging from "None" to "Wish to be dead," "Suicide ideation," "Suicide planning," "Suicide attempts," and "Suicide death."
  3. Suicide-Related Content: Multi-label detection of stylized vs. naturalistic depictions of methods/locations, protective factors, harmful factors, and third-person descriptions.
  4. Modality: Classification as text-only, image-only, or complementary (requiring both).
  5. Context: Determination of whether external knowledge is required for interpretation.
• Annotation Process: Two rounds of pilot annotation refined the scheme. Large-scale annotation was performed by two independent annotators (one from psychology, one from computing) with a third adjudicating disagreements. Inter-annotator agreement (IAA) was substantial to almost perfect for severity and content, but moderate for figurative phenomena and context, reflecting inherent subjectivity.
• Benchmarking: The authors evaluated 16 unimodal and multimodal models across three tasks: figurative language detection, suicide severity detection, and suicide-related content detection. Baselines included text-only models (BERT, MentalBERT, RoBERTa), image-only models (ResNet-50, ViT, DINOv2), and multimodal models (CLIP, BLIP-2). Additionally, several Multimodal Large Language Models (MLLMs) were tested using zero-shot and few-shot prompting, alongside specialized mental health meme models.

Key Contributions

1. FigSIM Dataset: A novel, fine-grained dataset of 1,049 suicide memes with manual annotations for figurative phenomena, clinical suicide severity, and specific content types.
2. Comprehensive Benchmark: A systematic evaluation of 16 models (20 configurations) across unimodal and multimodal architectures, establishing performance baselines for suicide meme analysis.
3. Empirical Analysis: Thorough analysis of dataset characteristics, model performance, and error patterns, highlighting specific challenges such as the underprediction of high-severity risks in figurative contexts.

Results

• Model Performance: Multimodal Large Language Models (MLLMs) consistently outperformed unimodal baselines. The best-performing model, Claude-sonnet-4-5 (zero-shot), achieved macro-F1 scores of 70.21% for figurative detection, 71.60% for suicide severity, and 58.51% for suicide-related content. Commercial MLLMs generally outperformed open-source counterparts.
• Modality Insights: Language models significantly outperformed image-only models, suggesting linguistic cues are primary for inferring suicide intent. However, multimodal approaches yielded the best overall results.
• Error Analysis:
  • Severity Bias: Models exhibited a tendency to over-predict lower-severity categories (e.g., "None," "Wish to be dead") and under-predict higher-severity categories (e.g., "Suicide planning," "Suicide attempts/death").
  • Figurative Impact: This underprediction bias was significantly more pronounced for memes containing figurative language, particularly metaphors. Even when figurative cues were correctly identified, models struggled to map indirect expressions to appropriate severity levels.
  • Context and Modality: Models struggled more with memes requiring external context or those relying on a single modality (text-only or image-only), though these cases were less frequent in the dataset.
• Moderation Analysis: Evaluation of existing moderation tools (Azure AI, OpenAI, Llama-Guard) showed that while flag rates increased with suicide severity, figurative memes often received lower severity scores than non-figurative ones, indicating a potential blind spot in current automated moderation.

Significance and Claims
The paper posits that suicide memes present unique challenges for both modeling and content moderation due to their inherent ambiguity, humor, and reliance on figurative language. The authors claim that FigSIM provides a necessary foundation for:

• Advancing Research: Enabling the study of fine-grained suicide risk detection beyond binary classification and text-only analysis.
• Improving Moderation: Informing the development of context-aware and domain-specific moderation strategies that can better distinguish between harmful content and coping mechanisms.
• Addressing Gaps: Highlighting the limitations of current models in interpreting indirect expressions and the specific difficulty of assessing severity when figurative language is present.

The authors maintain a modest stance, acknowledging that the dataset is derived from a single source (Reddit) and that annotations reflect interpretive judgments rather than clinical diagnoses. They emphasize that the dataset is intended to support hypothesis generation and evidence-building for safer online communication, rather than serving as a diagnostic tool. The work underscores the urgent need for more nuanced approaches to understanding and moderating suicide-related content in the evolving landscape of social media memes.