Before the Labels: How Dataset Construction Shapes Suicidality Detection in Clinical Text

Technical Summary: Before the Labels: How Dataset Construction Shapes Suicidality Detection in Clinical Text

Problem Statement
The paper addresses a critical gap in Clinical Natural Language Processing (NLP) regarding the construction of Electronic Health Record (EHR) datasets for suicidality detection. While recent work has shifted from social media to EHRs to mitigate issues of consent and self-report reliability, the authors argue that EHR-based datasets are often treated as "neutral ground truth." The paper posits that this framing obscures the specific operationalization of suicidality encoded within these datasets. Clinical notes are "documentation-mediated records" that synthesize patient reports, clinician observations, and institutional requirements; they do not provide direct access to the patient's internal state. Consequently, treating dataset labels as absolute truth ignores the structural assumptions embedded in how data is governed, selected, and annotated.

Methodology
The authors employ a qualitative and quantitative case study of the ScAN dataset (Rawat et al., 2022), a suicide attempt (SA) and suicidal ideation (SI) annotation layer built over the MIMIC-III clinical notes. The methodology involves three primary components:

1. Operationalization Analysis: The authors deconstruct the ScAN pipeline to identify three defining dimensions of its operationalization:

   • Documentation-Mediated: Analyzing how governance (PhysioNet credentialing), access constraints, and the exclusion of patient-authored text shape the perspective to clinician-authored documentation only.
   • Episodic: Examining how cohort selection (ICD codes) and labeling at the "hospital-stay" level flatten longitudinal processes into bounded episodes.
   • Intent-Resolved: Investigating how annotation schemes distinguish suicidal from non-suicidal self-harm and how ambiguous categories (e.g., "unsure") are collapsed in downstream pipelines.

2. Linguistic Framing Analysis: The authors perform a descriptive linguistic analysis on 15,585 annotated spans. Using lexical indicators and the MedSpaCy ConText algorithm, they classify spans based on the presence of:

   • Negation
   • Historical reference (temporal displacement)
   • Uncertainty markers
     They compare these profiles across different label types (e.g., "Present-SI" vs. "Absent-SI") and documentation contexts (e.g., Discharge Summaries vs. History of Present Illness).

3. Heterogeneity Assessment: The study tests the hypothesis that identical labels subsume heterogeneous clinical framings, specifically looking for variation in temporality, negation, and uncertainty within the same label class.

Key Contributions
The paper makes three primary contributions:

• A Positional Framework: It characterizes EHR-based suicidality datasets as encoding a specific operationalization that is documentation-mediated (reflecting clinician records rather than unmediated states), episodic (bounding suicidality to discrete hospital stays), and intent-resolved (imposing binary distinctions even when text is ambiguous).
• A Case Study of ScAN: It traces how specific governance, data sourcing, and annotation decisions (single-annotator labeling, ICD-based cohort selection, and label collapsing) give rise to this operationalization.
• Empirical Evidence of Linguistic Variation: It provides data demonstrating that identical labels contain systematic variations in linguistic framing, challenging the assumption that labels represent uniform clinical phenomena.

Results
The linguistic analysis reveals significant heterogeneity within label categories that are often treated as uniform:

• Temporal Flattening: 27.8% of spans labeled as "Present-SI" contain historical markers, indicating that ideation labeled as "present" is often documented in relation to prior psychiatric history rather than as a current event.
• Contextual Variation: The "unmodified" rate (absence of negation, history, or uncertainty) varies drastically by clinical note section. Discharge summaries are 83.6% unmodified, whereas History of Present Illness (HPI) sections are only 49.9% unmodified.
• Collapse of Uncertainty: The "Unsure" category, which retains 17.1% uncertainty markers, is frequently collapsed with "Negative" labels in downstream modeling. This merges qualitatively different situations: explicit negation of intent, unspecified intent, and genuine ambiguity.
• Intent Resolution: "Unspecified-intent" spans (T14.91) are predominantly historical (67.8%), suggesting that the lack of intent specification often co-occurs with temporal displacement.

Significance and Claims
The paper does not claim that clinical documentation is unreliable or that patient self-reports are bias-free. Instead, it argues that EHR-based suicidality labels should be interpreted as structured operationalizations of documented clinical evidence, not as neutral ground truth.

The significance of this work lies in its call for transparency in dataset construction. The authors assert that:

1. Downstream Effects: If training labels subsume heterogeneous linguistic framings, classifiers will likely exhibit systematic errors, such as over-flagging historical mentions as active ideation or failing to calibrate on cases of genuine uncertainty.
2. Design Transparency: Dataset documentation must explicitly state the operationalization choices (cohort seeding, aggregation levels, category collapsing) so that downstream users, including clinicians, can assess what the labels represent before acting on model outputs.
3. Modest Scope: The authors acknowledge that their findings are based on a single dataset from one medical center and that their linguistic analysis is conservative (lacking syntactic parsing or coreference resolution). However, they argue that the structural issues identified are inherent to any pipeline that relies on billing codes, hospital-stay aggregation, and intent resolution from clinical text.

The paper concludes that making these assumptions visible is a necessary first step toward developing clinical NLP systems that are transparent about what they predict.