Technical Summary: AnchorSIPS

Problem Statement

Progress in Artificial Intelligence for psychosis-risk assessment is currently constrained by a data-access bottleneck. Authentic clinical interviews, particularly structured assessments like the Mini-SIPS, are difficult to share due to privacy, governance, and consent constraints inherent in high-sensitivity psychiatric care. Existing synthetic clinical dialogue resources largely focus on note generation, broad psychiatric diagnosis, or counseling support, rather than the specific, structured workflow required for psychosis-risk measurement.

Furthermore, current evaluation methods often rely on final diagnostic labels, which can overstate a model's true interview competence. In psychosis-risk interviews, clinically relevant information often emerges gradually through partial disclosure, where patients may be guarded, vague, or inconsistent. Strong Large Language Models (LLMs) frequently commit to early answers and fail to recover from incorrect conversational turns or fail to cite the specific transcript evidence supporting their conclusions. There is a lack of resources that evaluate whether models can recover intermediate clinical states and ground their decisions in specific transcript turns.

Methodology

The authors present AnchorSIPS, a synthetic dataset and evaluation resource comprising 10,000 structured psychosis-risk interviews. The dataset is designed to mirror the observable workflow of the Mini-SIPS interview, capturing history, 24 symptom queries, follow-up evidence for endorsed items, symptom-class decisions (delusion-like, hallucination-like, disorganized communication), frank-psychosis exclusion, and a final Attenuated Psychosis Syndrome (APS) diagnosis.

Plan-then-Realize Generation Pipeline

To ensure structural fidelity and avoid the inter-turn inconsistencies typical of end-to-end LLM dialogue generation, AnchorSIPS employs a plan-then-realize pipeline:

1. Hidden Case Sheet (SourceSpec): A deterministic process defines the patient's latent clinical state, demographics, symptom patterns, and all workflow labels (endorsements, follow-up details, class decisions, and final diagnosis) before any text is generated.
2. Deterministic Planning: A planner fixes the interview structure, including the sequence of 24 symptom queries, follow-up policies based on the case state, and interviewer turn metadata. This ensures the interview backbone is consistent and covers all necessary steps.
3. Surface Realization: An LLM (specifically GPT-OSS-120B in the release run) is restricted to generating only the wording of patient utterances based on the planned target meanings. The LLM has no authority over interview structure, label assignment, or decision logic.
4. Validation and Bounded Repair: A deterministic validator checks the generated transcript for structural consistency, denial polarity, and truncation. Only flagged patient turns undergo bounded repair; interviewer turns and all released labels remain fixed.

Dataset Composition

The dataset includes 10,000 interviews with a clinical output mix of 63.8% no-APS/no-psychosis, 21.1% APS, and 15.1% psychosis. Demographics and context variables are sampled from predefined distributions to support stratification and stress testing. Difficulty characteristics (e.g., guardedness, vagueness, delayed revelation) are embedded within the interviews to simulate partial disclosure scenarios.

Expert Audit

Three experts with over 12 years of combined experience in youth mental health and structured interviewing conducted an audit of the label-evidence alignment. They evaluated whether the cited transcript snippets supported six criterion-level dimensions (symptom presence, frequency, worsening, distress, functional influence, and alternative explanations) across 15 interview-by-symptom-class decision records.

Key Contributions

1. Workflow-Aligned Synthetic Resource: AnchorSIPS is the first synthetic psychosis-risk resource to release field-level transcript evidence alongside workflow labels. It treats the structured interview workflow (query endorsements, follow-up fields, class-level decisions, and diagnostic exclusions) as the prediction target, with each target linked to specific supporting transcript turns.
2. Auditable Generation Pipeline: By separating content planning from surface realization, the authors ensure that all released labels are derived deterministically from a hidden case sheet, independent of the LLM's generative capabilities. This prevents "label drift" and ensures reproducibility.
3. Baseline Evaluation of Grounded Measurement: The paper provides a baseline evaluation across seven LLMs, quantifying the gap between coarse decision-making and evidence-supported measurement.

Results

The evaluation of seven LLM baselines on a 200-interview workflow shard reveals a consistent gap between coarse recognition and grounded measurement:

• Coarse Decisions: Models performed well on query endorsement (F1: 0.949–1.000) and frank-psychosis prediction (F1 up to 0.930), tasks that rely on detecting salient symptom mentions.
• Grounded Measurement: Performance dropped significantly for tasks requiring detailed extraction and citation.
  • Evidence Linking: The hardest task, with F1 scores topping out at 0.290 (DeepSeek V4 Flash), indicating models struggle to cite the specific transcript turns supporting a decision.
  • Follow-up Extraction: Token-level F1 ranged from 0.164 to 0.248, showing difficulty in recovering normalized field values (e.g., frequency, distress) from the text.
  • Grounded Correctness: Even for the strongest models, the rate of predictions that were both correct and supported by cited evidence remained low (e.g., 0.178 for GPT-5.5).

The results suggest that while models can predict final labels, they often fail to recover the intermediate clinical reasoning and evidence required to justify those labels, particularly in multi-turn settings with partial disclosure.

Significance and Claims

The paper claims that AnchorSIPS addresses a critical gap in high-stakes NLP evaluation by shifting the focus from final-answer plausibility to evidence-supported symptom measurement. Its significance lies in:

• Operationalizing Partial Disclosure: It explicitly measures a model's ability to handle guarded or vague patients where information emerges gradually, a scenario where models often fail to recover from early errors.
• Revealing Failure Modes: The benchmark exposes that high accuracy on final labels can mask a lack of true interview competence, specifically in evidence extraction, grounding, and intermediate workflow consistency.
• Research Utility: The resource is intended for methodological research on transcript-grounded extraction, intermediate clinical reasoning, and uncertainty handling under partial disclosure.

The authors explicitly state that AnchorSIPS is not a clinical simulator or a protocol-equivalent replacement for real instruments like Mini-SIPS. It is a synthetic benchmark designed to stress-test reasoning pipelines. They caution that good performance on AnchorSIPS does not imply deployment readiness for clinical diagnosis, and any system developed using this resource requires validation on real-world clinical data before clinical use. The dataset is released under CC BY-NC 4.0 with strict prohibitions against diagnostic, triage, or patient-facing use.