Technical Summary: UTS at PsyDefDetect

Problem Definition

The paper addresses the PsyDefDetect shared task at BioNLP 2026, which requires classifying target utterances in emotional support dialogues into nine levels of the Defense Mechanism Rating Scales (DMRS). The DMRS is a hierarchical clinical instrument ranging from Level 1 (Action-based defenses) to Level 7 (Highly adaptive coping).

The core challenge is twofold:

1. Contextual Ambiguity: Defense mechanisms are defined by psychological function rather than linguistic form. The same surface expression can indicate different levels depending on context (e.g., acknowledging difficulty without emotional response may indicate Level 6 Isolation of Affect rather than Level 7 Adaptive Coping).
2. Severe Class Imbalance: The dataset exhibits a "L7 Attractor" effect, where Level 7 (Highly Adaptive) comprises 51.9% of training data, while lower-level defense classes (L1–L5) collectively account for only 21.5%. This leads Large Language Models (LLMs) to over-predict the majority class, mistaking emotional engagement for adaptive coping.

Methodology

The proposed system, UTS at PsyDefDetect, employs a multi-phase deliberative council architecture augmented by a targeted override ensemble.

1. Multi-Phase Deliberative Council

Built on the Gemini 2.5 API, the system replaces standard voting with a structured evidence-evaluation process:

• Phase 1 (Proposal): Three specialist agents (Clinical Analyst, Mechanism Specialist, Pattern Analyst) independently generate candidate labels and alternatives.
• Phase 2 (Advocacy): Class-specific advocates evaluate the strength of evidence for each candidate label, rating them as STRONG, MODERATE, or WEAK. This phase utilizes retrieved few-shot exemplars.
• Phase 3 (Resolution): A resolution function adjudicates based on evidence quality rather than vote count. A unique STRONG rating wins immediately; ties are resolved via pairwise head-to-head comparison.

Clinical Knowledge Encoding:
A critical component is the encoding of the DMRS hierarchy as an affect-cognition integration spectrum. The system reasons about what is absent (missing affect, blocked cognition, denied reality) rather than just what is present. Five prompt-level rules were implemented:

1. Integration of 60+ DMRS-Q behavioral indicators.
2. Distinction between emotion and defense (e.g., "I feel sad" is not a defense without distortion).
3. Preference for lower-level (less mature) defenses in ambiguous cases.
4. A watchlist for high-confusion class pairs.
5. An L7 verification gate requiring a named adaptive mechanism.

2. Retrieval and Fine-Tuning

• Retrieval: TF-IDF with Maximal Marginal Relevance (MMR) is used for Phase 1 to ensure diversity, preventing the clustering of emotional content that semantic retrieval might cause. Dialogue-ID exclusion is strictly enforced to prevent retrieval leakage (same-dialogue exemplars).
• Fine-Tuning: Three Qwen3.5 models (9B, 35B-A3B MoE, and a specialized 9B variant) were fine-tuned using LoRA in 4-bit quantization.
  • Key Intervention: Training used completion-only loss (train_on_responses_only), computing loss only on the label token. This prevented the model from wasting gradient updates on dialogue auto-completion, boosting accuracy from 25–55% to 59–65%.
  • Self-Consistency: Inference involves multiple runs (temperature=0.3) to generate confidence scores.

3. Targeted Override Ensemble

To address the council's tendency to be "confidently wrong" on minority classes (specifically the L7 attractor), a targeted override system was developed:

• Type A Overrides: Corrects L7 over-predictions when a fine-tuned model shows ≥80% self-consistency for a minority class and is corroborated by a pairwise resolver.
• Type B Overrides: Corrects minority-to-minority confusions when ≥3 of 6 independent sources agree.
• Agentic Selection: The selection of overrides was managed by a structured multi-agent system comprising a Builder (scans candidates), a Critic (argues against proposals), and a Regression Guard (hard-rejects submissions failing structural checks). This formal role decomposition yielded a larger F1 gain in one iteration than eight prior ad-hoc attempts combined.

Key Results

The system achieved 2nd place among 64 entries (F1 0.406) on the shared task leaderboard.

• Clinical Rules Impact: The integration of clinical reasoning rules into the prompt provided the single largest gain, increasing F1 from 0.268 to 0.382 (+11.4 percentage points).
• Override Impact: The targeted ensemble of 16 overrides (7 Type A, 9 Type B) added +2.4pp to the F1 score, bringing the final result to 0.406.
• Precision: The system achieved the highest precision (0.461) among all top-5 teams, reflecting the conservative nature of the override strategy.
• Error Analysis:
  • L7 Attractor: 54% of stable errors were incorrect L7 predictions.
  • Confidently Wrong: For minority classes (L3, L4, L6, L8), 59–80% of stable predictions were incorrect, whereas unstable predictions were paradoxically more accurate.
  • Overfitting: Modifications that improved training-set F1 consistently degraded test performance, indicating prompt overfitting.

Significance and Claims

The paper posits that the primary insight for this task is that defense mechanisms are defined by what is absent. Encoding this "absence-based reasoning" into clinical rules proved more effective than architectural changes or model scaling.

The authors highlight that:

1. Formal Role Decomposition: Structuring the override selection process into distinct roles (Builder, Critic, Guard) was more effective than automation alone, preventing confirmation bias and improving evidence quality.
2. Retrieval Strategy: For few-shot selection under class imbalance, TF-IDF with MMR outperformed semantic retrieval by avoiding the clustering of emotional content, thereby preserving class diversity.
3. Training Intervention: The use of completion-only loss during fine-tuning was a critical, high-impact intervention that significantly improved model performance on this specific classification task.

The system is explicitly framed as a research tool for analyzing psychological constructs in dialogue, not a clinical diagnostic instrument. The work represents one of the first large-scale computational studies applying LLMs to the hierarchical classification of defense mechanisms in naturalistic therapeutic conversations.