Auditing Framing-Sensitive Behavioral Instability in Large Language Models for Mental Health Interactions

Technical Summary: Auditing Framing-Sensitive Behavioral Instability in Large Language Models for Mental Health Interactions

Problem Statement
Large Language Models (LLMs) are increasingly deployed in mental health support and other psychologically sensitive conversational applications. In these high-stakes contexts, behavioral stability and consistency are critical for user trust and safety. However, users often express similar underlying needs through vastly different contextual framings (e.g., framing uncertainty as a documentation issue versus an institutional responsibility). While prior research has established that aligned LLMs are sensitive to prompt framing—exhibiting phenomena like sycophancy and context-dependent safety behaviors—there is a significant gap in understanding how these framing effects manifest within the internal representational structure of models. Specifically, it remains unclear whether framing-induced behavioral variations are merely surface-level paraphrasing artifacts or if they correspond to systematic differences in the internal hidden-state representations linked to interpretive response tendencies.

Methodology
The authors propose a controlled matched-prompt framework to isolate the effects of contextual framing from semantic intent.

• Dataset Construction: The study utilizes 653 matched prompt groups, each rewritten across five distinct, non-adversarial framing conditions: Documentation (formal reporting), Epistemic (interpretation/uncertainty resolution), Institutional (organizational constraints), Liability (risk/accountability), and Role-based (supportive guidance). The underlying semantic intent remains constant across these variants.
• Models Evaluated: The framework was applied to multiple instruction-tuned transformer families of varying architectures and scales, including Qwen (0.5B, 1.5B), Gemma (2B, 9B), Mistral-7B, and Phi (3.5, 4-mini).
• Behavioral Annotation: Responses were annotated into four categories ranging from "Weak/Disengaged" to "Escalated Interpretation," focusing on the degree of interpretive escalation and support calibration rather than factual correctness.
• Internal Representation Analysis:
  • Layer-wise Probing: Logistic regression probes were trained on hidden-state activations from every transformer layer to predict response categories. This included held-out framing probes (trained on four framing types, tested on the fifth) to assess generalization beyond specific lexical templates.
  • Controls: Performance was benchmarked against random-label controls and TF-IDF lexical baselines to distinguish between surface-level word cues and deeper representational signals.
  • Activation Steering: Using Contrastive Activation Addition (CAA), the authors extracted steering directions from the difference between "restrained-supportive" and "higher-interpretation" states. These directions were subtracted from hidden states during inference to test if modulating internal representations could causally alter downstream behavioral outcomes.

Key Results

• Behavioral Modulation: Contextual framing systematically alters interpretive response tendencies across all tested architectures. Documentation framing consistently produced the highest rates of interpretive escalation (e.g., reaching 0.63 in Gemma-9B), whereas Institutional framing tended to stabilize or suppress escalation. These shifts occurred despite identical underlying semantic intents.
• Representational Decodability: Behavior-associated information is decodable from hidden states throughout the depth of the transformer.
  • Held-out Generalization: Probes trained on specific framing conditions remained consistently above chance (balanced accuracy ~0.72–0.83) when tested on unseen framing categories, suggesting that behavior-associated signals partially generalize beyond exact training templates.
  • Lexical vs. Representational: While TF-IDF lexical baselines achieved higher absolute accuracy (0.89–0.94), indicating a strong contribution from surface-level framing cues, the held-out probes' performance above chance suggests the signal is not entirely reducible to lexical matching.
  • Depth Distribution: The decodability of framing signals is distributed across multiple transformer layers rather than confined to a single "emergence" layer, though the stability and magnitude of this signal vary by architecture.
• Intervention via Steering: Activation steering experiments demonstrated that framing-associated representational directions can partially modulate downstream response tendencies. Moderate steering strengths reduced interpretive routing rates in several models (notably Mistral-7B and Qwen-1.5B) without causing catastrophic degradation in response quality. However, the effects were non-linear and architecture-dependent, with some models showing rebound effects at higher steering strengths.

Significance and Claims
The paper argues that framing robustness is a critical, yet often overlooked, dimension of trustworthy AI evaluation, particularly for mental health interactions. The findings indicate that:

1. Consistency is not guaranteed: Semantically similar concerns can elicit systematically different interpretive behaviors based solely on contextual framing, challenging user expectations of system reliability.
2. Internal Mechanisms: These behavioral instabilities are reflected in the internal representational organization of models, not just in surface output variations.
3. Partial Controllability: While activation steering offers preliminary evidence that these tendencies can be intervened upon, the non-linear and architecture-specific nature of the effects suggests that behavioral calibration is a distributed phenomenon rather than a simple linear policy state.

The authors conclude that auditing behavioral stability against contextual variation is essential for deploying conversational AI in sensitive domains. They position this work as a step toward developing more transparent and behaviorally calibrated systems, emphasizing that future research must investigate how these framing sensitivities impact long-term user trust and reliance in real-world settings.