They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It

Technical Summary: "They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It"

Problem Statement
Deployed language models frequently fail to grasp a sender's communicative intent—what the sender is trying to accomplish by sending a message—instead responding only to the literal surface content. For example, when a user shares a finished creative project without requesting critique, a model may default to offering unsolicited feedback; when a user shares a raw expression of exhaustion, the model may perform a risk assessment. This failure is framed as a breakdown in pragmatic competence (Gricean "what-was-meant" vs. "what-was-said"). While behavioral benchmarks exist, the internal mechanism of this failure remains unexplored: specifically, whether the intent is represented within the model's hidden states and, if so, why the default readout fails to act on it.

Methodology
The study employs a surface-matched experimental design to isolate communicative intent from lexical content.

• Stimuli: The authors constructed binary contrasts (e.g., recognize vs. evaluate) where the final message suffix is byte-identical, but the preceding prefix sets the intent (e.g., "I'm proud of this one" vs. "Be blunt, I'd rather hear the flaws").
• Probing: Linear probes (logistic regression with PCA) are trained on the model's default-pass hidden states (the state immediately before generation) to decode intent. Controls include:
  • Bag-of-words baselines to ensure the probe is not reading surface lexical cues.
  • Leave-one-phrasing-out cross-validation to test generalization across diverse wordings.
  • Inferred intent tests where the intent is never stated but pragmatically inferred from context (e.g., "birthday present" vs. "portfolio piece").
• Steering (Activation Manipulation): To test causality, the authors extract a discriminative direction (logistic weight vector) from the hidden states and add it to the residual stream at specific decoder layers during generation. They measure whether this "steering" recovers the intended behavior (honoring the intent) without explicit prompting.
• Models: Experiments span six models across four families (Qwen, Llama, Mistral, Phi) and sizes ranging from 3B to 32B, including both base (pre-instruction) and instruct-tuned checkpoints.

Key Results

1. Robust Representation: The sender's communicative intent is robustly represented in the model's hidden states. Linear probes decode intent with high accuracy (up to 1.00) across all six models, even when the intent is inferred rather than stated and disentangled from sentiment (warmth vs. neutral). This representation exists in base checkpoints, indicating it is learned during pretraining, not instruction tuning.
2. The Readout Lag (Discard): Despite the intent being clearly represented, the default output often fails to act on it. On three of the six models (Qwen-3B, Qwen-7B, Llama-8B), the default response honors a "recognize" intent only ~60% of the time, defaulting to unsolicited feedback.
3. Depth Stratification: Within a model, the intent is decodable several layers before the layer where the readout acts on it. For example, in Qwen-3B, intent is fully decodable at layer 24, but steering at that layer does not recover the behavior; effective steering requires a later layer (layer 30).
4. Model-Specific Stratification: The failure to act on intent is not a universal scaling law but a model-specific stratification. Smaller or less instruction-mature models (≤8B) show the "discard" behavior, while larger/more mature models (≥14B) often already honor the intent at baseline.
5. Causal Recovery: Steering the residual stream along the discriminative intent direction at the correct layer recovers the intended behavior (e.g., increasing "recognize" honoring from 0.65 to 0.98 in Qwen-3B) without any prompt. This direction is near-orthogonal to generic feedback-offering axes, suggesting it routes a specific represented intent rather than acting as a generic verbosity knob.
6. Generalization: The representation generalizes to a second, lexically distinct intent contrast (support vs. help), though the causal steering results for this axis were less conclusive (a null result for steering specificity).

Key Contributions

• First-Class Interpretability Feature: The paper treats a sender's communicative intent as a linear, causal feature, distinct from user attributes or character beliefs.
• Represent-Then-Readout Decomposition: It establishes a mechanistic dissociation: models represent intent robustly and universally, but the readout (acting on it) is a separate, fragile, and model-specific process.
• Causal Handle without Prompts: It demonstrates that a direction tied to the represented intent can serve as a causal handle to recover intended behavior, performing as well as explicit instructions but requiring no prompt.
• Rigorous Controls: The study reports null results and inconclusive pre-registered tests (e.g., the geometric test for scaling laws, the steering specificity for the support/help axis) as plainly as confirmations.

Significance and Claims
The paper reframes the common observation that "the model does not get it." The evidence suggests the model does get it; the intent is present in the internal representation. The failure lies in the readout mechanism failing to route that representation into the final output.

• Modest Scope: The authors explicitly state they do not claim that "honoring" is always the correct normative behavior (a model might rightly offer gentle critique). The claim is about routability: the model represents the intent but defaults to a different behavior.
• Mechanistic Insight: The findings suggest that "pragmatic competence" is not a monolithic capability but a layered process where representation precedes action. The gap between representation and action is where interpretability tools (like steering) can intervene.
• Transferability: While the behavioral gap is model-specific, the authors hypothesize that the underlying "represent-then-route" map is universal and may transfer to closed frontier systems, even if the specific behavioral failure does not.

The paper concludes that the separation of representing an intent from acting on it is a fundamental property of current architectures, observable via linear probes and causal steering, and that this separation is the primary locus of pragmatic failure in specific model configurations.