Rhetorical Questions in LLM Representations: A Linear Probing Study

1. Problem Statement

While rhetorical questions (RQs)—questions asked to persuade or signal stance rather than seek information—are a fundamental part of human communication, it remains unclear how Large Language Models (LLMs) internally represent them. Existing computational work has largely treated RQ detection as a classification task focused on prediction accuracy. This paper shifts the focus to interpretability, asking:

• How are rhetorical signals encoded in the internal representation space of LLMs?
• Are these signals captured by a single, unified linear direction, or are they heterogeneous?
• To what extent do representations learned on one dataset transfer to another?

2. Methodology

The authors employ linear probing techniques to analyze the internal representations of decoder-only LLMs (specifically Qwen3-32B and Llama-3.3-70B-Instruct) across two distinct social media datasets:

• Datasets:
  • RQ (Zhuang & Riloff, 2020): Twitter-based questions with conversational context.
  • SRAQ (Ikumariegbe et al., 2025): Reddit-based questions with paragraph-level or full-turn context.
• Representations: The study compares last-token representations ($h_T$) against mean-pooled representations ($\bar{h}$) across various layers. They project these high-dimensional embeddings into a 64-dimensional PCA subspace to reduce noise and stabilize comparisons.
• Probing Techniques: Three linear probes are applied to distinguish rhetorical from informational questions:
  1. DiffMean: A training-free population-level direction ($w = \mu_+ - \mu_-$).
  2. Logistic Regression: A trained classifier minimizing cross-entropy loss.
  3. Linear SVM (Hinge Loss): A trained classifier optimizing the margin.
• Evaluation Metrics:
  • AUROC: Measures discriminative separability.
  • Rank Agreement: Spearman correlation between probe-induced rankings.
  • Overlap: Jaccard index of top/bottom-ranked instances.
  • Causal Steering: Modifying residual stream activations to observe changes in generated rhetorical behavior.

3. Key Contributions

1. Representation Stability: The study establishes that rhetorical signals emerge early in the network but are most stably captured by last-token representations in decoder-only models, outperforming mean-pooled representations which tend to dilute signals in context-rich inputs.
2. Divergence of Performance and Alignment: The authors reveal a critical disconnect: while rhetorical questions are linearly separable and transferable across datasets (AUROC ~0.7–0.8), probes trained on different datasets induce significantly different rankings on the same target corpus. Top-ranked instance overlap is often below 0.2.
3. Heterogeneity of Rhetorical Meaning: Qualitative analysis demonstrates that rhetorical intent is not encoded along a single linear axis. Instead, it is represented by multiple, non-collinear linear directions that capture distinct phenomena:
   • Some directions capture discourse-level stance (extended arguments).
   • Others capture localized, syntax-driven interrogative acts (short, punchy questions).

4. Key Results

A. Representation Choices

• Last-Token vs. Mean Pooling: Last-token representations consistently achieve higher and more stable AUROC (peaking around 0.8–0.9) compared to mean pooling. Mean pooling performs well in early layers but degrades in deeper layers, suggesting that aggregating all tokens dilutes specific rhetorical cues in long contexts.

B. Linear Separability and Probe Alignment

• Within-Dataset: Trained probes (Logistic/SVM) outperform the training-free DiffMean probe. However, even trained probes show only moderate alignment with DiffMean.
• Cross-Dataset Transfer:
  • Discriminative Power: Probes transfer reasonably well, maintaining AUROC between 0.7 and 0.8.
  • Representational Misalignment: Despite similar AUROC, the ranking agreement between a probe trained on Dataset A and one trained on Dataset B is low (Spearman $\rho \approx 0.5$).
  • Top-Instance Divergence: The overlap of the top 20% of ranked instances between transferred and in-domain probes is extremely low (often < 0.2). This indicates that high transferability in accuracy does not imply a shared underlying representation.

C. Qualitative Insights

• SRAQ-derived probes prioritize long, discourse-heavy examples where rhetorical questions serve as structural scaffolding for arguments.
• RQ-derived probes prioritize short, syntax-driven examples where the rhetorical force is localized to a single sentence or phrase.
• Length Analysis: Top-ranked examples by SRAQ probes are significantly longer than those by RQ probes, confirming that different datasets emphasize different rhetorical scales.

D. Causal Steering

• Steering the model's internal representations along the identified DiffMean directions successfully induces more rhetorical behavior in generated follow-up questions, validating that these linear directions causally influence rhetorical intent.

5. Significance and Implications

• Challenging the "Single Direction" Hypothesis: The paper refutes the assumption that a property like "rhetorical intent" is encoded in a single, shared linear direction within LLMs. Instead, it suggests a heterogeneous, context-sensitive encoding where multiple distinct linear directions can achieve similar discriminative performance.
• Limitations of Transferability: High cross-dataset transferability in classification tasks does not guarantee that models are learning the same underlying concept. Researchers must look beyond AUROC to ranking alignment and qualitative analysis to understand representational structure.
• Methodological Guidance: For probing rhetorical or complex pragmatic phenomena, last-token representations are superior to mean pooling in decoder-only models, and researchers should be cautious about assuming a unified representation space for heterogeneous linguistic phenomena.

In conclusion, the study provides a nuanced view of how LLMs handle non-literal language, demonstrating that rhetorical questions are encoded via a complex, multi-directional manifold rather than a simple linear feature.