It Matters How You Say It: Exploring Rhetorical Patterns for AI-Assisted Information Evaluation

Technical Summary: It Matters How You Say It: Exploring Rhetorical Patterns for AI-Assisted Information Evaluation

Problem Statement
The proliferation of misinformation necessitates robust real-time information evaluation by everyday users. While Large Language Models (LLMs) are increasingly used to support claim verification, current AI advisory systems predominantly rely on "directive rhetoric," delivering confident verdicts that users passively accept. This approach often fails when ground truth requires nuanced, up-to-date technical analysis outside the model's training data, leading to confident but partially incorrect or misleading confirmations. Prior research has focused on what AI systems communicate (e.g., explanation types, warning labels) but has largely neglected how they communicate. Specifically, the rhetorical patterns that structure AI responses and their capacity to induce user reflection, uncertainty, or independent reasoning remain under-examined.

Methodology
The authors conducted a within-subject user study with $n=98$ participants to investigate the effects of eight distinct rhetorical patterns on fact verification performance.

• Task: Participants evaluated 40 claims for truthfulness over one hour. For each claim, they provided an initial rating and confidence level, then had the option to request advice from one of eight rhetorical patterns before submitting a final evaluation.
• Patterns: The study operationalized seven rhetorical patterns drawn from linguistics and psychology, compared against an "Oracle" baseline (direct, confident delivery of gold-standard evidence):
  1. Oracle: Direct, confident delivery of critical information.
  2. Scaffold Explanation: Step-by-step logical deduction connecting evidence to a conclusion.
  3. Socratic Questioning: Probing questions about missing evidence or assumptions.
  4. Alternative Framing: Redirecting attention via plausible but tangential context (Red Herring).
  5. Interpretive Alternative: Challenging assumptions by offering competing inferences.
  6. Information Distortion: Omitting key information to surface reasoning gaps.
  7. Triggering Distrust: Introducing deliberate absurdities to prompt critical scrutiny.
  8. Intentional Misleading: Confusing users with plausible but incorrect framings.
• Analysis: The study measured accuracy, confidence calibration, time on task, answer revision rates, and user preferences. Data was analyzed using Generalized Estimating Equations (GEE) to account for within-subject correlations.

Key Contributions

1. Rhetorical Taxonomy: The paper defines and operationalizes eight rhetorical patterns for AI advisory responses, grounded in linguistic and psychological theory, specifically tailored for a fact-verification context.
2. Empirical Evidence: It provides preliminary empirical evidence linking specific rhetorical styles to user accuracy, confidence calibration, and the depth of reflection.
3. Accuracy-Preference Divergence: The study identifies a systematic divergence between user performance and user satisfaction, challenging the assumption that user preference is a reliable proxy for system effectiveness.
4. Design Implications: The authors derive implications for conversational AI, arguing for adaptive rhetorical design in high-stakes information environments rather than a one-size-fits-all directive approach.

Results

• Accuracy: Contrary to the hypothesis that adversarial patterns would underperform, Scaffold Explanation and the Oracle baseline yielded the highest accuracy gains. Surprisingly, adversarial conditions (Intentional Misleading, Triggering Distrust, Information Distortion) also improved accuracy modestly, suggesting that detecting unreliable advice may trigger "motivated scrutiny" that extends to the claim itself. Socratic Questioning and Interpretive Alternative did not yield significant accuracy improvements.
• Reflection and Time: Scaffold Explanation was uniquely associated with encouraging deeper reflection. While Socratic Questioning and Interpretive Alternative increased time on task (a proxy for cognitive effort), this additional time did not translate into commensurate accuracy gains, indicating a dissociation between time spent and decision quality.
• User Preference: Participants most preferred Alternative Framing, citing its conciseness and ease of use. Conversely, Interpretive Alternative and Intentional Misleading were least preferred due to perceived time costs and confusion. Notably, the Oracle (highest accuracy) ranked only in the middle of preferences, and the most preferred pattern (Alternative Framing) showed no significant accuracy gain.
• Confidence: Confidence increased across all patterns after receiving advice, regardless of whether accuracy improved, highlighting the risk of confidence inflation without judgment improvement.

Significance and Claims
The paper argues that the rhetorical style of AI communication is as critical as the informational content. The findings challenge the prevailing design assumption that direct, confident responses are universally optimal. Instead, the authors suggest that "deliberate friction" (e.g., through scaffolding or adversarial cues) can be a productive resource for inducing critical thinking and improving accuracy in fact verification.

However, the authors maintain a modest tone regarding their claims:

• They characterize the study as an "exploratory design probe" due to the small number of manually constructed claims per pattern.
• They acknowledge that the observed accuracy gains in adversarial conditions might be driven by response bias (skepticism toward false claims) rather than the rhetorical mechanism itself, as no manipulation checks were performed.
• They caution against deploying adversarial patterns (misleading/distortion) in production systems without explicit user consent, citing ethical concerns regarding manipulation and trust.
• The paper concludes that no single rhetorical style is universally optimal; rather, systems may need to dynamically adapt their style based on user behavior and task context, balancing the trade-off between user satisfaction (ease) and system effectiveness (accuracy/reflection).