Technical Summary: Generics in Science Communication

Problem Statement

Scientists frequently employ generics—unquantified present-tense statements referring to entire categories (e.g., "statins reduce cardiovascular events")—when communicating research findings. While these statements suggest broad applicability, corpus analyses indicate they are often used even when findings are based on small or unrepresentative samples. This creates a risk of overgeneralization, where audiences apply findings to populations for which they are not applicable, potentially undermining public trust and decision-making.

A critical gap exists in understanding how different stakeholders interpret these linguistic forms. It is unclear whether laypeople and scientific experts share the same interpretation of generics, or if experts possess "epistemic vigilance" that leads them to interpret these claims more narrowly than the public. Furthermore, with the increasing reliance on Large Language Models (LLMs) like ChatGPT to summarize scientific literature, it is unknown whether these models interpret generics similarly to humans. If LLMs systematically overgeneralize scientific findings compared to human experts, they risk disseminating misleading summaries at scale.

Methodology

The study employed a mixed-methods design comparing three distinct groups: laypeople (individuals with at most an undergraduate degree), scientific experts (psychologists and biomedical researchers, plus other scientists/experts), and two leading LLMs (ChatGPT-5 and DeepSeek-V3.1).

Experimental Design

• Stimuli: Participants were presented with 18 one-sentence research conclusions drawn from top psychology and biomedicine journals.
• Linguistic Frames: Each conclusion was presented in three variations:
  1. Bare Generic: Unquantified present tense (e.g., "Statins reduce risk...").
  2. Past Tense: Treated as a non-generic, specific claim (e.g., "Statins reduced risk...").
  3. Hedged: Qualified with epistemic uncertainty (e.g., "The study suggests..." or "might...").
• Measures: Participants rated each statement on three dimensions using 5-point scales:
  • Generalizability: How broadly the claim applies (from "only to people studied" to "all people").
  • Credibility: How trustworthy the claim is.
  • Impact: The likelihood of engaging further with the finding (reading, sharing, using).
• Procedure:
  • Human Participants: 432 participants (192 laypeople, 240 experts) completed the survey via Qualtrics.
  • LLM Participants: 100 responses were collected (50 per model) via web-based user interfaces to simulate typical public interactions. Each model was treated as a "pseudo-participant."
• Qualitative Component: A free-response question asked participants to select the most appropriate framing for a statistically significant finding and explain their reasoning.
• Analysis: Linear mixed models (LMMs) were used to analyze ratings, controlling for participant and conclusion content. Thematic analysis was applied to qualitative responses.

Key Results

1. Effects of Linguistic Framing

Across all groups, bare generics were perceived as significantly more generalizable and more impactful than past-tense or hedged versions. However, a counter-intuitive finding emerged regarding credibility: past-tense statements were rated as more credible than generics, while hedged statements received the lowest credibility ratings.

2. Group Differences (Laypeople vs. Experts vs. LLMs)

• Laypeople vs. Experts: Contrary to the hypothesis that laypeople would interpret claims more broadly, psychologists and biomedical researchers rated scientific conclusions as less generalizable, less credible, and less impactful than laypeople. This suggests that field-specific expertise fosters greater epistemic vigilance and caution regarding overgeneralization.
• LLMs vs. Humans: Both LLMs (ChatGPT-5 and DeepSeek-V3.1) rated scientific conclusions as significantly more generalizable, credible, and impactful than both laypeople and experts. This indicates a systematic tendency for these models to overappraise the strength and scope of scientific claims.
• Interaction Effects: Experts (specifically psychologists and biomedical researchers) showed stronger sensitivity to linguistic framing than laypeople. They downgraded generics significantly more than laypeople did when moving to past-tense or hedged frames. LLMs, conversely, showed an upward adjustment in generalizability ratings when moving from generics to past tense, diverging from human patterns.

3. Qualitative Preferences

• Human Preferences: 64% of human participants preferred the "study suggests" framing, valuing a balance between confidence and caution. They frequently criticized bare generics as "overconfident" and "overgeneralizing."
• LLM Preferences: ChatGPT-5 strongly favored bare generics (52%), citing "confidence" and "ease of interpretation" for non-experts. DeepSeek-V3.1 aligned more closely with human preferences, favoring "study suggests."
• Reasoning: Humans emphasized avoiding extremes and relativizing findings to the specific study. LLMs prioritized clarity and broad applicability, often at the expense of the caution humans deemed necessary.

Key Contributions

1. Empirical Evidence of Misalignment: The study provides the first direct comparison of how laypeople, domain experts, and LLMs interpret scientific generics. It reveals a "double misalignment": experts interpret generics more narrowly than the public, while LLMs interpret them even more broadly than the public.
2. Expertise Effects: The findings challenge the notion that all experts interpret language similarly. Field-specific expertise (psychology and biomedicine) drives a more cautious interpretation of generics, likely due to awareness of replication crises and methodological limitations, whereas experts in other fields did not differ significantly from laypeople.
3. LLM Behavior in Science Communication: The paper demonstrates that leading LLMs systematically overgeneralize scientific findings. This suggests that LLMs may not merely summarize text but actively amplify the perceived scope and credibility of claims, potentially diverging from the intent of the original authors.
4. Framing Nuance: The study highlights that not all qualifications function identically. While "past tense" enhanced credibility, "hedging" (e.g., "might") reduced both credibility and impact, suggesting that the type of qualification matters significantly for public understanding.

Significance and Implications

The authors argue that these interpretive misalignments pose significant risks for science communication.

• For Scientists: Experts may inadvertently encourage overgeneralization by using generics, assuming their audience shares their nuanced, cautious interpretation. The data suggests that even past-tense framing may not fully prevent laypeople from overgeneralizing.
• For LLMs: If LLMs are used to summarize research for the public, their tendency to rate generics as highly credible and broadly applicable could lead to the systematic dissemination of inflated scientific claims. This is particularly concerning given that LLMs lack the "epistemic vigilance" or domain-specific skepticism that human experts apply.
• For Policy and Education: The findings underscore the need for greater attention to language choices in both human-mediated and AI-mediated science communication. There is a need to align expert and public perceptions and to monitor how LLMs handle generic claims to prevent the normalization of overgeneralization.

The paper concludes that while generics are rhetorically efficient for scientists, their use requires careful calibration to ensure accurate transmission of scientific uncertainty to diverse audiences, including AI systems.