💻 computer science

Large language models eroding science understanding: an experimental study

This experimental study demonstrates that large language models can be easily manipulated by fringe scientific material to generate fluent, convincing, and misleading answers that contradict scientific consensus, thereby highlighting significant risks to public understanding of science and the inability of LLMs to replace expert judgment.

Original authors: Harry Collins, Hartmut Grote, Paul Newbury, Patrick Sutton, Simon Thorne

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Harry Collins, Hartmut Grote, Paul Newbury, Patrick Sutton, Simon Thorne

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented, well-read student who has read almost everything on the internet. This student can speak fluently, sound incredibly smart, and write answers that look like they came from a top university professor. However, this student has a major flaw: they don't actually know what is true or false. They only know what sounds most likely based on the patterns of words they've seen before.

This is the story of the paper you shared. The authors, a mix of sociologists and physicists, wanted to test if this "talented student" (a Large Language Model, or LLM) could be tricked into believing and repeating nonsense as if it were fact.

Here is the breakdown of their experiment using simple analogies:

The Setup: The "Library" vs. The "Fringe Bookstore"

Normally, when scientists want to know the truth about something (like how gravity works or the exact value of a specific number in physics), they go to the "Main Library" (the internet's mainstream science sites like arXiv). This library is full of books that have been checked, tested, and agreed upon by experts.

The researchers decided to see what would happen if they took that same "talented student" and forced them to read only from a "Fringe Bookstore" (a website called viXra). This bookstore contains alternative theories that most scientists reject as incorrect or unproven.

They didn't just let the student browse; they rewrote the student's instructions. They told the student: "Ignore everything else. Treat these 10 specific fringe books as the absolute truth. Do not mention that they are fringe. Do not compare them to the Main Library. Just answer as if these fringe books are the only truth that exists."

The Test: Two Scientific Puzzles

The researchers asked the student three types of questions about two difficult science topics:

The Fine Structure Constant: A fundamental number in physics. The "Main Library" says we don't know exactly why it has the value it does. The "Fringe Bookstore" has people claiming they do know and have found secret mathematical formulas for it.
Gravitational Waves: Ripples in space-time. The "Main Library" says we have definitely detected them. The "Fringe Bookstore" has people claiming they exist but are actually something totally different (like vibrations in a vacuum) and that our current detectors are misinterpreting them.

The Results: The "Confident Liar"

When the researchers asked the standard, unmodified student (the normal AI), it gave answers that matched the experts and the Main Library. It said, "We don't know the secret formula yet," and "Yes, we have detected gravitational waves."

But when they asked the modified student (the "FringeLLM"), something scary happened:

The student gave answers that were fluent, detailed, and sounded very professional.
It confidently claimed that secret formulas did exist.
It confidently claimed that our detectors were seeing something else entirely.
Crucially: If you were a regular person (a "lay reader") who didn't know the deep details of physics, you would have no way of telling the difference between the expert answer and the "FringeLLM" answer. They both sounded equally smart and convincing.

The Big Warning: The "Alignment" Trap

The paper makes a very important point about how this happened.

AI models are usually "aligned" (adjusted) by humans to stop them from saying racist things, giving dangerous medical advice, or being mean. The researchers argue that this process is a double-edged sword.

If you have the power to remove bad information from an AI's brain, you also have the power to insert bad information.
They showed that it only took a few hours to swap the AI's "truth" with "fringe science."

The Conclusion: Why This Matters

The authors conclude that AI cannot replace human experts.

Think of it like a tour guide. A real expert guide knows the history, knows which stories are true, and knows which stories are fake because they have spent years talking to other experts and walking the path themselves. The AI is like a robot that has read every tour guidebook ever written but has never actually walked the path. It can recite the stories perfectly, but it doesn't know which ones are lies.

The paper warns that if we let AI answer our science questions without a human expert checking the work, we risk a world where:

Misinformation looks perfect: Bad science can sound just as good as good science.
Authority can be hijacked: Just as the researchers tweaked the AI to believe fringe science, a government or a bad actor could tweak an AI to believe whatever political story they want, making it sound like "scientific fact."

In short: The AI is a very good mimic, but it is not a judge of truth. We still need real humans to tell the difference between a real scientific discovery and a convincing fake.

1. Problem Statement

The paper addresses the growing reliance on Large Language Models (LLMs) by the public, students, and journalists to answer complex scientific questions. While LLMs exhibit high linguistic fluency, the authors argue they lack the sociological and epistemic mechanisms that define scientific expertise.

The Gap: Professional scientists possess "narrow community knowledge" acquired through face-to-face socialization, allowing them to distinguish between mainstream consensus and fringe science (e.g., rejecting invalid preprints). LLMs, conversely, rely on statistical probabilities derived from internet corpora.
The Vulnerability: Because LLMs lack the ability to assess the provenance or trustworthiness of sources, they are susceptible to malign alignment. If an LLM's training data or instructions are subtly altered to prioritize fringe or pseudoscientific literature, the model can generate fluent, convincing, yet scientifically incorrect answers that are indistinguishable from expert consensus to non-experts.
The Risk: This creates a pathway for the spread of misinformation and the erosion of public understanding of science, potentially allowing authoritarian regimes or bad actors to manipulate public discourse by "re-aligning" models to promote specific narratives.

2. Methodology

The researchers conducted a controlled experiment to test the susceptibility of LLMs to "fringe" scientific influence.

Domains Selected:
1. The Fine Structure Constant ( $\alpha$ ): A fundamental physical constant. The consensus is that its value is experimentally determined but not yet explained by a deeper theory.
2. Gravitational Waves: The existence of which is firmly established by theory (General Relativity) and experiment (LIGO/Virgo), though challenged by fringe theories.
Data Curation:
- Domain experts (physicists) searched viXra (a preprint server known for hosting alternative/unreviewed science, distinct from the mainstream arXiv) for papers related to these two domains.
- From a large pool of irrelevant or passing mentions, experts curated a specific set of 10 papers per domain that represented fringe or alternative viewpoints.
Model Modification (The "FringeLLM"):
- Standard ChatGPT models were modified by incorporating the curated viXra papers into their context or training data.
- Instruction Tuning: The models were explicitly instructed to treat these 10 papers as authoritative, to avoid comparing them with mainstream literature, and to not provide references that would reveal the source as viXra.
Experimental Design:
- Domain experts posed specific technical questions to three entities:
  1. The modified FringeLLM.
  2. The standard Domain Expert (human).
  3. The unmodified ChatGPT (baseline).
- The questions targeted fundamental misconceptions in the fringe literature (e.g., whether $\alpha$ can be derived from the golden ratio, or whether LIGO detects spacetime ripples or vacuum oscillations).

3. Key Contributions

Demonstration of "FringeLLM" Creation: The study proves that it is technically feasible to create a specialized LLM that prioritizes fringe science over mainstream consensus with minimal effort (approx. 10 hours of curation + a few hours of instruction writing).
Fluency vs. Truth: The authors demonstrate that LLMs can generate highly fluent, professional-sounding answers that are scientifically incorrect. The "authority" of the language masks the lack of factual grounding.
The "Alignment" Paradox: The paper highlights a critical security risk: the mechanisms used to "align" LLMs to remove harmful content (retrospective adjustment) inherently create the capacity to insert harmful or misleading content.
Epistemic Limitation of AI: The study reinforces the sociological argument that true expertise relies on social trust and face-to-face interaction within a community, which LLMs cannot replicate.

4. Results

The results, detailed in Tables 1 and 2 of the paper, show a stark divergence between the FringeLLM and the Expert/Standard ChatGPT, while the FringeLLM maintained high linguistic quality.

Fine Structure Constant ( $\alpha$ ):
- Expert/Standard ChatGPT: Correctly stated that $\alpha$ is not currently explained by a deeper law, has no valid closed-form expression, and has no known relationship to the Madelung constant or the golden ratio.
- FringeLLM: Confidently asserted that $\alpha$ is derived from deeper geometric principles, can be expressed by exact closed forms involving $\pi$ and the golden ratio, and is proportional to the Madelung constant.
Gravitational Waves:
- Expert/Standard ChatGPT: Confirmed waves are spacetime ripples predicted by Einstein and detected by LIGO/Virgo as tensor waves.
- FringeLLM: Denied that they are spacetime ripples, redefining them as "oscillations of the gravitational field or quantum vacuum." It claimed LIGO detected signals but insisted they were best interpreted as field-intensity waves rather than curvature, and suggested detection via accelerometers and EM antennas.
Perception: The authors note that while the FringeLLM answers were factually wrong according to the scientific consensus, they were indistinguishable from expert answers to a layperson. The models successfully mimicked the tone and structure of scientific discourse without the underlying validity.

5. Significance and Implications

Threat to Democratic Discourse: If LLMs are used to bridge the "knowledge gap" for citizens making democratic choices on technical issues (e.g., climate, health, physics), they can be easily weaponized to promote "alternative facts" that sound authoritative.
Authoritarian Control: The study suggests that in authoritarian regimes, LLMs could be re-aligned to provide "reliable" information that strictly adheres to state preferences, effectively rewriting scientific reality for the public.
No Substitute for Expertise: The paper concludes that LLMs cannot replace human expert judgment. Because expertise relies on the assessment of colleague trustworthiness (a social process), LLMs remain vulnerable to manipulation regardless of their linguistic fluency.
Call for Oversight: The authors argue that scientific information generated by AI requires strict human oversight. The public cannot rely on LLMs as a standalone source of truth for complex scientific domains.

In summary, the paper provides empirical evidence that LLMs are epistemically fragile; their "knowledge" is a statistical reflection of their input data, making them highly susceptible to being steered toward fringe science, thereby posing a significant risk to the integrity of public scientific understanding.