Authority Signals in AI Cited Health Sources: A Framework for Evaluating Source Credibility in ChatGPT Responses
This study introduces an Authority Signals Framework to evaluate source credibility in ChatGPT's health responses, finding that over 75% of the 615 cited sources in a sample of 100 questions originate from established institutional organizations rather than alternative health information providers.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
Imagine you are walking into a massive, chaotic library where the librarian is an incredibly smart robot named ChatGPT. You ask, "How do I fix my headache?" or "What causes diabetes?" The robot doesn't just guess; it runs to the shelves, grabs a stack of books, and reads them to answer you.
This paper is like a detective report investigating which books that robot actually grabs and why it chooses them.
The Problem: Trusting the Robot's Bookshelf
More and more people are asking robots for health advice. Some even use it to diagnose themselves. The problem is, we don't always know if the robot is picking up a book written by a world-famous doctor or a pamphlet written by someone with no medical training.
The authors of this study wanted to build a "Trust Checklist" to see if the robot is picking the most reliable sources. They called this checklist the Authority Signals Framework. Think of it like a four-part inspection for every book the robot pulls off the shelf:
- Who wrote it? (Author Credentials) – Is there a doctor's name on the cover, or is it anonymous?
- Who published it? (Institutional Affiliation) – Is it from a famous hospital like Mayo Clinic, a government agency, or a random blog?
- How was it vetted? (Quality Assurance) – Did experts check the facts? Are there references to other studies? Is the information up-to-date?
- How does the robot find it? (Digital Authority) – Does the book have special "digital tags" (like a barcode) that make it easy for the robot to spot?
The Investigation
The researchers acted like librarians. They took 100 common health questions (like "What are the symptoms of the flu?") and asked ChatGPT to answer them. They then looked at every single website the robot cited in its answers. In total, they analyzed 615 different sources.
What They Found: The "Big Three" Strategies
The study found that the robot generally plays it safe, but different types of websites use different tricks to get noticed.
1. The "Famous Names" Strategy (75% of the time)
Most of the time (over 75%), the robot grabs books from established institutions. These are the heavy hitters: Mayo Clinic, Cleveland Clinic, the NHS, Wikipedia, and government health sites.
- The Analogy: These are the celebrities of the health world. They don't need to shout or wear flashy costumes to get attention; their reputation does the work for them. Even if they don't have a doctor's name on every single page or the newest "digital tags," the robot trusts them because they are famous and established.
2. The "Hard Work" Strategy (Commercial Sites)
The remaining sources were mostly commercial health websites (like health blogs or product sites). These sites don't have the famous reputation of a hospital, so they have to work harder to get the robot's attention.
- The Analogy: These are the underdogs trying to get on stage. To win over the robot, they wear "digital costumes" (technical tags called schema markup), write very long, detailed articles, and put big stickers on their pages saying "Medically Reviewed." They are trying to prove they are just as good as the famous hospitals by checking every box on the Trust Checklist.
3. The "Freshness" Strategy (Professional Practice Sites)
Some sources were websites for individual doctors or small clinics.
- The Analogy: These sites rely on being the "fresh produce." They often have the most recent dates on their articles. The robot seems to like that they are up-to-date, even if they aren't the biggest names in the library.
The Surprising Details
- Anonymity is common: In about two-thirds of the sources, the robot couldn't even tell who wrote the article. There was no author name listed.
- References are rare: Only about 40% of the sources actually listed the studies or data they used to back up their claims.
- The "AI" factor: When they checked if the content was written by a human or another AI, about 23% of the analyzed content appeared to be AI-generated.
The Bottom Line
The study concludes that right now, ChatGPT mostly relies on fame and reputation (institutional authority) to pick its health sources. It prefers the "Big Names" over the rest.
However, the paper warns that as the internet evolves, smaller or commercial websites might start using "tricks" (like better digital tags or very recent dates) to trick the robot into picking them over the trusted hospitals. This study sets a "baseline"—a snapshot of the library as it is today—so we can watch and see if the robot starts picking different books in the future.
Important Note: The authors emphasize that this is a preprint (a draft) and hasn't been peer-reviewed yet. They also state clearly that this research should not be used to make actual medical decisions. It's just a map of how the robot is currently reading the library.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.