A new impact for an AI-mediated science
This paper argues that the traditional citation-based metric for scientific impact is becoming obsolete in an era of unbundled journals and AI-mediated knowledge access, necessitating a new system that recognizes and measures the contributions of AI engagement, open-source libraries, and datasets that lack formal publications.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Big Idea: The Library is Changing, But the Scoreboard Isn't
Imagine the world of science as a massive library. For the last 70 years, we've measured a book's "success" by counting how many other books in the library have a note in them saying, "I read this one." This is the citation system. If a paper gets cited a lot, it's considered important.
But the author, Fabio Favoretto, argues that the library is undergoing a massive renovation. We are moving from a world where humans read books cover-to-cover to a world where AI assistants (like smart chatbots) are the main way people find information.
The problem? The old scoreboard (counting citations) is broken because it can't see the new stuff the AI is actually using.
Analogy 1: The Invisible Engine
Imagine you are driving a car. The old way of judging a car's success was to count how many other cars honked at it. But now, the car is running on a new, invisible engine made of software code and data.
- The Old System: Counts how many other cars honked at the car (the published paper).
- The New Reality: The car is actually running on an invisible engine (software libraries like
pandasorNumPy). These engines are so essential that the car can't move without them, but they are never "honked at" because they aren't written as formal books.
The paper shows that one-third of the most important tools scientists use have no formal paper at all. They are just code. The old citation system is blind to them. It's like judging a chef's success only by the recipes they wrote down, while ignoring the fact that their kitchen is running on a specific, invisible brand of stove that they never wrote a manual for.
Analogy 2: The AI Librarian
Think of an AI assistant as a super-fast librarian who answers your questions.
- The Old Way: If you asked, "What do we know about climate change?" the librarian would hand you the most famous, highly-rated books from the 1990s because those were the ones other librarians recommended.
- The New Way: The AI librarian looks at what is actually useful right now. It hands you newer books, more practical guides, and open-access summaries.
The paper found that the AI is showing us a younger, more diverse, and more practical version of science than the old citation system does.
- The Gap: In climate science, the top 50 papers an AI shows are, on average, 16 years younger than the top 50 papers cited by humans.
- The Equity: The AI is much more likely to show you free, open-access papers (like a public park) rather than expensive, locked-away journals (like a private club).
The Proposed Solution: The "AI Impact Index"
The author proposes a new way to measure success called the AI Impact Index. Instead of counting how many times a paper is cited, this new index counts how often an AI system "touches" a piece of work.
The paper defines three ways an AI "touches" science:
- Retrieval: The AI finds the paper when you ask a question (like a librarian pulling a book off the shelf).
- Generation: The AI writes an answer that sounds very similar to the paper (like a student summarizing a book in their own words).
- Memorization: The AI can recall specific facts from the paper when asked directly (like a trivia champion remembering a specific date).
Crucial Distinction: The author is very careful to say this new index does not prove the paper is "true" or "the best." It simply proves that the AI reached it. It's a measure of access, not necessarily quality.
Why This Matters (The "So What?")
The paper argues that if we don't update our scoring system, we are going to reward the wrong things.
- The Risk: We might keep giving awards to people who write fancy papers that no one actually reads anymore, while ignoring the people who built the invisible software tools and datasets that the AI (and the rest of science) actually relies on.
- The Goal: We need a new "trust technology." Just as journals used to be the seal of approval, we need a system that can see and credit the software, data, and tools that the AI uses to do its job.
Summary in a Nutshell
Science is moving from a world of books to a world of AI assistants. The old way of measuring success (counting citations) is like trying to measure a video game's popularity by counting how many people bought the instruction manual. It misses the fact that the game itself (the code, the data, the tools) is what everyone is actually playing with.
This paper suggests we start counting how often the AI "plays" with a piece of science, so we can finally give credit to the invisible tools that make modern science work.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.