🤖 AI

A robust association between LLM use and scientific productivity: Assessing stopping-time selection

This paper refutes claims that stopping-time selection bias invalidates findings of a positive link between LLM use and scientific productivity, demonstrating through multiple robust designs that the observed association remains significant and cannot be explained by the identified artifact.

Original authors: Keigo Kusumegi, Xinyu Yang, Paul Ginsparg, Mathijs de Vaan, Toby Stuart, Yian Yin

Published 2026-08-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Keigo Kusumegi, Xinyu Yang, Paul Ginsparg, Mathijs de Vaan, Toby Stuart, Yian Yin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great Paper-Counting Puzzle

Imagine the world of science as a massive, bustling library where researchers are constantly writing new books. For decades, the way we measured a researcher's success was simple: count how many books they wrote. But recently, a new tool arrived on the scene: Large Language Models (LLMs). Think of these as super-smart, digital co-pilots that can help writers draft text, brainstorm ideas, and polish sentences. The big question everyone is asking is: Do these digital co-pilots actually help scientists write more books, or are they just a fancy distraction?

To answer this, researchers had to play a tricky game of "spot the difference." They needed to figure out exactly when a scientist started using the AI and then count their output before and after that moment. However, there's a sneaky trap in this game called "stopping-time selection." Imagine you are trying to measure how much faster a runner gets after drinking a special energy drink. If you decide to start the stopwatch the exact moment the runner crosses the finish line of their first race, you've created a weird problem. The race before that moment might have been a slow, bad day, making the runner look like a superhero immediately after. In the world of science papers, if a researcher's first AI-assisted paper is flagged by a detector, the month before that flag might look unusually slow just by chance. This could trick us into thinking the AI caused a boom in productivity, when really, it was just a statistical fluke.

The Detective Work: Is the Boom Real or a Glitch?

This paper is a response to a group of critics (let's call them the "Skeptics") who pointed out this timing trap. The Skeptics argued that the original study's finding—that AI makes scientists more productive—might just be an illusion caused by that "bad month before the flag." They suggested that if you look at random, meaningless flags, you'd see the same fake "boom" pattern, proving the AI isn't actually doing anything special.

The authors of this paper, a team of data detectives, decided to put this claim to the test. They didn't just argue back; they ran a series of clever experiments to see if the "AI boom" could survive when they removed the timing trap.

First, they fixed the "Fake Flag" Test.
The Skeptics had used random flags to show the pattern, but the authors realized those random flags were too weak. It's like trying to test a new engine by comparing it to a bicycle; the comparison isn't fair. The authors recalibrated the test so the random flags fired at the exact same rate as the real AI detector. Even with this "fair fight," the real AI detector showed a much bigger boost in productivity than the random flags. The "fake boom" the Skeptics predicted was there, but it was tiny—like a small ripple—while the real AI effect was a massive wave.

Second, they tried four different ways to measure without the trap.
To be absolutely sure, the authors designed four new experiments where the "bad month before the flag" couldn't mess up the results:

The Time-Traveler Test: Instead of comparing month-to-month, they looked at a whole year before AI use and compared it to a whole year after. This skips the weird "first month" entirely. The result? Scientists who used AI still produced about 17.3% more work (at a lower confidence threshold) and 25.1% more work (at a stricter threshold) than before.
The "Future User" Control: They compared current AI users not just to people who never used AI, but also to people who would use AI later. This creates a fairer baseline. Even with this conservative setup, the AI users still showed a positive boost, ranging from 0.05 to 0.13 log-points higher than the control group.
The "Intensity" Check: Instead of picking a specific start date, they looked at how much AI a scientist used over a whole quarter. They found that the more AI a scientist used in the previous three months, the more they wrote in the current month. This pattern vanished when they ran the same test on data from before AI existed.
The "Top vs. Bottom" Race: They compared the most AI-heavy papers against the least AI-heavy papers, while keeping the total number of flagged papers the same. The AI-heavy papers consistently outperformed the random chance baseline, while the low-AI papers did worse.

The Verdict
The paper concludes that the "stopping-time" glitch the Skeptics identified is real, but it is too small to explain the massive jump in productivity. It's like finding a tiny scratch on a car that explains a dent, but the dent is actually caused by a much bigger crash. When the authors removed the timing trick, the positive link between AI and scientific output didn't disappear; it stayed strong.

However, the authors are careful not to claim they have "solved" the mystery of why this happens or that the AI is the sole cause. They admit that scientists who choose to use AI might already be different in other ways. But they have successfully proven that the "AI boom" isn't just a statistical illusion caused by bad timing. The association is real, the direction is positive, and the effect is robust enough to survive even the toughest scrutiny.

Technical Summary: Assessing Stopping-Time Selection in LLM-Productivity Studies

Problem Statement
This paper responds to a critique by Renault, Bergeaud, and Bosquet (RBB) regarding the authors' original findings on the association between Large Language Model (LLM) use and scientific productivity. RBB argued that the original study's methodology—dating LLM adoption to the first month an author's abstract is flagged by a detector—induces a "stopping-time selection" bias. Specifically, because the month prior to the first flag must, by definition, contain no flagged papers, it represents an artificially depressed baseline (discounted by a factor of $1-p$ , where $p$ is the flag rate). Comparing subsequent months against this depressed reference could generate a positive event-study path even in the absence of a causal productivity effect. The authors acknowledge this mechanism is mathematically plausible but argue it does not constitute proof of a null effect. The central question addressed is whether this artifact explains the magnitude of the observed association or if a genuine positive association persists once the artifact is accounted for.

Methodology
The authors employ a multi-pronged strategy to isolate the stopping-time artifact from the true signal, utilizing both recalibrated placebo tests and four alternative identification designs:

Recalibrated Random Placebo: The authors argue that RBB's original placebo tests were biased because they did not match the detector's realized flag rate. They construct a new random placebo that fires at the exact same rate as the empirical LLM detector. This creates a conservative benchmark where the "stopping-time" mechanical artifact is held constant, allowing for a direct comparison of the residual signal.
Alternative Design 1: Separated Periods (Before-and-After): To eliminate the depressed reference month entirely, the authors date adoption in one year (2023) and measure output in a subsequent, non-overlapping period (Jan–June 2024), comparing it to a pre-LLM baseline (Jan–June 2022). This design removes the immediate post-detection spike artifact.
Alternative Design 2: Conservative Control Group (Difference-in-Differences): The authors revise the control group in a DiD framework by combining "never-treated" authors with "not-yet-treated" authors. Since "not-yet-treated" authors are future adopters, they experience the same mechanical selection bias (depressed pre-adoption months) as the treated group. This symmetry allows the artifact to difference out, while the control group remains conservative because future adopters will eventually show productivity boosts, attenuating the estimate toward zero.
Alternative Design 3: Intensity-Based Specification: Instead of defining a discrete adoption date, the authors regress current productivity on the intensity of AI use in the previous quarter (measured by average detector score or share of papers flagged). This breaks the mechanical link between contemporaneous flagging and contemporaneous output.
Alternative Design 4: Rank-Based Measurement: Holding the flag rate ( $p$ ) fixed, the authors compare three rules: flagging the top- $p\%$ of papers by detector score, the bottom- $p\%$ , and a random Bernoulli draw. If the artifact were the sole driver, all three should yield identical results. Divergence indicates that the content of the flag (the $\alpha$ score) carries signal beyond the mechanical timing effect.

Key Results

Placebo Benchmark: When the random placebo is calibrated to the detector's realized flag rate, the empirical LLM detector's path remains significantly above the placebo benchmark throughout the post-treatment period. The average excess is 0.119 log points (at $\alpha > 0.1$ ) and 0.183 log points (at $\alpha > 0.5$ ). The placebo exhibits negative pre-treatment coefficients, unlike the real data, further suggesting the real signal is not merely an artifact.
Separated Periods: In the design comparing 2024 output to 2022 baselines, the authors observe a 17.3% to 25.1% increase in productivity for adopters. The same design applied to pre-ChatGPT placebo data yields null results (-0.9% to 0.0%), confirming the artifact does not survive separation of the dating and measurement periods.
Conservative Control: Using the combined control group, the authors find a persistent positive coefficient (pooled estimate: 0.068 log points) that is statistically significant. The placebo application yields insignificant results. Because this design attenuates the estimate, the positive result is interpreted as a lower bound.
Intensity and Rank: The intensity-based specification shows positive, highly significant coefficients for prior-period AI use, while placebo data shows negative, insignificant coefficients. Similarly, the rank-based test shows that top- $\alpha$ rules significantly outperform the random baseline, while bottom- $\alpha$ rules underperform, a pattern absent in placebo data.

Significance and Claims
The authors conclude that the stopping-time selection mechanism identified by RBB is real but bounded. It does not account for the pattern of productivity changes reported in their original work. The convergence of results across four distinct designs—each of which would fail to produce a positive signal if the effect were purely mechanical—provides robust evidence that a positive association between LLM use and scientific productivity exists.

Crucially, the authors maintain the modesty of their original claims:

They do not claim these estimates are causal. They explicitly state that endogeneity in technology adoption (who adopts and when) remains unresolved.
They do not claim to have identified a precise, universal magnitude for the effect. They argue the magnitude is local to the specific generation of models and time window studied.
Their primary contribution is establishing that the direction of the association (positive) survives the removal of the specific dating artifact, refuting the hypothesis that the original findings were entirely spurious.

The paper serves as a methodological defense, demonstrating that while event-study designs dating treatment from outcomes require caution, the specific artifact identified does not invalidate the observed correlation between LLM adoption and increased scientific output.

The Great Paper-Counting Puzzle

The Detective Work: Is the Boom Real or a Glitch?

Technical Summary: Assessing Stopping-Time Selection in LLM-Productivity Studies

More like this