Placebo Failures in Domain-Stratified Interrupted Time Series of Mean Lexical Diversity on the Post-2022 Web: A Methodological Caution on Common Crawl MATTR/MTLD Break- point Identification (with Secondary LLM-Associated Vocabulary Markers)

Every major communication technology leaves a trace on how we speak and write. When the printing press arrived, it standardized spelling and grammar. When the internet took hold, it accelerated the spread of slang and shortened sentences. Now, a new force has entered the conversation: artificial intelligence that writes for us. These large language models do not just transmit human language; they generate it in vast quantities, often favoring common, safe, and predictable word choices over rare or unusual ones. This raises a simple but profound question for the future of human expression: if machines write a significant portion of the text on the open web, will the language itself become simpler, less varied, and more uniform?

Researchers have long suspected that this might happen. In controlled tests, computer-generated text often uses fewer distinct words than human writing. Some studies suggest that as people use these tools to polish their own drafts, their writing might drift toward a shared, middle-ground style, losing its unique complexity. However, proving this on the real, messy internet is difficult. The web is a chaotic place where news cycles, advertising trends, and platform rules change constantly. To find a signal of artificial influence, scientists must distinguish between a genuine shift caused by AI and the natural noise of the internet. They need to know if the language is truly changing or if they are simply seeing a trick of the light.

A new study set out to answer this by treating the internet like a time machine. The researchers gathered a massive collection of English web pages from a public archive known as Common Crawl, covering the period from early 2020 to late 2024. This timeline is crucial because it includes the time before and after the release of a popular AI chatbot in November 2022, a moment often cited as the beginning of widespread AI adoption. The team divided the web into different neighborhoods to see if the effect varied by context. They looked at news sites, personal blogs, public discussion forums, and online commerce pages. For each of these groups, they measured two specific things: how many different words appeared in the text (lexical diversity) and how those words were arranged. They also looked for a specific set of words that have become popular with AI models, such as "delve," "tapestry," and "landscape," to see if these markers were rising in frequency.

The researchers used a method called an interrupted time series, which is essentially a way of tracking a trend line over time to see if something sudden happens at a specific date. They expected that if AI was reshaping the web, the trend lines for word variety would show a sharp drop right after November 2022. They also ran a rigorous check to ensure their method was sound. Before looking at the real data, they tested their system on fake dates in the past. If the system claimed to find a sudden change on a date when nothing happened, it meant the method was too sensitive and likely to produce false alarms. This step is vital because the internet is full of random spikes and dips that have nothing to do with artificial intelligence.

The results of this careful investigation were surprisingly quiet. When the researchers looked at the real data, they did not find a statistically robust, conclusive drop in word variety that many had predicted. In fact, when they applied their strict rules for what counts as a valid discovery, the remaining evidence was inconclusive; the data was too noisy to confirm a change, but also insufficient to prove that no change occurred. The only groups that showed a potential decline were small and statistically weak, meaning the data was too noisy to say for sure if a change had occurred. The study explicitly notes that no "informative powered null" remains, meaning it cannot definitively rule out a drop in language diversity, nor can it confirm one. Even the "placebo" tests, which acted as a control group, showed that the method could easily mistake random internet noise for a real trend. For example, in the news category, the system falsely detected a major shift at a fake date in 2021, proving that the method was not robust enough to trust the real results without further proof.

The study did find some signs that the specific vocabulary associated with AI was becoming slightly more common, particularly in commercial and news writing. However, this increase was small and did not survive the strict statistical tests used to rule out chance. It suggests that while AI might be leaving a faint stylistic fingerprint, it has not yet caused the language to become significantly simpler or more uniform in a way that can be clearly measured. The researchers concluded that the idea of a sudden, widespread decline in language complexity following the release of AI tools is not supported by the data from this period, but the lack of a clear signal is due to the limitations of the data rather than a definitive proof that nothing happened. The web's language, at least in the years immediately following the AI boom, appears to have retained its diversity, though the study cannot confirm this with absolute certainty.

This does not mean that artificial intelligence has no effect on language, only that the effect is harder to detect than a simple drop in word variety. The study highlights the difficulty of measuring such changes in a living, breathing ecosystem like the internet. The researchers noted that their data was limited by the number of documents they could process and the specific way the internet archives are collected. They emphasized that their findings are specific to the time period they studied and the types of websites they examined. They did not find a "smoking gun" that proves AI has homogenized human writing, but they also did not prove that it never will. The story of how AI changes language is still being written, and this research serves as a cautionary note: the changes may be subtle, complex, and easily mistaken for the normal noise of the digital world.