Event-Time Confounding Under Bursty Human Dynamics
This paper demonstrates that aligning digital behavior to user-initiated events creates an "episode-selection bias" where post-event activity surges reflect the natural continuation of ongoing tasks rather than causal effects, a confound that standard user fixed effects fail to address but which can be detected and corrected using the proposed diagnostic protocol and the "burstcheck" audit tool.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the digital world, we leave a trail of breadcrumbs every time we click, search, or ask a question. For years, researchers and companies have tried to read these trails to understand cause and effect. They want to know if a specific action, like asking an artificial intelligence for help, actually changes what a person does next. The standard way to find this out is to pick a moment in time—say, the second a user types a question into a chatbot—and then measure how much activity happens in the minutes that follow. If the user searches for more products or reads more articles after that moment, the assumption is that the chatbot caused the surge. It is a logical way to think, but it relies on a hidden assumption: that the moment the user chose to act was a neutral starting point, like the starting gun at a race.
However, human behavior does not work like a race with a single starting gun. Instead, our attention comes in waves. We often dive into a topic, spending twenty or thirty minutes jumping between different websites, search engines, and tools to solve a problem or satisfy a curiosity. These bursts of activity are intense and focused, followed by long periods of quiet. The problem arises when a researcher picks a moment inside one of these busy waves to study. If a person asks a question while they are already deep in a research session, the activity that follows is not necessarily a reaction to the question. It is simply the continuation of the wave they were already riding. The question did not start the wave; it just happened to be the moment the researcher looked down at their watch. Distinguishing between a cause and a coincidence in this environment is difficult because the person choosing the moment is the same person driving the activity.
This is the core puzzle tackled by a new study from Michael Iannelli and Alan Ai at Scrunch AI. They investigated whether the standard method of measuring digital behavior is actually measuring the event itself, or just the busy period the event happened to fall into. To solve this, they did not rely on theory alone; they built a test using real data from thousands of users who had agreed to share their browsing history, search queries, and interactions with AI assistants. The researchers wanted to see if they could trick their own measurement tools into finding a "cause" where none existed.
The team designed a clever experiment using what they call "known-null" timestamps. They took the exact same users and the exact same patterns of activity, but they picked moments when the user was busy but had not asked an AI a question. They treated these random moments as if they were the event. Since these moments were just random points in time with no actual intervention, they could not possibly cause any change in behavior. If the standard measurement tools were working correctly, they should have seen zero effect after these fake moments. Instead, the tools reported a massive surge in activity. When the researchers looked at the data, they found that these random, fake moments produced an increase in search activity that was nearly as large as the increase seen after real AI responses. Specifically, the fake moments generated about 3.4 times the usual amount of search activity, while the real AI moments generated about 4.3 times the usual amount.
This result was a shock because it showed that the "effect" of the AI was largely an illusion created by timing. The researchers found that the activity did not jump up the moment the AI responded. Instead, the activity had been climbing for about eight minutes before the response, peaked, and then slowly declined. The AI response was just a timestamp taken in the middle of a task that was already in full swing. The user was already searching and browsing because they were in the middle of a "task episode," a period of high engagement. The AI response was a marker of that episode, not the spark that started it.
To prove this was a general problem and not just a quirk of AI, the researchers looked at other types of digital events. They examined moments when users clicked on shopping links, visited news sites, or opened coding documentation. In every case, the same pattern appeared. The activity on other parts of the web—like general browsing and searching—was already high before the user clicked the link. The event was always sitting inside a larger, rising wave of activity. The researchers even simulated a world where an event had absolutely no power to change behavior. In these computer simulations, they created a scenario where a user's activity would naturally surge and then fade, and they placed a "treatment" event randomly inside that surge. Even though the event did nothing, the standard analysis methods still reported a huge, positive effect. The methods failed because they were counting the natural rise and fall of the user's attention, mistaking it for a reaction to the event.
The study also tested whether common statistical tricks could fix the problem. Researchers often try to control for this by comparing a user to themselves at a different time, or by matching the user's recent history. The authors found that these methods did not work. Because the "busy state" is hidden and changes rapidly from minute to minute, looking at a user's history from an hour ago is like trying to predict a storm by looking at the weather from yesterday. The hidden state of high engagement is not captured by these older data points. The researchers showed that even when they tried to match users to moments with similar activity levels, the bias remained. The only way to truly separate the event from the episode is to compare entire periods of time—comparing a whole research session with an AI to a whole research session without one—rather than just looking at the minutes immediately after a click.
The implications of this finding are significant for how we understand digital life. It suggests that many reports claiming that a specific tool or ad drives a surge in activity might be misreading the data. The surge might have been happening anyway. The researchers do not say that AI or ads have no effect; they simply argue that the standard way of measuring them is flawed. If a person is already in the middle of a shopping trip, an AI assistant might help them find a product, but the assistant did not create the shopping trip. The study concludes that to get the right answer, we must stop treating a single click as the start of a story and start looking at the whole chapter. We need to compare similar chapters of a user's day, with and without the specific event, to see if the story actually changes. Until we do that, the "effects" we see in digital logs are likely just the echo of a wave that was already breaking.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.