💬 NLP

Disentangling Answer Engine Optimization from Platform Growth: A Log-Based Natural Experiment on ChatGPT Referral Traffic

This paper uses a log-based natural experiment on a single domain with an on-site control group to demonstrate that while raw ChatGPT referral growth is largely driven by platform expansion, targeted Answer Engine Optimization (AEO) interventions yield a statistically suggestive but not conclusive causal lift of approximately 1.82x, highlighting the need to disentangle platform tailwinds from genuine optimization effects.

Original authors: Keisuke Watanabe, Kazuki Nakayashiki

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Keisuke Watanabe, Kazuki Nakayashiki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet is a massive, bustling city. For years, people found places in this city using a giant, traditional map (Google Search). But recently, a new, magical tour guide has appeared: ChatGPT. Instead of just giving you a list of addresses, this tour guide answers your questions directly and points you to specific shops (websites) to visit.

This paper is a scientific investigation into a question many business owners are asking: "If we change our shop to look better for this new tour guide, will we get more customers? And how many of those customers are actually because of our changes, versus just because the tour guide itself is becoming popular?"

Here is the story of what the researchers found, broken down simply.

The Problem: The "Rising Tide" Confusion

Imagine the whole city is experiencing a massive population boom. Every single shop is getting more customers just because there are more people in town.

The Trap: If a shop owner says, "We got 5 times more customers this month! Our new sign must have worked!" they might be wrong. Maybe the sign did help, but maybe they just got lucky because the whole city grew.
The Study's Goal: The researchers wanted to separate the "sign effect" (their optimization) from the "city growth effect" (the platform getting popular).

The Experiment: A "Twin" City Strategy

The researchers studied one specific website called Glasp (think of it as a giant library of summaries about YouTube videos).

The Treatment Group: They took a specific section of the library (the YouTube summaries) and gave them a "makeover" in January 2026. They rewrote titles to look like questions, fixed duplicate pages, and made the summaries clearer for the AI tour guide.
The Control Group (The Twin): They left the rest of the library alone. No changes were made to the other pages.
Why this works: Since both groups are in the same building (the same website), they both feel the exact same "city boom." If the treated section grows faster than the untouched section, that extra growth is likely due to the makeover, not just the city getting bigger.

The Makeover (What they actually did)

The researchers applied a bundle of four changes to the YouTube pages:

Cleaning up addresses: Making sure every video had only one unique web address (no duplicates).
Listening to the robots: When AI bots tried to visit pages that didn't exist, the researchers created new pages to match what those bots were looking for.
Rewriting the headlines: Changing titles to look like questions and adding short, clear answers at the top.
The "Safety Net": They promised not to change pages that were already doing well on Google, so they wouldn't accidentally hurt their existing traffic.

The Results: What Actually Happened?

1. The Raw Numbers are Misleading
If you just looked at the treated pages, they grew 6.1 times bigger. That sounds amazing! But the untouched pages also grew 3.5 times bigger just because the AI tour guide (ChatGPT) was getting more users.

Analogy: If you run a lemonade stand and sales go up 6x, but your neighbor's stand (who didn't change anything) also went up 3.5x because a heatwave hit the whole neighborhood, you can't claim all that growth is your fault.

2. The Real "Makeover" Effect
After doing the math to subtract the "neighbor's growth" from the "total growth," the researchers found the actual effect of their changes was a 1.8 to 2.3 times increase.

The Catch: While this is a solid improvement, the statistical test was a bit shaky because the "before" data was short and noisy. The researchers call this result "suggestive" rather than 100% proven. It's a strong hint, but not a slam-dunk guarantee.

3. The Safety Net Worked
A big fear was that changing the pages for the AI tour guide might ruin their performance on the traditional map (Google Search).

The Result: The traditional traffic dropped slightly, but only because everyone's traffic dropped slightly that year. The researchers' changes did not cause a special crash. They successfully optimized for the new guide without breaking the old map.

The Big Takeaway

The paper teaches us a valuable lesson about marketing and data: Don't get fooled by the "Platform Tailwind."

When a new technology (like AI answer engines) explodes in popularity, everyone gets a boost. If you see a business claim "We grew 10x because of our AI strategy," they might just be riding the wave of the platform's own popularity.

To know if your strategy actually works, you need to compare your results against a "control group" that didn't change anything but was still riding the same wave. In this case, the real magic of the strategy was a solid ~2x boost, not the headline-grabbing 6x number.

In short: The new AI tour guides are great, and optimizing for them helps, but you have to be careful not to credit your own hard work for the sheer size of the crowd showing up at the door.

Technical Summary: Disentangling Answer Engine Optimization from Platform Growth

Problem Statement
Large Language Model (LLM) "answer engines" (e.g., ChatGPT) have emerged as a distinct referral channel, prompting the practice of Answer Engine Optimization (AEO). While public case studies often cite dramatic raw growth multiples for sites adopting AEO, these figures are confounded by the explosive, platform-level growth of the answer engines themselves. The central challenge is distinguishing the causal effect of optimization interventions from the "platform tailwind" that lifts traffic for all sites indiscriminately. Existing literature often relies on synthetic benchmarks or third-party citation estimators, failing to measure realized downstream referral traffic on live domains while controlling for external growth factors.

Methodology
The authors conduct a longitudinal field study on a single high-traffic domain (glasp.co) using a natural experiment design.

Experimental Design: The study utilizes an on-domain control group. A specific corpus of hundreds of thousands of YouTube question-and-answer pages (/youtube/) received a defined bundle of AEO interventions in January 2026. The remainder of the domain (articles, profiles, etc.) received no interventions and serves as a contemporaneous control. Both groups share the same domain authority, analytics configuration, and exposure to the platform tailwind.
Interventions (The Treatment): Applied to the /youtube/ corpus, the treatment bundle included:
1. URL canonicalization (consolidating duplicate patterns).
2. Demand mining from bot 404 logs to generate new pages.
3. Rewriting titles into question forms and lead summaries into standalone answers.
4. An "SEO Guard" rule to lock pages with existing organic traffic from rewriting.
Data Sources: The study relies on first-party server logs and Google Analytics 4 (GA4) Data API for referral sessions, and Google Search Console for organic search metrics. Absolute counts are withheld for confidentiality; all analysis uses relative growth multiples and indices.
Statistical Approach: The authors employ an Interrupted Time Series (ITS) model on the weekly ratio of treated-to-control traffic ( $R(t) = \text{treated}(t)/\text{control}(t)$ ). This segmented regression isolates the treatment effect by netting out the multiplicative platform shock. To address serial correlation and the short pre-period, the study uses HAC (Newey–West) standard errors and a "placebo-in-time" permutation test to assess the robustness of the observed break.

Key Results

Raw Multiples are Misleading: Total ChatGPT referrals on the domain grew $5.7\times$ from January to May 2026. However, the untreated control group grew $3.5\times$ over the same period, indicating that the majority of raw growth is attributable to the platform tailwind.
Causal Effect Estimation: The ITS model on the treated/control ratio estimates a discrete, intervention-aligned level increase of $1.82\times$ (95% CI 1.31–2.54, HAC $p=0.001$ ). This result is robust across specifications, including engagement-filtered sessions ( $2.27\times$ ) and alternative pre-period windows.
Statistical Caution: A conservative placebo-in-time permutation test yields a $p$ -value of 0.16. The authors conclude that while the regression effect is significant under HAC errors, the short and noisy pre-period means the effect is suggestive, not conclusive.
SEO Safety: Google organic clicks to treated pages declined by approximately 25%, mirroring a similar site-wide decline of 20%. There was no evidence of de-indexation or a collapse specific to the treated group, supporting the "SEO-protection rule" that targeted rewriting need not harm organic search performance.
Pre-Trend: A significant upward pre-trend ( $+11\%$ /month) existed in the treated/control ratio before the intervention, which the model explicitly accounts for.

Significance and Claims
The paper argues that the methodological contribution—separating treatment effects from platform tailwind using an on-domain control—is more valuable than any single growth multiple.

Overstated Claims: The authors posit that headline AEO multiples in public discourse substantially overstate causal effects by conflating optimization with platform growth.
Defensible Metric: The "honest unit" of an AEO result is a conservative, range-reported effect estimate (e.g., $1.8\times$ – $2.3\times$ ) adjusted for platform growth, rather than raw before/after multiples.
Safety: The study validates that AEO interventions can be implemented without measurable damage to existing organic search performance, provided a "guarded" rewrite strategy is used.
Limitations: The authors explicitly acknowledge the study is observational, not randomized, and limited to a single domain and engine (ChatGPT). They note that the "flywheel" hypothesis (that AEO citations lift organic search) was not supported in this window.

Conclusion
The study demonstrates that while AEO can yield a measurable lift in referral traffic (approximately $1.8\times$ – $2.3\times$ after adjusting for platform growth), the raw growth figures often cited in the industry are dominated by the rapid expansion of the answer engines themselves. The paper advocates for the adoption of on-domain controls and first-party log analysis as a standard for reproducible and honest AEO measurement.