📊 statistics

From Stochastic to Stable: Rank Stability and Structural Sufficiency in AI Visibility Measurement

This paper introduces a sequential convergence framework that determines the sufficiency of AI visibility data by evaluating rank stability and structural sufficiency, enabling practitioners to stop data collection based on observed distributional regularities rather than arbitrary fixed budgets.

Original authors: Ronald Sielinski

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ronald Sielinski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out which ten websites are the "superstars" of the internet for a specific topic, like "how to bake the perfect chocolate cake." You ask a bunch of AI chatbots (like Gemini, SearchGPT, or Perplexity) to give you answers, and you count how often they mention each website.

But here's the catch: these AI chatbots are a bit like moody artists. If you ask the exact same question twice, they might give you slightly different answers and cite different websites. Sometimes they mention a site you love; other times, they forget it entirely. This makes measuring who is "winning" feel like trying to catch smoke with a net.

For a long time, people trying to measure this just guessed. They'd say, "Okay, let's ask 100 questions and see what happens," or "Let's ask until the list looks 95% similar to the last one." The paper by Ronald Sielinski says, "Stop guessing!" It argues that there is no magic number of questions that works for every situation. Asking 100 questions might be plenty for one topic but totally useless for another.

Instead, the paper introduces a smart, self-checking system that acts like a two-step quality control gate. You don't stop collecting data until both gates swing open.

Gate 1: The "Stop Wiggling" Check (Rank Stability)

Imagine you are watching a race where the runners keep swapping places every few seconds. At first, the order is chaotic. The paper says you can't declare a winner until the runners stop wiggling around and settle into a steady order.

The authors tried a simple rule: "Stop when the order is 95% the same as before." But they found this rule fails for some AI chatbots. Because some bots are "sparse" (they only mention a few websites per answer), their lists are naturally noisy. Even if the true order has settled, the noise might keep the similarity score below 95%, making you think the race is still chaotic when it's actually over.

So, the paper's new method doesn't look for a specific score. Instead, it looks for a flat line. It asks: "Has the order stopped changing significantly?" It uses a mathematical trick to find the moment the list stops shifting and just sits there, even if the score is only 88% or 92%. This is the first gate: the list must be stable.

Gate 2: The "Signal vs. Noise" Check (Structural Sufficiency)

Okay, the list has stopped changing. Great! But is it accurate?

Imagine you are trying to hear a whisper in a noisy room. Even if the whisper doesn't change its words (it's stable), you still can't understand it if the room is too loud. In this analogy, the "whisper" is the difference in popularity between two websites, and the "noise" is the uncertainty caused by the AI's random behavior.

The paper introduces a second gate called Structural Sufficiency. It asks a simple question: "Is the difference between the websites big enough to be heard over the noise?"

It calculates a ratio (called a Signal-to-Noise Ratio, or SNR).

If the SNR is less than 1, the noise is louder than the signal. The websites are so close in popularity that you can't tell who is actually better; the order might just be a fluke.
If the SNR is 1 or higher, the signal is strong enough. The spread of popularity is wide enough that you can trust the ranking.

This gate is clever because it doesn't demand a specific number of questions. If the websites are very different in popularity, you might reach this gate quickly. If they are all very similar, you might need to ask hundreds more questions to prove who is truly on top.

The Big Reveal

The paper tested this two-gate system on 30 different combinations of topics (like "marathon shoes" or "identity protection") and AI platforms. Here is what they found:

No Fixed Number Works: They proved that you cannot just say "ask 50 questions" for everything. Some topics needed as few as 33 responses, while others needed 94. Three combinations on one specific platform (SearchGPT) never even reached the finish line within the 125 responses they tested, because the noise was just too high to separate the winners from the losers.
The "95% Rule" is Flawed: They showed that sticking to a rigid "95% similarity" rule would have made you stop too late for some topics and never stop at all for others.
The Two Gates are Both Necessary: Sometimes the list stops wiggling (Gate 1) but is still too noisy to trust (Gate 2). Other times, the noise is low, but the list is still flipping around. You need both gates to open to know you have a reliable answer.

Why This Matters

Think of this like baking a cake. You can't just say, "Bake for 30 minutes." If your oven is hot, 30 minutes burns it. If your oven is cold, 30 minutes leaves it raw. You have to check if the cake is done (stable) and if it's cooked through (sufficient).

The paper gives us a thermometer and a toothpick test for AI visibility. It tells us to stop collecting data only when the AI's answers have settled down and when the differences between the websites are clear enough to trust. It's a way to stop wasting time asking questions that don't help, and to stop making decisions based on guesses.

In short: Don't guess the number. Watch the data. Wait until the list stops shaking and until the differences are loud enough to hear. Only then is the answer ready.

Technical Summary: From Stochastic to Stable

Problem Statement

AI visibility measurement in generative search engines is fundamentally a comparative exercise aimed at determining which domains are cited most frequently and whether observed differences are statistically significant enough to inform competitive strategy. However, the industry lacks a principled method to determine when sufficient data has been collected to support these comparisons. Current practices rely on fixed collection budgets or arbitrary correlation thresholds, which fail to account for the stochastic nature of generative search.

Generative search engines produce variable outputs for the same query, making citation share a sample statistic subject to sampling variability rather than a fixed quantity. This variability creates two distinct challenges:

Rank Instability: Early in data collection, the ordering of domains shifts significantly as top-cited domains have not yet accumulated enough citations to establish a consistent lead.
Measurement Uncertainty: Even after a rank ordering stabilizes, the confidence intervals (CIs) around citation shares may be too wide to distinguish between adjacent domains. A stable rank order is not necessarily a precise one; if the "signal" (spread of shares) is smaller than the "noise" (uncertainty of estimates), the ranking cannot support reliable inference.

The core difficulty is that citation density and distributional structure vary significantly across platforms and topics. Consequently, no fixed query count or fixed rank-correlation threshold (e.g., $\rho \ge 0.95$ ) can be justified universally.

Methodology

The paper introduces a sequential convergence framework based on two complementary criteria derived directly from the observed citation distribution, requiring no externally specified targets for query count, correlation level, or CI width.

1. Data and Estimation

Dataset: The framework was applied to 30 platform-topic combinations across three generative search engines: Google Gemini, OpenAI SearchGPT, and Perplexity. Ten diverse topics were used, with 125 queries generated per topic per platform.
Bootstrap Resampling: To quantify uncertainty, the authors use multinomial bootstrap resampling at the query level (not the citation level) to preserve within-response citation clustering. Citation shares are recomputed for each bootstrap iteration, and 95% BCa (bias-corrected and accelerated) confidence intervals are calculated.
Established vs. Observed Domains: A distinction is made between "observed" domains (cited at least once) and "established" domains (those whose citation-share 95% CI excludes zero). Structural sufficiency is evaluated only over the set of established domains.

2. Criterion I: Rank Stability

Rank stability is determined by detecting a structural plateau in the rank-correlation trajectory, rather than crossing a fixed threshold.

Metric: Weighted Spearman rank correlation ( $\rho$ ) is computed between successive cumulative windows of responses. Weights are the average citation shares across the two windows, ensuring that shifts among high-share domains contribute more to the correlation than shifts in the long tail.
Detection: A segmented model fit (using the Pool-Adjacent-Violators Algorithm for isotonic regression) is applied to the correlation trajectory. The model tests for a "rising phase" followed by a "flat tail."
Decision Rule: Stability is declared when the Bayesian Information Criterion (BIC) selects a model with a flat tail over a pure rising model, and that flat tail spans at least 15 consecutive responses. This approach detects when the ordering has stopped changing, regardless of the absolute correlation value (which may be lower than 0.95 in sparse citation environments).

3. Criterion II: Structural Sufficiency

Structural sufficiency evaluates whether the spread of citation shares among established domains is large enough relative to their measurement uncertainty to support inference.

Metric: A Structural Signal-to-Noise Ratio (SNR) is calculated:
$\text{SNR} = \frac{\sigma(s_d : d \in D_{est})}{\bar{h}_{CI}}$
Where $\sigma$ is the standard deviation of citation shares across established domains (signal), and $\bar{h}_{CI}$ is the mean half-width of their 95% bootstrap confidence intervals (noise).
Decision Rule: Sufficiency is achieved when $\text{SNR} \ge 1$ . This indicates that the spread of shares exceeds the typical measurement uncertainty. The raw SNR series is smoothed using an Exponentially Weighted Moving Average (EWMA) to prevent transient fluctuations from triggering premature stopping.

4. Conjunctive Convergence

A platform-topic combination is deemed "converged" only when both rank stability and structural sufficiency are satisfied simultaneously. The stopping point is the later of the two individual firing orders.

Key Results

The framework was applied to 30 platform-topic combinations (10 topics $\times$ 3 platforms).

Convergence Rates: Within a 125-response collection window, 27 of the 30 combinations converged. The three non-converging cases were all on SearchGPT for specific topics ("identity protection," "travel destinations," "homemade chili"), where the citation shares were too tightly packed relative to the measurement noise for the SNR to reach 1.
Binding Constraints:
- In 16 of the 27 converged cases, rank stability was the binding constraint (the last condition to be met).
- In 11 of the 27 converged cases, structural sufficiency was the binding constraint.
- This demonstrates that stability alone is insufficient; precision often lags behind stability.
Failure of Fixed Thresholds: A fixed correlation threshold of $\rho \ge 0.95$ sustained for 15 responses was met by only 19 of the 30 combinations. The plateau-based detection method successfully identified stability in all 30 cases, including those where citation density was too low to ever reach $\rho = 0.95$ .
Validation: The authors compared the ordering at the stopping point against the ordering at the full 125-response window.
- Stopping based on rank stability alone resulted in poor agreement in some cases (e.g., SearchGPT/"identity protection" had a correlation of only 0.364 with the final ranking).
- The conjunctive criterion improved the "floor" of agreement, with the worst-case certified agreement rising to 0.717.
- The median agreement between stopped and final orderings for converged cases was 0.844.
Platform Differences: Convergence speed was not determined by platform identity but by citation density and within-query clustering (Design Effect). SearchGPT, despite citing fewer domains per response, had lower clustering, meaning each response contributed more "effective" data than a response from Gemini or Perplexity.

Significance and Contributions

The paper makes three primary contributions to the field of generative search optimization and evaluation:

Rank Stability as a Convergence Criterion: It establishes that rank stability should be detected via a structural plateau in the correlation trajectory (using model selection) rather than by crossing a fixed correlation threshold. This resolves the issue where low citation density prevents fixed thresholds from ever being met, even when the underlying distribution has stabilized.
Structural Sufficiency as a Stopping Rule: It introduces a data-driven stopping criterion based on the ratio of signal (share spread) to noise (CI width). This moves beyond simple sample size estimation to ensure that the collected data is precise enough to distinguish between competitors.
Rejection of Fixed Budgets: The results demonstrate that no fixed collection budget (query count) is universally sufficient. The amount of data required depends entirely on the specific citation distribution of the platform and topic. The framework provides a practical, self-calibrating mechanism to determine when a measurement is ready for comparative analysis.

The framework is presented as a practical basis for determining when AI visibility measurements have transitioned from stochastic noise to a stable, reliable signal, enabling practitioners to make competitive decisions with known uncertainty bounds.