LLMs are non-deterministic. The brands and citations in an AI answer change based on several factors at once: prompt quality and phrasing, how often you run it, region and locale, which AI engine you ask, whether you query the API or the chat interface, and the persona held in memory. Run the same prompt twice and you get two different answers. In the largest public study of the problem, fewer than 1 in 100 repeat runs produced the same set of brands and fewer than 1 in 1,000 produced the same order. Rank is therefore unmeasurable; appearance frequency across many runs is not.
AI brand visibility changes with almost every run
Ask an AI engine to recommend brands in your category a hundred times and you will get close to a hundred different answers.
That is not a metaphor. In the largest public study of the question, fewer than 1 in 100 repeat runs produced the same set of brands, and fewer than 1 in 1,000 produced the same brands in the same order. Rand Fishkin of SparkToro and Patrick O'Donnell of Gumshoe.ai had 600 volunteers run 12 brand-recommendation prompts through ChatGPT, Claude and Google's AI surfaces 2,961 times across November and December 2025, at 60 to 100 runs per prompt per engine. The brands changed. The order changed. Even the number of items in the list changed.
Fishkin's conclusion was blunt: "any tool that gives a 'ranking position in AI' is full of baloney." His broader framing: "These tools are probability engines: they're designed to generate unique answers every time. Thinking of them as sources of truth or consistency is provably nonsensical."
But the same study contains the part most people skipped. Presence is stable even when position is not. Leading headphone brands appeared in 55% to 77% of 994 responses. City of Hope turned up in 69 of 71 ChatGPT answers about West Coast cancer care, a 97% appearance rate, while ranking first in only 25 of them. The consideration set held. The leaderboard did not.
That gap is the whole story of AI visibility tracking right now. Rank is noise. Frequency is signal. Most dashboards sell the noise.
The variables that are actually moving
Non-determinism is the floor, not the whole story. Sampling guarantees some variation even with everything else held still, but in production at least eight variables move at once, and most teams change several without realising it. The matrix below separates the ones you control from the ones you rent.
The practical read: search indexation, prompt quality, run frequency, engine choice and collection surface are yours. Region is partly yours. Persona, memory and model version are not. That split determines where measurement effort is worth spending, and it is why a single number with no stated design behind it is uninterpretable. Two teams can report wildly different visibility for the same brand on the same day and both be right, because they were not measuring the same thing.
LLM non-determinism: why identical prompts diverge at temperature zero
Two separate things make AI answers move, and only one of them is deliberate. The deliberate one is sampling: temperature and top-p pick each token from a probability distribution, so variation is the design. Turn temperature to zero and that layer stops. The answers still change.
In September 2025, Thinking Machines Lab published the clearest explanation of the second layer. Researcher Horace He sampled 1,000 completions of the same prompt from Qwen3-235B at temperature 0 and got 80 unique completions, the most common appearing 78 times. Every completion was identical for the first 102 tokens, then split: 992 continued one way, 8 another. The popular explanation, that GPU concurrency and floating-point rounding cause the drift, turns out to be wrong. The forward pass is deterministic for a fixed batch.
The real culprit is that inference kernels are not batch-invariant. As the paper puts it: "the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies." Your answer depends on how many strangers hit the server at the same moment. With batch-invariant kernels enabled, all 1,000 completions came back identical.
Academic work backs this up from a different angle. Atil and colleagues, testing five API models configured to be deterministic, found accuracy varying by up to 15% across runs and gaps of up to 70% between the best and worst run. Their summary: no model consistently delivers the same outputs or accuracies, regardless of task. GPT-5 does not expose a temperature parameter at all, so for that model the question is settled.
Three more layers sit on top in production. Retrieval: the live index moves under the model, and whether web search fires at all varies by query. Session context: memory, account state, geography, device and silent A/B tests. Model version: a rollout resets everything, usually without a changelog you can read.
What AI visibility metrics do when you run the same prompt twice
Most of it is the model. That is the most useful finding of the most rigorous study on this question, and it changes what you should do about it.
Schulte, Bleeker and Kaufmann at the University of St. Gallen scraped four engines across four verticals for 45 days, then ran a second dataset where identical prompts were fired up to ten times within the same day. If instability came mainly from the outside world, the simultaneous runs would agree closely. They did not.
- Day to day, source overlap ran 0.34 to 0.42 on Jaccard similarity. Roughly two thirds of cited sources changed daily. Brands were steadier at 0.45 to 0.59.
- Same day, back to back, source overlap was 0.32 to 0.43 and brand overlap 0.33 to 0.48. Almost as unstable, with the external world held still.
- Citation concentration is severe. Mean Gini coefficient of 0.715 across engines, peaking at 0.782 for Google AI Mode. A handful of domains absorb most citations.
The authors' own framing: "single observations of AI visibility are misleading and risk over- or underestimating true brand presence... a substantial fraction of observed instability originates from the model's stochastic generation process itself."
Per-engine stability differs, and the ranking is not what most people assume. Perplexity and ChatGPT held brands most consistently. Gemini held sources most consistently. ChatGPT was least stable on sources, partly because it suppresses web search on a majority of definitional queries and answers from weights instead.
AI search visibility: no two engines cite the same sources
If brand mentions are unstable, citations are worse. They shift between runs, between engines, and between AI answers and the classic search results everyone has spent twenty years optimising for.
Profound analysed 100,000 identical prompts across ChatGPT and Perplexity and found only 11.0% of cited domains appeared in both. 37.4% were ChatGPT-exclusive and 51.6% Perplexity-exclusive. Their wider dataset explains why: Wikipedia accounts for roughly 47.9% of ChatGPT's top-ten source share, while Reddit accounts for roughly 46.7% of Perplexity's. These are different machines reading different libraries.
Inside Google it is no better. Ahrefs compared 540,000 query pairs and found AI Mode and AI Overviews cite the same URLs only 13.7% of the time, and share an identical opening sentence just 2.5% of the time, yet reach 86% semantic similarity. Despina Gavoyannis of Ahrefs summarised it neatly: "9 out of 10 times, AI Mode and AI Overview agreed on what to say." They agreed on the message and disagreed on the receipts.
And the link to organic rankings is weaker than the industry wants it to be. Moz's Tom Capper, analysing roughly 40,000 queries in February 2026, found that 88% of AI Mode citations are not in the organic SERP for the same query. Only about one in ten citations match the exact URLs in Google's top ten.
Why a 1% AI visibility score tells you almost nothing
A brand mentioned once in a hundred answers is statistically indistinguishable from a brand mentioned never, and from one mentioned five times as often. This is the part of AI visibility measurement that vendors quietly skip, and it is arithmetic, not opinion.
The right tool is the Wilson score interval, not the textbook Wald formula, because Wald collapses near 0% and 100%. Wilson answers the question you actually have: given what I observed, what range of true rates is consistent with it?
| Runs | Observed | 95% Wilson interval | What you can honestly say |
|---|---|---|---|
| 20 | 5.0% | 0.9% to 23.6% | Nothing |
| 50 | 2.0% | 0.4% to 10.5% | Nothing useful |
| 100 | 1.0% | 0.2% to 5.5% | Rare, magnitude unknown |
| 500 | 0.2% | 0.04% to 1.1% | Genuinely rare |
| 1,000 | 0.1% | 0.02% to 0.6% | Rare, and now bounded |
The zero case is worse, and it is the one people quote in board meetings. Observing zero mentions does not mean your rate is zero. The rule of three puts the 95% upper bound at roughly 3/n. Zero mentions in 20 runs is consistent with a true rate as high as 16%. You need around 60 runs before "we have no AI visibility" is a defensible sentence rather than a feeling.
The number that should end the argument: To distinguish a true 1% mention rate from a true 2% rate at 95% confidence with 80% power, you need roughly 2,318 runs per condition. No vendor runs that per prompt. So when a dashboard reports your visibility moved from 1.3% to 2.1%, it has reported nothing. Distinguishing 1% from 5% takes about 284 runs. Distinguishing 10% from 15% takes about 686.
How to measure AI visibility: the number of runs you actually need
Seven runs per prompt per engine is the floor for a usable per-brand number, and the returns fall off a cliff after eight. The St. Gallen team derived this directly, and it is the most immediately actionable finding in the literature.
A single run carries a standard error of 0.370 on per-brand detection rate. The authors describe it as essentially uninformative, which is a polite way of saying a one-run reading is a coin flip with extra steps. The error drops below 0.10 at seven runs, giving a 95% interval of roughly plus or minus 15.8 points, and below 0.08 at eight runs, tightening to about plus or minus 12.1 points. Source coverage needs eight.
Time behaves the same way. Per-brand error falls below 0.10 at around ten days of observation and below 0.05 at twenty-four days. A 14-day window still carries 0.080. Which gives a simple operating rule: at least seven runs per prompt per engine per day, aggregated over a two to four week rolling window.
| What you want to claim | Minimum runs | Window |
|---|---|---|
| Directional read only | 3-5 | Same day |
| Per-brand detection rate | 7-8 | 2-4 weeks rolling |
| Source coverage | 8+ | 2-4 weeks rolling |
| Broad consumer category | 60-100 | 4-8 weeks |
| "We have zero visibility" | 60+ | 2-4 weeks rolling |
| 1% vs 2% movement | ~2,300 | Not commercially viable |
AI visibility monitoring across four horizons of volatility
Volatility in AI answers is not one phenomenon. It is four, nested inside each other, and each needs a different fix. Conflating them is how teams end up celebrating a model rollout as a content win.
API or interface: what your AI visibility numbers actually capture
Measuring through an API and measuring through the consumer interface answer different questions, and the gap is large enough to change your conclusions.
APIs are fast, cheap, repeatable and hold variables still. They also query a developer surface that often runs without live web search, without personalisation, without interface modules, and sometimes on a different default model than the one your buyers use. Interface capture reflects what a customer actually sees, at the cost of noise and expense.
Vendor-run comparisons put the gap at roughly 24% brand overlap and 4% source overlap between API and interface results, with API responses averaging 406 words against 743 for the interface, and API capture missing brand mentions around 8% of the time.
Share of voice in AI search: measure like a pollster, not a rank tracker
Everything above points to one shift: stop treating AI answers as a leaderboard to be scraped and start treating them as a population to be sampled. Pollsters solved this problem decades ago. The tooling in this category has mostly not caught up.
- Kill rank as a headline metric. Report share of voice and appearance frequency. The data does not support anything finer.
- Put a Wilson interval on every number. Suppress or grey out any rate whose interval crosses the threshold a decision depends on.
- Set a sampling floor and publish it. Seven runs per prompt per engine minimum, aggregated over two to four weeks.
- Segment by engine, always. With 11 to 17% cross-engine citation overlap, a blended AEO score averages four unrelated distributions into one meaningless figure.
- Pin the model version and collection date to every datapoint, and mark rollouts on the chart.
- Treat sub-5% rates as a presence flag, not a trend line. Movement inside the noise band is not movement.
- Publish the methodology. Estimate, interval, effective sample, design, engine, window. Six fields. It is the cheapest trust moat available in this category.
Is an AI visibility audit worth running if the data is this noisy?
Yes, but for a different reason than most vendors claim. The honest case against AI visibility tracking is strong and worth stating properly.
- Semrush found AI-referred visitors converting at 4.4x the rate of organic (June 2025).
- Similarweb measured 2.15x in 2026 cross-site e-commerce data, at 11.4% against 5.3%.
- Ahrefs' own first-party data found 0.5% of visitors came from AI search but drove 12.1% of signups, a 23x conversion rate.
- Semrush and Kevin Indig found only 15.2% of 1,094 ChatGPT topic categories had a clear owner. The other 85% remain contestable.
The synthesis: Visibility is measurable if you sample properly. Rank is not measurable at all. And the number that survives every methodological objection is not a visibility score, it is revenue: did an AI-referred session become a customer. That question needs no confidence interval, because you are counting outcomes rather than estimating a proportion.
What to look for in an AI visibility tool: the citedby loop
Everything above is a measurement design problem. Here is the operating cadence that solves it in practice.
The shift is from audit to loop. A one-off scan inherits every problem in this report: one run, one day, one engine, no interval. A loop run on a cadence accumulates runs against a stable prompt panel, so the confidence interval narrows over time while you are also closing the content gaps the data exposes.
Set an AI visibility benchmark you can defend
citedby samples across ChatGPT, Perplexity, Gemini, Claude, Copilot and AI Overviews, then ties AI referrals to first-touch revenue.
Start freeExplore foundational concepts and research on AI visibility, non-determinism, and revenue attribution:
AI visibility measurement - FAQ
Why does ChatGPT give different answers to the same question?
Two reasons stack. Temperature and top-p sampling select each token probabilistically, so variation is by design. On top of that, inference kernels are not batch-invariant: server load changes the batch your request joins, which changes the arithmetic and the output. Thinking Machines Lab showed that even at temperature 0, 1,000 identical prompts produced 80 unique completions. Retrieval, session memory and model rollouts add further variance.
How many times should I run a prompt to measure AI visibility?
At least seven runs per prompt per engine per day. Research from the University of St. Gallen found per-brand detection-rate standard error falls below 0.10 at seven runs and below 0.08 at eight, while a single run carries an error of 0.370 and is essentially uninformative. Aggregate over a two to four week rolling window. Broad consumer categories need 60 to 100 runs, the range SparkToro used.
Is a 1% brand mention rate meaningful?
Not on its own. One mention in 100 runs carries a 95% Wilson confidence interval of 0.2% to 5.5%, so the true rate could be twenty-five times higher than observed. Distinguishing a genuine 1% rate from a genuine 2% rate at 95% confidence with 80% power requires roughly 2,318 runs per condition, which no commercial tool performs. Report sub-5% rates as a band, not a point estimate.
Does zero mentions mean I have no AI visibility?
No. The rule of three puts the 95% upper bound on a true rate at roughly 3/n when you observe zero events. Zero mentions across 20 runs is consistent with a true rate as high as about 16%. You need around 60 runs before the claim that a brand has no AI visibility is statistically defensible.
Can I trust an AI visibility tool that reports my ranking position?
Treat it with heavy scepticism. SparkToro and Gumshoe found fewer than 1 in 1,000 repeat runs produced the same brands in the same order, which means position is close to pure noise. Rand Fishkin's conclusion was that any tool giving a ranking position in AI is unreliable. Appearance frequency and share of voice across many runs remain valid; ordinal rank does not.
Why do ChatGPT and Perplexity cite completely different sources?
They read different indexes and weight them differently. Profound analysed 100,000 identical prompts and found only 11.0% of cited domains appeared in both engines. Wikipedia accounts for roughly 47.9% of ChatGPT's top-ten source share while Reddit accounts for roughly 46.7% of Perplexity's. This is why a single blended AI visibility score across engines is misleading.
Should AI visibility be measured through APIs or the chat interface?
They answer different questions. APIs are cheap and repeatable but query a developer surface that often lacks live search, personalisation and interface modules. Interface capture reflects what buyers actually see. Vendor comparisons report roughly 24% brand overlap between the two. Whichever you choose, publish which surface, model version, geography and session state produced each number.
What is the right way to report AI visibility to a leadership team?
Report an appearance rate with a 95% confidence interval, the effective sample, the design, the engine and the time window. For example: 34% mention rate (95% CI 29-39%), n=280 effective, 200 prompts x 3 runs, ChatGPT, 6-12 July. Then pair it with first-touch revenue attribution, since counted conversions need no confidence interval.
Related Glossary Specs & Myth Fact-Checks
Explore definitions and statistical fact-checks on AI visibility tracking and non-determinism:
Sources & References
- Horace He et al., Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference" (10 Sept 2025)
- Rand Fishkin & Patrick O'Donnell, "AIs are highly inconsistent when recommending brands or products" (28 Jan 2026)
- Schulte, Bleeker & Kaufmann, University of St. Gallen, "Don't Measure Once: Measuring Visibility in AI Search", arXiv 2604.07585 (Apr 2026)
- Atil et al., "Non-Determinism of Deterministic LLM Settings", arXiv 2408.04667 (2025)
- SE Ranking, "AI Mode Research: Sources, Volatility & Differences" (Aug 2025)
- Ahrefs, AI Mode vs AI Overviews citation study (Dec 2025)
- Search Engine Journal, "AI Recommendations Change With Nearly Every Query" (Jan 2026)