AI visibility tracking:
Why the same prompt returns different brands

Gururaj Pandurangi
Gururaj PandurangiFounder, ThriveStack
Aug 2026 · 14 min read

Fewer than 1 in 100 repeat runs return the same brands. That breaks rank tracking, but measurement still works. AI engines return a different answer almost every time you ask. This report explains why, quantifies it with first-party research, and gives you the sample sizes and confidence intervals needed to measure AI visibility without fooling yourself.

ThriveStack citedby
8 things change which brands AI names. You control 5.
Same prompt, different answer. Here is what moves it, and who holds the lever.
5You controlMeasure and fix these
  • Search indexationof unindexed pages stay invisible100%
  • Prompt qualitysimilarity between real prompts0.081
  • Run frequencyof reruns return the same brands<1%
  • AI enginesources ChatGPT and Perplexity share11%
  • API vs appbrand overlap between the two~24%
1Partly yoursPick the regions you sell in
  • Region and localemention rate, by country18% to 50%
2Out of your handsMark them on your charts
  • Persona and memorywith account, history, sessionVaries
  • Model versionon every rollout, no changelogResets
Sources: SparkToro (2026); Profound (100k prompts); St. Gallen (2026); Wellows; Semrush.
Figure 1: Eight variables move every AI answer. Explained in section 01.
Read on

LLMs are non-deterministic. The brands and citations in an AI answer change based on several factors at once: prompt quality and phrasing, how often you run it, region and locale, which AI engine you ask, whether you query the API or the chat interface, and the persona held in memory. Run the same prompt twice and you get two different answers. In the largest public study of the problem, fewer than 1 in 100 repeat runs produced the same set of brands and fewer than 1 in 1,000 produced the same order. Rank is therefore unmeasurable; appearance frequency across many runs is not.

80
unique completions from 1,000 identical prompts at temperature 0
11%
source overlap between ChatGPT and Perplexity on the same prompt
0.370
standard error of a single-run detection rate
88%
of AI Mode citations absent from the organic top 10
01 - The finding

AI brand visibility changes with almost every run

Ask an AI engine to recommend brands in your category a hundred times and you will get close to a hundred different answers.

This is measured. In the largest public study of the question, fewer than 1 in 100 repeat runs produced the same set of brands, and fewer than 1 in 1,000 produced the same brands in the same order. Rand Fishkin of SparkToro and Patrick O'Donnell of Gumshoe.ai had 600 volunteers run 12 brand-recommendation prompts through ChatGPT, Claude and Google's AI surfaces 2,961 times across November and December 2025, at 60 to 100 runs per prompt per engine. The brands changed. The order changed. Even the number of items in the list changed.

Fishkin's conclusion was blunt: "any tool that gives a 'ranking position in AI' is full of baloney." His broader framing: "These tools are probability engines: they're designed to generate unique answers every time. Thinking of them as sources of truth or consistency is provably nonsensical."

But the same study contains the part most people skipped. Presence is stable even when position is not. Leading headphone brands appeared in 55% to 77% of 994 responses. City of Hope turned up in 69 of 71 ChatGPT answers about West Coast cancer care, a 97% appearance rate, while ranking first in only 25 of them. The consideration set held. The leaderboard did not.

That gap is the whole story of AI visibility tracking right now. Rank is noise. Frequency is signal. Most dashboards sell the noise.

ThriveStack citedbyBrands keep showing up, but their order keeps changingSparkToro and Gumshoe: 2,961 runs of 12 prompts on 3 AI engines, Nov to Dec 2025HOW OFTEN THE EXACT SAME ANSWER COMES BACKSame brands, same order< 0.1%Same set of brands< 1%HOW OFTEN A BRAND SHOWS UP AT ALLCity of Hope (cancer care)97%Smartsites (marketing)89%Leading headphone brands55-77%Source: SparkToro & Gumshoe.ai, Jan 2026. Each prompt run 60-100x per engine.Not peer reviewed; authors disclose a commercial interest in AI tracking.
The two halves of the SparkToro finding. Exact repeats are vanishingly rare; appearance rates are consistent enough to act on.

The variables that are actually moving

Non-determinism is only the floor. Sampling guarantees some variation even with everything else held still, but in production at least eight variables move at once, and most teams change several without realising it. The matrix below separates the ones you control from the ones you rent.

The practical read: search indexation, prompt quality, run frequency, engine choice and collection surface are yours. Region is partly yours. Persona, memory and model version are not. That split determines where measurement effort is worth spending, and it is why a single number with no stated design behind it is uninterpretable. Two teams can report wildly different visibility for the same brand on the same day and both be right, because they were not measuring the same thing.

ThriveStack citedby
8 things change which brands AI names. You control 5.
Same prompt, different answer. Here is what moves it, and who holds the lever.
5You controlMeasure and fix these
  • Search indexationof unindexed pages stay invisible100%
  • Prompt qualitysimilarity between real prompts0.081
  • Run frequencyof reruns return the same brands<1%
  • AI enginesources ChatGPT and Perplexity share11%
  • API vs appbrand overlap between the two~24%
1Partly yoursPick the regions you sell in
  • Region and localemention rate, by country18% to 50%
2Out of your handsMark them on your charts
  • Persona and memorywith account, history, sessionVaries
  • Model versionon every rollout, no changelogResets
Sources: SparkToro (2026); Profound (100k prompts); St. Gallen (2026); Wellows; Semrush.
Eight variables, all live at once (also shown at the top of this page). Focus your measurement design on the green group, the five you control.
02 - The mechanism

LLM non-determinism: why identical prompts diverge at temperature zero

Two separate things make AI answers move, and only one of them is deliberate. The deliberate one is sampling: temperature and top-p pick each token from a probability distribution, so variation is the design. Turn temperature to zero and that layer stops. The answers still change.

In September 2025, Thinking Machines Lab published the clearest explanation of the second layer. Researcher Horace He sampled 1,000 completions of the same prompt from Qwen3-235B at temperature 0 and got 80 unique completions, the most common appearing 78 times. Every completion was identical for the first 102 tokens, then split: 992 continued one way, 8 another. The popular explanation, that GPU concurrency and floating-point rounding cause the drift, turns out to be wrong. The forward pass is deterministic for a fixed batch.

The real culprit is that inference kernels are not batch-invariant. As the paper puts it: "the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies." Your answer depends on how many strangers hit the server at the same moment. With batch-invariant kernels enabled, all 1,000 completions came back identical.

Academic work backs this up from a different angle. Atil and colleagues, testing five API models configured to be deterministic, found accuracy varying by up to 15% across runs and gaps of up to 70% between the best and worst run. Their summary: no model consistently delivers the same outputs or accuracies, regardless of task. GPT-5 does not expose a temperature parameter at all, so for that model the question is settled.

Three more layers sit on top in production. Retrieval: the live index moves under the model, and whether web search fires at all varies by query. Session context: memory, account state, geography, device and silent A/B tests. Model version: a rollout resets everything, usually without a changelog you can read.

ThriveStack citedbyWhy the same prompt gets a different answer: five causesEach layer adds change. Only the first one is under your control.Identicalprompt01SamplingTemperature and top-p pick tokens probabilistically. Intended.02Batch non-invarianceServer load changes batch size, which changes the maths. Not intended.03RetrievalThe live index moves. Whether search fires at all varies by query.04Session contextMemory, account state, geography, device, silent A/B tests.05Model versionA rollout resets the whole surface with no notice and no changelog.Layers 01-02 after Thinking Machines Lab (Sept 2025); 03-05 observed in tracking.
Only the top layer is under your control. Everything below it is infrastructure you rent.
03 - The evidence

What AI visibility metrics do when you run the same prompt twice

Most of it is the model. That is the most useful finding of the most rigorous study on this question, and it changes what you should do about it.

Schulte, Bleeker and Kaufmann at the University of St. Gallen scraped four engines across four verticals for 45 days, then ran a second dataset where identical prompts were fired up to ten times within the same day. If instability came mainly from the outside world, the simultaneous runs would agree closely. They did not.

  • Day to day, source overlap ran 0.34 to 0.42 on Jaccard similarity. Roughly two thirds of cited sources changed daily. Brands were steadier at 0.45 to 0.59.
  • Same day, back to back, source overlap was 0.32 to 0.43 and brand overlap 0.33 to 0.48. Almost as unstable, with the external world held still.
  • Citation concentration is severe. Mean Gini coefficient of 0.715 across engines, peaking at 0.782 for Google AI Mode. A handful of domains absorb most citations.

The authors' own framing: "single observations of AI visibility are misleading and risk over- or underestimating true brand presence... a substantial fraction of observed instability originates from the model's stochastic generation process itself."

Per-engine stability differs, and the ranking is not what most people assume. Perplexity and ChatGPT held brands most consistently. Gemini held sources most consistently. ChatGPT was least stable on sources, partly because it suppresses web search on a majority of definitional queries and answers from weights instead.

Cross-study disagreement worth naming: St. Gallen finds Perplexity most stable on brands; SparkToro found Claude marginally most consistent on brand set but worst on ordering. Different prompt sets, categories, languages and metrics. There is no settled answer on which engine is most deterministic, and anyone who tells you otherwise is extrapolating from one dataset.
ThriveStack citedbySame prompt, same day: how much two answers agreeJaccard similarity by engine: 1.00 means identical, 0.00 means nothing in commonBrands mentionedSources citedPerplexity0.490.28ChatGPT0.440.23Gemini0.410.51Google AI Mode0.380.32Source: Schulte, Bleeker & Kaufmann, Univ. of St. Gallen, arXiv 2604.07585(Apr 2026). Swiss-German prompt set; up to 10 same-day runs per prompt.
Same prompt, same day, two runs. Even with the outside world held still, agreement rarely passes 50%.
04 - Citations

AI search visibility: no two engines cite the same sources

If brand mentions are unstable, citations are worse. They shift between runs, between engines, and between AI answers and the classic search results everyone has spent twenty years optimising for.

Profound analysed 100,000 identical prompts across ChatGPT and Perplexity and found only 11.0% of cited domains appeared in both. 37.4% were ChatGPT-exclusive and 51.6% Perplexity-exclusive. Their wider dataset explains why: Wikipedia accounts for roughly 47.9% of ChatGPT's top-ten source share, while Reddit accounts for roughly 46.7% of Perplexity's. These are different machines reading different libraries.

Inside Google it is no better. Ahrefs compared 540,000 query pairs and found AI Mode and AI Overviews cite the same URLs only 13.7% of the time, and share an identical opening sentence just 2.5% of the time, yet reach 86% semantic similarity. Despina Gavoyannis of Ahrefs summarised it neatly: "9 out of 10 times, AI Mode and AI Overview agreed on what to say." They agreed on the message and disagreed on the receipts.

And the link to organic rankings is weaker than the industry wants it to be. Moz's Tom Capper, analysing roughly 40,000 queries in February 2026, found that 88% of AI Mode citations are not in the organic SERP for the same query. Only about one in ten citations match the exact URLs in Google's top ten.

ThriveStack citedbyAI engines rarely cite the same sources for one questionShare of cited sources that two answers have in common, same queryAI Mode vs AI Overviews13.7%ChatGPT vs Perplexity11.0%Four engines, average pair~17%Four engines, all agree3.8%AI Mode vs organic top 1012%Sources: Ahrefs (540k query pairs, Dec 2025); Profound (100k prompts);Writesonic (161,286 prompts); Moz (~40k queries, Feb 2026).
Overlap between cited source sets. The last two bars use warm grey because they measure disagreement with the organic index rather than between engines.
05 - The 1-in-100 problem

Why a 1% AI visibility score tells you almost nothing

A brand mentioned once in a hundred answers is statistically indistinguishable from a brand mentioned never, and from one mentioned five times as often. This is the part of AI visibility measurement that vendors quietly skip, and it is plain arithmetic.

The right tool is the Wilson score interval rather than the textbook Wald formula, because Wald collapses near 0% and 100%. Wilson answers the question you actually have: given what I observed, what range of true rates is consistent with it?

ThriveStack citedby
Table 1. What one observed mention rate really means at five sample sizes (95% Wilson range)
#RunsObserved95% Wilson intervalWhat you can honestly say
1205.0%0.9% to 23.6%Nothing
2502.0%0.4% to 10.5%Nothing useful
31001.0%0.2% to 5.5%Rare, magnitude unknown
45000.2%0.04% to 1.1%Genuinely rare
51,0000.1%0.02% to 0.6%Rare, and now bounded

The zero case is worse, and it is the one people quote in board meetings. Observing zero mentions does not mean your rate is zero. The rule of three puts the 95% upper bound at roughly 3/n. Zero mentions in 20 runs is consistent with a true rate as high as 16%. You need around 60 runs before "we have no AI visibility" is a defensible sentence rather than a feeling.

The number that should end the argument: To distinguish a true 1% mention rate from a true 2% rate at 95% confidence with 80% power, you need roughly 2,318 runs per condition. No vendor runs that per prompt. So when a dashboard reports your visibility moved from 1.3% to 2.1%, it has reported nothing. Distinguishing 1% from 5% takes about 284 runs. Distinguishing 10% from 15% takes about 686.

ThriveStack citedbyMore runs shrink the guesswork around one mention95% Wilson range of true rates when a brand shows up once in n runs0%5%10%15%20%25%n = 200.9-23.6n = 500.4-10.5n = 1000.2-5.5n = 5000.04-1.1n = 10000.02-0.6Dot = observed rate. Bar = range of true rates consistent with it.Wilson score interval, z = 1.96. Calculated directly for this report.
One observed mention, five sample sizes. The dot barely moves; the honest uncertainty around it collapses by a factor of forty.
06 - Sample size

How to measure AI visibility: the number of runs you actually need

Seven runs per prompt per engine is the floor for a usable per-brand number, and the returns fall off a cliff after eight. The St. Gallen team derived this directly, and it is the most immediately actionable finding in the literature.

A single run carries a standard error of 0.370 on per-brand detection rate. The authors describe it as essentially uninformative, which is a polite way of saying a one-run reading is a coin flip with extra steps. The error drops below 0.10 at seven runs, giving a 95% interval of roughly plus or minus 15.8 points, and below 0.08 at eight runs, tightening to about plus or minus 12.1 points. Source coverage needs eight.

Time behaves the same way. Per-brand error falls below 0.10 at around ten days of observation and below 0.05 at twenty-four days. A 14-day window still carries 0.080. Which gives a simple operating rule: at least seven runs per prompt per engine per day, aggregated over a two to four week rolling window.

ThriveStack citedby
Table 2. Minimum runs and time window needed for each AI visibility claim
#What you want to claimMinimum runsWindow
1Directional read only3-5Same day
2Per-brand detection rate7-82-4 weeks rolling
3Source coverage8+2-4 weeks rolling
4Broad consumer category60-1004-8 weeks
5"We have zero visibility"60+2-4 weeks rolling
61% vs 2% movement~2,300Not commercially viable
ThriveStack citedbyHow many runs you need before a brand number is usableStandard error of a brand detection rate (lower is better)SE 0.10: USABLE FLOOR1 run"essentially uninformative"0.3707 runs95% CI approx +/- 15.8 pts< 0.108 runs95% CI approx +/- 12.1 pts< 0.08Source: St. Gallen, arXiv 2604.07585 (Apr 2026). Only these three values arereported directly; the curve between them is not plotted here.
The three sample sizes St. Gallen reports directly. A single-run reading carries almost four times the error of a seven-run one.

To build a panel big enough for stable results, start with Prompt Research in ThriveStackcitedby.

07 - The volatility clock

AI visibility monitoring across four horizons of volatility

Volatility in AI answers comes in four kinds, nested inside each other, and each needs a different fix. Conflating them is how teams end up celebrating a model rollout as a content win.

ThriveStack citedby
Four speeds of change in AI answers, and how to handle each
What changes, how fast, and what it does to your number. Fastest at the top.
  1. 1
    Seconds
    80
    unique texts from 1,000 runs
    Sampling and batch noise
    Same prompt, same second, different answer. 1,000 greedy completions produced 80 unique texts.
    Fix: Repeat the run.
  2. 2
    Same day
    9.2%
    exact URLs shared in one day
    Retrieval churn
    Three runs of AI Mode on one day shared just 9.2% of exact URLs; 21.2% of queries shared none.
    Fix: Spread runs across the day.
  3. 3
    Days to weeks
    ~70%
    of the time, AI Overviews change between checks
    Index and answer drift
    AI Overviews change ~70% of the time between observations and persist about 2.15 days.
    Fix: Use 2 to 4 week rolling windows, never daily deltas.
  4. 4
    Months
    Reset
    with each model rollout
    Model version resets
    A new model reshuffles the board.
    Fix: Annotate, do not celebrate.
Sources: Thinking Machines Lab (2025); SE Ranking (10,000 keywords, 2025); Ahrefs (43,000+ keywords, Nov 2025); Semrush (1,500 keywords, 2024).
Each horizon has its own cause and its own remedy. Reading a monthly problem with a daily instrument is the most common error in AI visibility reporting.
08 - Collection method

API or interface: what your AI visibility numbers actually capture

Measuring through an API and measuring through the consumer interface answer different questions, and the gap is large enough to change your conclusions.

APIs are fast, cheap, repeatable and hold variables still. They also query a developer surface that often runs without live web search, without personalisation, without interface modules, and sometimes on a different default model than the one your buyers use. Interface capture reflects what a customer actually sees, at the cost of noise and expense.

Vendor-run comparisons put the gap at roughly 24% brand overlap and 4% source overlap between API and interface results, with API responses averaging 406 words against 743 for the interface, and API capture missing brand mentions around 8% of the time.

09 - What to do

Share of voice in AI search: measure like a pollster instead of a rank tracker

Everything above points to one shift: stop treating AI answers as a leaderboard to be scraped and start treating them as a population to be sampled. Pollsters solved this problem decades ago. The tooling in this category has mostly not caught up.

  • Kill rank as a headline metric. Report share of voice and appearance frequency. The data does not support anything finer.
  • Put a Wilson interval on every number. Suppress or grey out any rate whose interval crosses the threshold a decision depends on.
  • Set a sampling floor and publish it. Seven runs per prompt per engine minimum, aggregated over two to four weeks.
  • Segment by engine, always. With 11 to 17% cross-engine citation overlap, a blended AEO score averages four unrelated distributions into one meaningless figure.
  • Pin the model version and collection date to every datapoint, and mark rollouts on the chart.
  • Treat sub-5% rates as a presence flag rather than a trend line. Movement inside the noise band does not count.
  • Publish the methodology. Estimate, interval, effective sample, design, engine, window. Six fields. It is the cheapest trust moat available in this category.
10 - The counterargument

Is an AI visibility audit worth running if the data is this noisy?

Yes, but for a different reason than most vendors claim. The honest case against AI visibility tracking is strong and worth stating properly.

  • Semrush found AI-referred visitors converting at 4.4x the rate of organic (June 2025).
  • Similarweb measured 2.15x in 2026 cross-site e-commerce data, at 11.4% against 5.3%.
  • Ahrefs' own first-party data found 0.5% of visitors came from AI search but drove 12.1% of signups, a 23x conversion rate.
  • Semrush and Kevin Indig found only 15.2% of 1,094 ChatGPT topic categories had a clear owner. The other 85% remain contestable.

The synthesis: Visibility is measurable if you sample properly. Rank is not measurable at all. And the number that survives every methodological objection is revenue: did an AI-referred session become a customer. That question needs no confidence interval, because you are counting outcomes rather than estimating a proportion.

11 - Putting it to work

What to look for in an AI visibility tool: the ThriveStackcitedby loop

Everything above is a measurement design problem. Here is the operating cadence that solves it in practice.

The shift is from audit to loop. A one-off scan inherits every problem in this report: one run, one day, one engine, no interval. A loop run on a cadence accumulates runs against a stable prompt panel, so the confidence interval narrows over time while you are also closing the content gaps the data exposes. Run these as controlled tests with our AEO experimentation roadmap.

ThriveStack citedbyMeasure AI visibility in a loop you repeatEach pass adds runs, so your number gets more certain01TopicsReview what youmonitor02PromptsReview promptlanguage03GapsCitations andcompetitor share04PublishOwned and earnedmediaRE-MEASURE: DID THE GAP CLOSE?Each pass adds runs to the same prompt panel, so the interval narrowswhile the content gap closes.
The cadence that turns a noisy one-off reading into a trend you can defend.

Set an AI visibility benchmark you can defend

ThriveStackcitedby samples across ChatGPT, Perplexity, Gemini, Claude, Copilot and AI Overviews, then ties AI referrals to first-touch revenue. Find the prompts worth tracking in Prompt Research, then keep them in Manage Prompts so every pass adds runs to the same panel.

Related Concepts & Glossary Definitions

Explore foundational concepts and research on AI visibility, non-determinism, and revenue attribution:

Frequently asked questions

AI visibility measurement - FAQ

Why does ChatGPT give different answers to the same question?

Two reasons stack. Temperature and top-p sampling select each token probabilistically, so variation is by design. On top of that, inference kernels are not batch-invariant: server load changes the batch your request joins, which changes the arithmetic and the output. Thinking Machines Lab showed that even at temperature 0, 1,000 identical prompts produced 80 unique completions. Retrieval, session memory and model rollouts add further variance.

How many times should I run a prompt to measure AI visibility?

At least seven runs per prompt per engine per day. Research from the University of St. Gallen found per-brand detection-rate standard error falls below 0.10 at seven runs and below 0.08 at eight, while a single run carries an error of 0.370 and is essentially uninformative. Aggregate over a two to four week rolling window. Broad consumer categories need 60 to 100 runs, the range SparkToro used.

Is a 1% brand mention rate meaningful?

Not on its own. One mention in 100 runs carries a 95% Wilson confidence interval of 0.2% to 5.5%, so the true rate could be twenty-five times higher than observed. Distinguishing a genuine 1% rate from a genuine 2% rate at 95% confidence with 80% power requires roughly 2,318 runs per condition, which no commercial tool performs. Report sub-5% rates as a band rather than a point estimate.

Does zero mentions mean I have no AI visibility?

No. The rule of three puts the 95% upper bound on a true rate at roughly 3/n when you observe zero events. Zero mentions across 20 runs is consistent with a true rate as high as about 16%. You need around 60 runs before the claim that a brand has no AI visibility is statistically defensible.

Can I trust an AI visibility tool that reports my ranking position?

Treat it with heavy scepticism. SparkToro and Gumshoe found fewer than 1 in 1,000 repeat runs produced the same brands in the same order, which means position is close to pure noise. Rand Fishkin's conclusion was that any tool giving a ranking position in AI is unreliable. Appearance frequency and share of voice across many runs remain valid; ordinal rank does not.

Why do ChatGPT and Perplexity cite completely different sources?

They read different indexes and weight them differently. Profound analysed 100,000 identical prompts and found only 11.0% of cited domains appeared in both engines. Wikipedia accounts for roughly 47.9% of ChatGPT's top-ten source share while Reddit accounts for roughly 46.7% of Perplexity's. This is why a single blended AI visibility score across engines is misleading.

Should AI visibility be measured through APIs or the chat interface?

They answer different questions. APIs are cheap and repeatable but query a developer surface that often lacks live search, personalisation and interface modules. Interface capture reflects what buyers actually see. Vendor comparisons report roughly 24% brand overlap between the two. Whichever you choose, publish which surface, model version, geography and session state produced each number.

What is the right way to report AI visibility to a leadership team?

Report an appearance rate with a 95% confidence interval, the effective sample, the design, the engine and the time window. For example: 34% mention rate (95% CI 29-39%), n=280 effective, 200 prompts x 3 runs, ChatGPT, 6-12 July. Then pair it with first-touch revenue attribution, since counted conversions need no confidence interval.

Related Glossary Specs & Myth Fact-Checks

Explore definitions and statistical fact-checks on AI visibility tracking and non-determinism:

Sources & References

  1. Horace He et al., Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference" (10 Sept 2025)
  2. Rand Fishkin & Patrick O'Donnell, "AIs are highly inconsistent when recommending brands or products" (28 Jan 2026)
  3. Schulte, Bleeker & Kaufmann, University of St. Gallen, "Don't Measure Once: Measuring Visibility in AI Search", arXiv 2604.07585 (Apr 2026)
  4. Atil et al., "Non-Determinism of Deterministic LLM Settings", arXiv 2408.04667 (2025)
  5. SE Ranking, "AI Mode Research: Sources, Volatility & Differences" (Aug 2025)
  6. Ahrefs, AI Mode vs AI Overviews citation study (Dec 2025)
  7. Search Engine Journal, "AI Recommendations Change With Nearly Every Query" (Jan 2026)