AEO Experimentation Roadmap:
How to Test AI Visibility and Prove Conversions

An AEO strategy needs experiments because AI answers change on every run, so a one-time audit mostly shows noise. In a SparkToro study of nearly 3,000 runs, the chance of getting the same brand list twice was below 1 in 100. Tests with controls and repeated runs separate real lifts from luck.

Interest is rising fast. Searches for “aeo playbook” grew from 10 to 90 a month between October 2025 and August 2026, and AI search optimization reached 1,600. Most advice online lists tactics. A roadmap says which tactic to run first and how to read the result.

Researchers at the University of St. Gallen reached the same view in April 2026. They argue that AI visibility is a distribution, so a single observation misleads. Our research on run-to-run variation covers the eight variables behind it.

Fishkin leads the SparkToro team behind that study. His point shapes this whole roadmap: judge changes on many runs rather than one answer.

An AEO framework for testing defines five parts before any change ships: a hypothesis, a unit of treatment, an observation window, one primary metric and a control. Skip one and the result cannot be trusted. Windows run 14 to 28 days, longer than the 7 days common in paid search tests.

The figure groups those five parts into three stages. Plan before launch, run during the window and read after it. The hypothesis names one change and one expected lift. The unit is a set of 5 to 10 similar pages, matched with an untouched control set.

The window is the part most teams get wrong. A single page may earn only a handful of citations a week, so short windows give thin data. StackMatix suggests repeating a cycle two to three times before trusting a directional claim. Our AEO guide explains what each change type targets.

Set up AI visibility tracking by building a prompt panel of 30 to 100 real buyer questions, then running each prompt at least 7 times per engine across 2 to 4 weeks. One run has a standard error of 0.370, which makes a single snapshot nearly useless for judging a test.

The rings show why. Exact lists almost never repeat, yet a brand can still appear in most answers. In one ChatGPT test, City of Hope showed up in 69 of 71 answers but ranked first in only 25. Presence is stable enough to test. Rank is not.

Run the panel on at least ChatGPT, Perplexity, Gemini and Google AI Overviews. Claiming zero visibility takes about 60 runs, because zero mentions in 20 runs fits a true rate as high as roughly 16%. Our prompt panel guide shows how to pick the prompts.

An experiment should move one primary metric, usually citation rate: cited prompts divided by tracked prompts. Share of voice, answer inclusion and passage lift are the backups. Rank is unstable inside AI answers, so skip it. Then follow the metric down the funnel to signups and revenue.

Pick the metric before launch and lock it. Share of voice divides your citations by your citations plus rivals’ citations in the same prompt set. Answer inclusion counts prompts where your brand appears at all. Passage lift counts new passages quoted after a change.

Traffic is the lagging stage. Conductor found AI referrals were 1.08% of website visits across 3.3 billion sessions in 2026, and ChatGPT sent 87.4% of them. That volume is small, so tie each experiment to signups as well as visits. The metrics guide for CMOs lists what to report upward.

GEO experiments run in four steps over 12 weeks: two weeks of baseline, four weeks for wave 1, two weeks for the readout and four weeks for wave 2. Each wave tests one change type on 5 to 10 pages and watches 28 days against an untouched control.

Pick change types from published evidence. The Princeton GEO paper reported that its methods can boost visibility by up to 40% in generative engine responses. Click Laboratory suggests three families to start with: structure (FAQ blocks, clear definitions, summaries), freshness (updated stats and dates) and entity work (author bios and internal links to hub pages).

Treat the cadence as a template. Keep the readout honest: keep winners, retire losers and write down the mechanism, so wave 2 builds on a reason rather than a hunch. Teams under ten people can run about three experiments a month, per Click Laboratory.

AEO measurement proves conversions by comparing a treated group with a matched control that you leave alone. Net lift equals the test change minus the control change. In MaxAEO’s sample data, the test group rose 12.8 points and the control rose 5.2, so the net lift is 7.6 points.

That example runs 35 days: 14 days of baseline, a launch on day 15, a 7-day lag buffer, a post window to day 35 and analysis on day 36. The noise floor is twice the control group’s daily standard deviation. With a deviation of 2.1 points the floor is 4.2, and the 7.6-point lift clears it. If a lift falls inside the floor, call it inconclusive.

Then join the lift to money. ChatGPT adds utm_source=chatgpt.com to its links, so tagged visits keep their source. Build a custom AI channel in GA4, as the GA4 AI traffic guide shows, and add a free-text “How did you hear about us?” question at signup. Report three layers: counted, asked and modeled. Our AI search revenue attribution research and UTM source guide cover the setup.

Studies disagree on how well AI visitors convert. Amsive found 4.87% for AI traffic against 4.60% for organic across 54 sites, a gap that was not significant, while Ahrefs reported a far larger gap on its own site. Measure your own data rather than borrowing a benchmark.

A good AEO playbook for next quarter holds three test waves, one named owner and a standing holdout. Teams under ten people can run about three experiments a month, at roughly 20% of one person’s week. Score each test, retire the losers and carry the winners into the next wave.

Start where your gap is biggest. The figure above maps four common situations to a first test. Find the row that sounds like you and read its note below:

1. You have no baseline yet: run a prompt panel. Pick 30 to 100 real buyer questions and run each one 7 or more times per engine. Without a baseline you cannot tell a real gain from noise. See the baseline section.
2. Pages get cited but signups do not follow: build an attribution join. Tag AI visits with UTMs and add a signup question such as “How did you hear about us?” Then match cited pages to signups. See the metrics section.
3. Rivals win the prompts you want: run a structure test. Add FAQ blocks and clear one-line definitions to the pages that target those prompts. Leave a matched set of pages unchanged as the control. See the holdout section.
4. Older pages lose citations over time: run a freshness test. Update the stats, dates and examples on a set of older pages. Compare them with similar pages you leave alone.

Setup is light: ThriveStack estimates about 30 minutes to set up and about 15 minutes per weekly run. Use an MCP and build an automated scheduler, so you spend less and less time on monitoring and experimenting.

Keep a log with the hypothesis, the pages, the dates and the result for every test. That log is the real asset. Pair it with our AI visibility playbook for the content side.