Run the Same AI Test the Same Way or It Proves Nothing

An AI visibility test only produces data if it's run the same way every time: the same questions, the same platform with the model version written down, the same location, a set number of runs, and the same fields logged on each run. Anything else is a folder of screenshots. As of September 2026 there's an extra problem. The model underneath a chatbot changes often, and you can only see part of what it read. On 15 August 2026 a practitioner on r/bigseo asked how others standardise tests of which domains models cite. They wanted "a repeatable protocol", and actionable data rather than a spreadsheet of random observations. The thread drew 17 replies.

I want actionable data not just spreadsheet full of random observations.

r/bigseo, August 2026

  ONE RUN = ONE ROW, SAME FIELDS EVERY TIME
  date · platform · model version · location · question (verbatim)
  run # (1..N) · businesses/brands named · named as a pick? (y/n)
  pages cited · retrieval queries, if the platform shows them
  ──────────────────────────────────────────────────────────────────
  N runs per question → share of runs naming you, and how stable it is

The Model Changed Six Times in Our Own Archive

We looked at the model version recorded against each ChatGPT answer in our cached archive of local SEO and Business Profile questions, answered between October 2025 and September 2026. The 114 answers came from six model versions: gpt-5, 5-1, 5-2, 5-3, 5-5 and 5-6. A test that doesn't log the version can't tell a real change in visibility from a model change. When your numbers move, the model is the first thing to check, before anyone changes the client's site.

You Only See Part of What the Model Read

The practitioner wanted to log "what the model retrieved versus what it ignored". Our archive shows how far you can get with that. For 75 ChatGPT answers where the search results were recorded alongside the cited sources, the median answer cited domains from about 12% of the retrieved domains. And in the median answer, half of the cited domains didn't appear in the recorded results at all. So the logged retrieval is a partial view. Record it when the platform exposes it, but don't treat "retrieved and not cited" as a complete list.

Location and Wording Are Test Variables, Not Details

For local questions, the answer can depend on where the question is asked from, and small wording changes can shift which businesses appear. Assistants also rewrite the customer's question before searching. Our archive shows questions like "how do i advertise my home salon?" becoming longer queries with category terms and a year. So the question set should use customers' own wording, kept exactly the same between runs, and the location should be fixed and recorded.

Runs Per Question Turn Noise Into a Share

The same question can return different businesses on different runs. One run gives a yes or a no. Ten runs give a share, such as named in 4 of 10, and a stability figure, such as the same top pick in 7 of 10. Keep the run count the same every time you test, so this month's share can be compared with last month's.

  WEEKLY PROTOCOL — one client, one town
  questions    8 (5 buying, 3 informational), verbatim, frozen
  platforms    each one you report on, version logged
  runs         5–10 per question per platform
  output       share named · share named as the pick · stability

Keep the Test Small Enough to Repeat

A protocol that's too big to run every time will be skipped, and a skipped month breaks the series. Eight questions, ten runs, and two platforms comes to 160 runs per client per round. That's too many to do by hand weekly for more than one client. Cut the question set or run monthly before you cut the run count, because the run count is what makes the share mean something.

One more rule: don't change the questions because the results look bad. A question set rewritten after a poor month produces a better-looking number that can't be compared with anything before it.

Setting Up the Protocol

1. Freeze the question set in customers' words, and version it if you ever change it.
2. Log the platform and model version on every row.
3. Fix and record the location for local questions.
4. Run each question the same number of times, and report shares, not single results. What to report is covered in count who gets recommended, not who gets cited.
5. Keep a map pack grid alongside it for the same service and town. See AI answers and the map pack pick businesses differently, and for the starting point, local businesses invisible to ChatGPT.