# Run the Same AI Test the Same Way or It Proves Nothing
URL: https://www.getrankonmap.com/blog/standardising-an-llm-citation-test
Published: 2026-10-30

An AI visibility test only produces data if it's run the same way every time:
the same questions, the same platform with the model version written down, the
same location, a set number of runs, and the same fields logged on each run.
Anything else is a folder of screenshots. As of September 2026 there's an
extra problem. The model underneath a chatbot changes often, and you can
only see part of what it read. On **15 August 2026** a practitioner on r/bigseo
asked how others standardise tests of which domains models cite. They wanted
"a repeatable protocol", and actionable data rather than a spreadsheet of random observations. The thread drew 17 replies.

> I want actionable data not just spreadsheet full of random observations.
>
> — r/bigseo, August 2026

```
  ONE RUN = ONE ROW, SAME FIELDS EVERY TIME
  date · platform · model version · location · question (verbatim)
  run # (1..N) · businesses/brands named · named as a pick? (y/n)
  pages cited · retrieval queries, if the platform shows them
  ──────────────────────────────────────────────────────────────────
  N runs per question → share of runs naming you, and how stable it is
```

## The Model Changed Six Times in Our Own Archive

We looked at the model version recorded against each ChatGPT answer in our
cached archive of local SEO and Business Profile questions, answered between
October 2025 and September 2026. The **114** answers came from **six** model
versions: gpt-5, 5-1, 5-2, 5-3, 5-5 and 5-6. A test that doesn't log the
version can't tell a real change in visibility from a model change. When your
numbers move, the model is the first thing to check, before anyone
changes the client's site.

## You Only See Part of What the Model Read

The practitioner wanted to log "what the model retrieved versus what it
ignored". Our archive shows how far you can get with that. For **75** ChatGPT
answers where the search results were recorded alongside the cited sources,
the median answer cited domains from about **12%** of the retrieved domains.
And in the median answer, **half** of the cited domains didn't appear in the
recorded results at all. So the logged retrieval is a partial view. Record it
when the platform exposes it, but don't treat "retrieved and not cited" as a
complete list.

## Location and Wording Are Test Variables, Not Details

For local questions, the answer can depend on where the question is asked from,
and small wording changes can shift which businesses appear. Assistants also
rewrite the customer's question before searching. Our archive shows questions
like "how do i advertise my home salon?" becoming longer queries with category
terms and a year. So the question set should use customers' own wording, kept
exactly the same between runs, and the location should be fixed and recorded.

## Runs Per Question Turn Noise Into a Share

The same question can return different businesses on different runs. One run
gives a yes or a no. Ten runs give a share, such as named in 4 of 10, and a
stability figure, such as the same top pick in 7 of 10. Keep the run count the
same every time you test, so this month's share can be compared with last
month's.

```
  WEEKLY PROTOCOL — one client, one town
  questions    8 (5 buying, 3 informational), verbatim, frozen
  platforms    each one you report on, version logged
  runs         5–10 per question per platform
  output       share named · share named as the pick · stability
```

## Keep the Test Small Enough to Repeat

A protocol that's too big to run every time will be skipped, and a skipped
month breaks the series. Eight questions, ten runs, and two platforms comes to
160 runs per client per round. That's too many to do by hand weekly for more
than one client. Cut the question set or run monthly before you cut the run
count, because the run count is what makes the share mean something.

One more rule: don't change the questions because the results look bad. A
question set rewritten after a poor month produces a better-looking number that
can't be compared with anything before it.

## Setting Up the Protocol

1. **Freeze the question set** in customers' words, and version it if you ever
   change it.
2. **Log the platform and model version** on every row.
3. **Fix and record the location** for local questions.
4. **Run each question the same number of times**, and report shares, not
   single results. What to report is covered in
   [count who gets recommended, not who gets cited](/blog/what-to-measure-for-aeo-beyond-citations).
5. **Keep a map pack grid alongside it** for the same service and town. See
   [AI answers and the map pack pick businesses differently](/blog/ranks-in-ai-but-not-in-the-map-pack),
   and for the starting point,
   [local businesses invisible to ChatGPT](/blog/local-businesses-invisible-to-chatgpt).

## Sources

**Our own data.** Collected for this page and not published elsewhere.

- **Model versions**: 114 recorded ChatGPT answers to local SEO and Business
  Profile questions (US, English), October 2025 to September 2026, from the
  DataForSEO AI-optimisation archive pulled 20 September 2026. Six model
  versions. Counted 21 September 2026.
- **Retrieved vs cited**: for the 75 answers with recorded search results,
  compared by domain. Median share of retrieved domains cited ≈ 12%. Median
  share of cited domains present in the recorded results = 50%.

**Cited externally.** Accessed September 2026.

| source | what it supports |
|---|---|
| [r/bigseo, 15 August 2026](https://www.reddit.com/r/bigseo/comments/1vp0bgk/how_are_you_standardizing_your_tests_for_llm/) — standardising LLM citation tests | the case; "repeatable protocol"; retrieved vs ignored; quoted phrases; 17 replies |