Ask three different people to check whether ChatGPT recommends a brand and you may get three different answers. Ask the same tool twice in one afternoon and the answer can still change. That variance is the first thing to understand before you build any AI visibility metrics, because most of the bad measurement practices in this space come from treating a single answer as a fact.
Why one screenshot proves nothing
AI answers are generated, not retrieved from a fixed index. The model samples from a distribution of plausible responses, and small differences in phrasing, context, or whatever the system retrieved that minute can push the answer toward a different brand. So when a founder sends you a screenshot showing their product named first, or a competitor named first, they are showing you one draw from a distribution. It is evidence that a mention is possible. It is not evidence of a rate.
The practical consequence: any number you report should come with the number of runs behind it. "Named in 1 of 1 runs" and "named in 34 of 50 runs" are different claims, and only one of them is worth putting in a deck.
Step 1: Fix the prompt set before you measure anything
You cannot track a moving target. The single biggest source of fake movement in AI visibility reporting is a prompt set that quietly changes between measurement periods. Someone adds a question, someone rewords another, and the week-over-week comparison is now comparing two different things.
Write your prompts down and freeze them:
- Store them in a file, not in someone's head. A spreadsheet or a text file in the repo both work. The point is that the exact strings are versioned.
- Record the exact wording, including punctuation. "best CRM for small law firms" and "What is the best CRM for a small law firm?" can produce different answers. If you change the wording, that is a new prompt, not an edit.
- Version the set. When you genuinely need to add or retire a prompt, note the date and keep reporting the old set as a separate series, or accept a break in the trend line and say so.
- Keep the set small enough to run consistently. Twenty to fifty prompts you actually run every week beats two hundred you run once.
A useful prompt set usually mixes a few shapes: category questions ("best X for Y"), comparison questions ("X vs Z"), and problem-first questions ("how do I solve Y"). Buyer questions are what matter, not questions about your own brand.
Step 2: Decide what you are counting
"Visibility" is not one metric. At minimum, separate these:
| Metric | What it counts | Why it matters |
|---|---|---|
| Mention rate | Share of runs where the brand appears at all | The floor. If this is low, nothing else matters yet |
| Position | Where the brand appears in the answer, when it appears | Distinguishes "listed" from "recommended first" |
| Share of voice | Brand mentions relative to all brand mentions in the answer | Shows whether you are gaining against specific competitors |
| Citation rate | Share of runs where a page you own is cited as a source | Connects answers to content you control |
| Sentiment or framing | How the brand is described | Catches "named, but with a caveat" |
Mention rate and position are the two people most often conflate. For example, suppose a brand shows up in most answers but is almost never the first recommendation. Reporting only "we're mentioned in most answers" hides that. Report both, and report position as a distribution (first, second, third, not listed) rather than an average, because averaging ranks produces a number nobody can interpret.
If you want a fast read on a single brand and category before you build the full tracker, our free AI visibility check generates three buyer questions for a brand and category, asks Perplexity with web search, and shows whether the brand is named, which competitors are named, and which sources are cited. It gives you three checks per day and does not track over time, so treat it as a snapshot to shape your prompt set, not as your measurement system.
Step 3: Get the sample size right
There is no universal correct sample size, but there is a correct way to choose one. Run a pilot: take one prompt and run it 20 times in a row. Count how often your brand appears. If the answer flips a lot, you need more runs per prompt to detect a real change. If it is stable, fewer runs will do.
A rough rule that works in practice: if you want to detect a change of about 10 percentage points in mention rate, you need on the order of dozens of runs per prompt, not three. If your budget only allows a handful of runs, be honest about it and report ranges rather than point estimates. "Named in 3 of 5 runs" is an honest statement. "60% mention rate" from the same five runs implies precision you do not have.
Also decide how you handle model and product changes. ChatGPT, Gemini, and Perplexity all ship changes on their own schedule. A jump in mention rate may be your content work or may be a model update. Log the date you ran each batch and note any visible product changes, so you can tell the two apart later.
Step 4: Compare week over week, not day over day
Daily measurement of a noisy signal produces noise. Weekly measurement with a fixed prompt set and a consistent run count produces something you can act on.
A workable cadence:
- Weekly: run the full frozen prompt set, same number of runs per prompt, same day of week.
- Weekly: record mention rate, position distribution, share of voice, and citation rate per prompt and in aggregate.
- Monthly: review the trend, not the last data point. Look for three consecutive weeks moving the same direction before you call it a change.
- Quarterly: revisit the prompt set itself. Buyer language drifts; your questions should drift with it, deliberately and with a version note.
When you report, always show the run count next to the rate. A chart with a line and no sample size is a chart that will embarrass you the first time someone asks how many runs it is based on.
Step 5: Separate the bot questions from the answer questions
Measurement gets confusing when people mix up two different things: what AI systems are allowed to fetch, and what they say. The vendor documentation is worth knowing here, and it is more specific than most people assume.
For OpenAI, OAI-SearchBot is what surfaces websites in ChatGPT search, and sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. GPTBot collects content for foundation model training, and blocking it does not affect ChatGPT search. ChatGPT-User handles user-initiated actions, and because those are user-initiated, robots.txt rules may not apply to it; it is not used to decide whether content appears in search. Robots.txt changes can take about 24 hours to take effect.
For Anthropic, ClaudeBot collects content that may be used for training, Claude-User fetches pages when a user asks a question, and Claude-SearchBot indexes content to improve search results. Anthropic says its bots honor robots.txt, support Crawl-delay, and that it publishes its IP ranges.
For Perplexity, PerplexityBot surfaces and links sites in search results and is not used to crawl content for foundation models, while Perplexity-User fetches pages for user questions and generally ignores robots.txt because the user requested it. Robots.txt changes may take up to 24 hours.
For Google, Google-Extended controls training and grounding for Gemini, does not affect inclusion in Google Search, and is not a ranking signal. AI Overviews are part of Google Search, so they follow normal Googlebot crawling and indexing.
Two things follow from this. First, if your visibility problem is "we are not being cited," check whether the relevant crawler is actually allowed before you rewrite anything. Second, robots.txt is a statement of intent, not access control, per RFC 9309: a crawler follows the most specific group that names it and ignores the * group in that case, and within a group the longest matching rule wins. Do not treat it as a security boundary.
Common ways teams fool themselves
- Reporting a single run as a rate. The most common error, and the easiest to fix.
- Changing prompts and calling it a trend. Freeze the set or break the series.
- Averaging positions. A mean rank hides whether you are first or absent.
- Ignoring citation rate. Being named without a source you own is harder to defend and harder to improve.
- Measuring only your own brand. Share of voice against named competitors is usually the number leadership actually wants.
- Comparing across tools without noting it. A rate measured on one assistant is not comparable to a rate measured on another, and mixing them into one line is misleading.
A next step you can do this week
Write down 20 buyer questions your sales team actually hears, freeze them in a file with today's date, and run each one 10 times on one assistant. Record mention rate, position, and whether a page you own was cited. That gives you a baseline with a known sample size, which is the only kind of baseline worth comparing against next week.