Most AI visibility problems are not mysterious. They come from a short list of mistakes that show up in audit after audit: a robots.txt rule nobody meant to write, a homepage that says nothing, a report built from one prompt on one afternoon. Here are the twelve we see most often, and what to do instead.
1. Blocking the search bots by accident
This is the most common and the most damaging. A team blocks GPTBot to keep content out of model training, then assumes ChatGPT search is handled too. It is not. According to OpenAI's documentation, blocking GPTBot does not affect ChatGPT search. The crawler that matters for search visibility is OAI-SearchBot, and sites opted out of OAI-SearchBot are not shown in ChatGPT search answers.
The same confusion exists elsewhere. Anthropic separates ClaudeBot (training), Claude-User (fetching pages when a user asks a question) and Claude-SearchBot (indexing for search). Perplexity separates PerplexityBot (surfacing and linking sites in search results) from Perplexity-User (fetching pages for user questions, which generally ignores robots.txt because the user requested it).
A blanket Disallow: / for anything with "bot" in the name can quietly remove you from several answer engines at once while leaving you fully indexed in Google. Check the rules per crawler, not per mood.
A robots.txt detail people get wrong
Under RFC 9309, a crawler follows the most specific group that names it and ignores the * group in that case. So if you have a User-agent: * block and a separate User-agent: OAI-SearchBot block, the search bot obeys only its own group. Within a group, the longest matching rule wins. And remember that robots.txt expresses intent, not access control. It does not stop a crawler that chooses to ignore it.
2. Assuming robots.txt changes take effect immediately
OpenAI says robots.txt changes can take about 24 hours to take effect. Perplexity says up to 24 hours. If you unblock a bot on Monday morning and check for citations on Monday afternoon, you may be looking at stale behavior. Give it a day before drawing conclusions.
3. Never checking whether /llms.txt exists
Whether or not you have decided to publish an /llms.txt file, you should know whether one is live. It is a small thing that takes thirty seconds to confirm, and it is easy to forget after a site migration.
Our free AI crawler checker reads a site's robots.txt and reports, bot by bot, whether 16 crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot are allowed, partly restricted or blocked, and whether /llms.txt exists. Be clear about its limits: it does not see server logs, traffic, visits or CDN and firewall blocks. If a bot is allowed in robots.txt but stopped at the edge, the checker will not catch that.
4. Shipping JavaScript-only content
If your product descriptions, pricing, or comparison pages only render after client-side JavaScript runs, you are gambling on every crawler rendering the page the way a browser does. Some do a good job. Some do not. You usually cannot tell from the outside which is which.
The fix is unglamorous: make the substantive text present in the initial HTML. Server-render the parts that matter. Test by fetching the page with JavaScript disabled and reading what comes back. If the answer to "what does this company do" is not in that response, assume a retrieval system may miss it too.
5. A homepage that says nothing specific
"We help businesses unlock their potential" tells a model nothing it can repeat. When someone asks an assistant for a tool in your category, the answer is assembled from concrete claims: what you do, who it is for, what it costs, where you operate, what you integrate with.
Write the first two sentences of your homepage so they could be lifted verbatim into an answer and still be true and useful. Then do the same for your top five pages. Vague positioning is not a branding choice in this context; it is a retrieval problem.
6. Tracking a single prompt
One prompt, checked once, is an anecdote. AI answers vary by phrasing, by session, by model version, and by what the system retrieved that day. A team checks "best CRM for small business," sees their name, declares victory, and stops looking.
Build a small set of prompts that reflect how buyers actually ask: category questions, comparison questions, problem-first questions ("how do I stop missing invoices"), and brand questions. Track the set over time. Watch which competitors appear alongside you, because that is the more stable signal.
7. Treating a screenshot as data
A screenshot proves that one answer existed at one moment. It does not tell you frequency, trend, or which sources the answer drew on. If you are reporting to a founder or a client, a folder of screenshots invites the obvious question: how often?
Record the prompt, the date, the answer, and whether you were mentioned. Even a simple spreadsheet beats screenshots, because it lets you see movement. If you cannot say how often you appear across a defined prompt set, you do not yet have a measurement.
8. Buying mentions or seeding fake reviews
Paying for listicle placements, stuffing forums with planted recommendations, or generating fake review volume is a short-term trade with a long tail. Answer engines pull from sources that carry some signal of trust, and manufactured consensus tends to look manufactured once you read more than one page of it.
The durable version of this work is boring: get cited in places that already have authority in your category, publish material people link to on purpose, and make sure your own pages state facts clearly enough to be quoted.
9. Keyword-stuffed FAQ blocks
An FAQ section that repeats "best project management software for agencies" six times in eight questions helps nobody. Models are summarizing, not matching strings. Write the questions a real prospect would type, and answer them in complete sentences with specifics.
If two questions have the same answer, merge them. If a question exists only to hold a keyword, delete it.
10. Ignoring the pages that actually get quoted
Teams optimize the homepage and forget the pages that answer engines actually pull from: comparison pages, pricing pages, integration docs, help center articles, and third-party review profiles. Those pages tend to answer specific questions, which is exactly what gets retrieved.
Audit your top twenty pages by how directly each one answers a question someone would ask an assistant. Then fix the ones that ramble.
11. Assuming Google rankings carry over automatically
Google-Extended controls whether content is used to train future Gemini models and for grounding in Gemini apps and the Vertex AI Gemini API. It does not affect inclusion in Google Search and is not a ranking signal. Separately, AI Overviews are part of Google Search, so they are governed by normal Googlebot crawling and indexing.
The practical implication: a strong Google presence is a real advantage, but it is not a guarantee of appearing in AI answers, and blocking Google-Extended will not remove you from Search. Treat AI visibility as its own channel with its own diagnostics.
12. Not knowing why a competitor gets named
When a competitor shows up in answers and you do not, the cause is usually one of the items above: they are crawlable where you are not, their pages state facts plainly, or they are cited in sources the model trusts. Guessing wastes weeks.
Start by comparing the two sites on the mechanical things: bot access, rendered content, and how directly each page answers the question. Why AI Recommends Your Competitor Instead of You walks through that comparison in order.
A quick self-check
| Mistake | Fast test |
|---|---|
| Accidental bot blocking | Check robots.txt per crawler |
| JS-only content | Fetch with JS disabled |
| Vague homepage | Could a sentence be quoted verbatim? |
| Single-prompt tracking | Do you have a prompt set? |
| Screenshot reporting | Can you state a frequency? |
Run the crawler check on your own domain today, then fetch your homepage with JavaScript disabled and read the first paragraph. Those two steps catch more problems than any dashboard will.