AI crawler checker
Enter your domain. We read your robots.txt and tell you, bot by bot, whether AI search engines and AI training crawlers can access your site. We also check for an llms.txt file.
FAQ
Should I block AI training crawlers?
It's your call. Blocking training bots (GPTBot, CCBot, Google-Extended) keeps your content out of model training but has little effect on AI search answers. Blocking search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) removes you from those answers.
Does Google-Extended affect Google Search or AI Overviews?
No. Google-Extended controls use for Gemini training and grounding. AI Overviews are served from the regular Googlebot index.
What is llms.txt?
A proposed plain-text file at /llms.txt that gives AI assistants a short, curated map of your site. It's optional and not yet a standard every assistant reads.
Latest guides
All guidesThe AI Visibility Audit: A 30-Point Checklist
A 30-point AI visibility audit checklist for marketers: crawler access, entity clarity, answerable content, mentions, and measurement, with pass/fail tests.
TechnicalSchema Markup for AI Search: Which Types Are Worth Your Time
Schema markup for AI search: what Organization, Product, LocalBusiness, FAQPage and Article tell machines, with JSON-LD examples and Google's content-match rule.
MeasurementShare of Voice in AI Answers: How to Calculate It
Learn how to calculate AI share of voice across prompts and engines, with a worked example and a simple formula you can run in a spreadsheet.
Want this every week, across ChatGPT, Gemini and Perplexity?
PromptRanking tracks your brand on every platform, shows the competitors named instead of you, and writes content to close the gap.
Start free