robots.txt tester
Is this URL blocked? See the exact rule and line that decides it, using the same matching rules Google documents (most specific user-agent group, longest matching rule, * and $ wildcards).
How does Google decide if a URL is blocked?
Google picks the group whose User-agent line most specifically matches its crawler (e.g. Googlebot beats *). Within that group, the rule with the longest matching path wins; if an Allow and a Disallow tie, Allow wins. * matches any sequence of characters and $ anchors the end of the URL. Matching is case-sensitive.
Does blocking a page in robots.txt remove it from Google?
No. It stops crawling, not indexing. A blocked URL can stay indexed (often without a description) if other pages link to it. To remove a page, let Google crawl it and serve noindex — which Google can't see while robots.txt blocks it.
Is my site blocking ChatGPT or Perplexity?
Pick GPTBot or ClaudeBot above to test a single bot, or run the AI visibility checker to see every AI search and training crawler at once, plus CDN blocks and JavaScript-only content. To write new rules, use the AI robots.txt generator.
What happens if robots.txt returns an error?
A 4xx (like 404) means "no rules": everything is allowed. A 5xx or timeout makes Google pause crawling the site, so a broken robots.txt can be worse than a missing one.