AI crawler robots.txt checker

How to read the result

Every crawler lands in one of three lanes, and a block means something different in each:

• Search. The crawler builds the index an AI engine answers from. Blocking it is how sites vanish from ChatGPT search, Claude and Perplexity citations. A search-lane block is almost always a mistake unless you mean to leave that engine entirely.
• User fetch. The agent fetches a page live because a person asked about it. Vendors disagree on whether robots.txt binds these: Anthropic says Claude-User honours it, OpenAI says the rules “may not apply” to ChatGPT-User, and Perplexity says Perplexity-User “generally ignores” them. The checker shows the verdict your file would give and quotes each vendor’s position next to it.
• Training. Blocking opts your content out of future model training. For most sites this is the low-cost block: it does not remove you from that vendor’s answers, because answers come from the search lane.

The reasoning behind the lanes, with the citation-share evidence, is in AI crawler decisions.

What the checker catches

• The inheritance trap. A crawler with its own User-agent group ignores everything under User-agent: *. The classic case gives OAI-SearchBot an Allow: / group and silently lets it into /wp-admin/ and every sort-parameter URL. The checker lists every * disallow that no longer applies.
• Tokens that match nothing. User-agent: ChatGPT or User-agent: Claude block no real crawler. No vendor publishes those tokens, so the rule gives a false sense of control.
• Search-lane blocks, including a Googlebot block, which removes a site from Google Search and its AI features at once. Google publishes no token that blocks AI Overviews alone.
• Rules outside any group, which every crawler ignores, and a missing Sitemap: line.

How matching works

The checker applies the rules of RFC 9309, the robots.txt standard. User-agent matching is case-insensitive. Every group that names a crawler is combined, and the * group is used only when none does. The longest matching path rule wins, and Allow wins a tie. * matches any run of characters and a trailing $ anchors the end of the URL. Paths are case-sensitive, so /Blog/ and /blog/ are different rules.

Like Google’s open-source parser, it reads only the leading letters, hyphens and underscores of a User-agent value. That makes GPTBot/1.2 equivalent to GPTBot. Parsers that compare the whole value exactly would not match it, so write the bare token.

What it cannot tell you

robots.txt is a request, not a lock. The checker reports what a compliant crawler should do, not what arrives at your server. To see which bots actually visit, check your server or CDN logs for these user-agents, and verify them against each vendor’s published IP ranges, because some crawlers impersonate others. Common Crawl, for one, warns of “crawlers falsely identifying themselves as CCBot” (Common Crawl).

Every crawler it checks

Taken from each vendor’s own documentation, last verified September 27, 2026. Vendors add and rename tokens. If one here is out of date, report it.