Robots.txt for AI Crawlers: What to Block, What to Allow, and What 63 Real Sites Do

This post contains affiliate links. If you buy something through one of them, Veravix may earn a commission at no extra cost to you.

Your robots.txt file is where you tell AI companies whether their crawlers may read your site, and the answer is not one switch but at least three separate ones: training, search, and requests made on behalf of a person. When I fetched the robots.txt of 63 well-known sites on 8 November 2026, only 17 mentioned any AI crawler at all. Just 2 of them, the BBC and the New York Times, shut out OpenAI’s training crawler completely. The other 46, including WordPress hosts and plugin makers, said nothing, which means every AI crawler is welcome by default.

This guide explains what each AI crawler token does according to the companies’ own documentation, what the survey showed, and gives copy-paste rules for the two choices most site owners actually face: stay visible in AI search while opting out of training, or opt out of everything. It also covers where WordPress keeps robots.txt, because the default file is easy to miss.

What robots.txt can and cannot do for AI crawlers

Robots.txt is a plain text file at the root of your domain, such as https://example.com/robots.txt. A crawler fetches it first and is expected to follow the rules in the group that names it. The format is a formal internet standard, RFC 9309, and the same document states plainly: “These rules are not a form of access authorization.” It is a request that honest crawlers follow, not a lock. A crawler that ignores it still receives your pages.

Two parts of the standard matter for AI rules. A crawler obeys the group that names it, and only falls back to the User-agent: * group when no group names it. And when two rules conflict, the most specific match wins, meaning the one with the longest path. So a rule aimed at GPTBot is the one OpenAI’s training crawler reads, no matter what the general group says.

The companies that publish their rules say their crawlers follow robots.txt, with one important exception covered next. Whether every smaller crawler honours it is outside what a robots.txt file can enforce; if you need a hard block, that happens at the server or firewall, and How to Check Your Website’s Security Headers, and What 63 Real Sites Actually Send covers the server-side layer that sits next to this one.

The three kinds of AI crawler, and why one rule is not enough

The vendors have split their crawling into separate user-agent tokens, each with a different job. Blocking the wrong one is the most common mistake, because blocking a training crawler does not remove you from the same company’s search product.

Token	Company	What it is for	Does robots.txt control it?
GPTBot	OpenAI	Crawling content that may be used to train its generative AI models	Yes. Disallowing it says your content should not be used in training
OAI-SearchBot	OpenAI	Surfacing sites in ChatGPT’s search features	Yes. Opted-out sites will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI says changes can take about 24 hours
ChatGPT-User	OpenAI	Visits a page when a person asks ChatGPT or a custom GPT about it	Not reliably. OpenAI says “robots.txt rules may not apply” because a user started the action
ClaudeBot	Anthropic	Collecting web content that could contribute to model training	Yes. Disabling it excludes future material from training datasets
Claude-SearchBot	Anthropic	Improving the quality of search results	Blocking it stops indexing for search, which may reduce your visibility there
Claude-User	Anthropic	Fetching a page when a person asks Claude a question	Anthropic says blocking it prevents retrieval for user queries; its examples only show ClaudeBot
Google-Extended	Google	A control token for whether crawled content may be used for training Gemini models and for grounding	Yes, but it is not a crawler. Google fetches with its normal user agents and uses this token only as a switch. Google states it does not affect inclusion in Google Search or act as a ranking signal

The practical reading is this. Training tokens (GPTBot, ClaudeBot, Google-Extended) decide whether your writing feeds future models. Search tokens (OAI-SearchBot, Claude-SearchBot) decide whether you can be listed in those companies’ answer engines. User-triggered tokens (ChatGPT-User, Claude-User) cover a person pasting or asking about your page, and OpenAI itself says robots.txt may not stop that. Googlebot, the crawler behind regular Google Search, is a different token entirely and none of the rules below touch it.

What 63 real sites put in robots.txt

I requested /robots.txt from 69 well-known sites in the WordPress, hosting, ecommerce and media space, plus this site and its sister site. Six refused automated requests or had no file (Bluehost, Cloudways, SiteGround, WPBeginner, Debian and Hismile), which leaves 63 readable files. I counted a site as naming a crawler only when it had a rule group for that exact token.

Crawler token	Sites naming it (of 63)	Blocking the whole site
GPTBot	13	2
ClaudeBot	12	3
PerplexityBot	12	3
ChatGPT-User	10	2
OAI-SearchBot	9	2
Bytespider	9	5
Google-Extended	8	2
CCBot	8	3
Claude-SearchBot	8	3
Claude-User	7	2
Applebot-Extended	6	3

Four things stand out.

• Silence is the norm. 46 of the 63 sites, including Shopify, Stripe, WooCommerce, Yoast and Semrush, name no AI crawler. Neither this site nor Repairpicks did either, before this guide: both served the default WordPress file, which only restricts /wp-admin/ and our affiliate redirect folder.
• Full blocks are rare and come from publishers. The BBC and the New York Times block almost every token in the table. The Guardian blocks the Claude crawlers, Perplexity, CCBot and others but does not name GPTBot. Content publishers have the clearest reason to protect text; most business sites do not.
• Many named rules are partial, not blocks. Eight of the 13 sites naming GPTBot only disallow certain paths for it, for example internal search or account pages, as GitHub, Hostinger, Klaviyo and Ubuntu do. That is housekeeping, not an opt-out, and a count of “sites that block AI” that includes them overstates the case.
• A few sites welcome the crawlers on purpose. WordPress.org, CSS-Tricks and Cloudflare list several of them with nothing disallowed.

Six sites, including Cloudflare, Hostinger, Kinsta and WP Engine, also publish a Content-Signal line in robots.txt. That is a newer, non-standard way of stating preferences for search, AI input and AI training in one place. I would treat it as a statement of intent, because none of the vendor pages I read describe honouring it, and the strict way to say no is still a rule for the named token.

Which rule to use: stay in AI search, or opt out of everything

Pick based on what you want, not on a blanket “block AI” guide.

Opt out of training, stay eligible for AI search

This keeps your pages available to answer engines while telling the training crawlers to stay away. It is the combination most small publishers want, because the search crawlers are the ones that can send a visitor back.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Do not add OAI-SearchBot or Claude-SearchBot here, and leave the User-agent: * group alone.

Opt out of AI entirely

Add the search and user tokens as well:

User-agent: OAI-SearchBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: ChatGPT-User
Disallow: /

Remember what you give up. OpenAI says opted-out sites will not appear in ChatGPT search answers, and Anthropic says blocking its search crawler may reduce your visibility there. Remember too that ChatGPT-User may ignore the rule, so a page someone pastes into ChatGPT can still be read.

Common mistakes

• Blocking GPTBot and expecting ChatGPT search to drop you. It is a separate token; OpenAI documents the two settings as independent.
• Treating Google-Extended as a way to leave Google Search. It is not. Blocking it leaves Googlebot and normal Search untouched.
• Disallowing / in the * group. That blocks Googlebot too and removes you from search results.
• Waiting for an instant effect. OpenAI says search systems can take around 24 hours to adjust, and crawlers cache the file.

Where WordPress keeps robots.txt, and how to change it

If no physical robots.txt file exists in your site root, WordPress generates one on the fly. That is why many sites look as if they have no file to edit; this site served exactly the default output at the start of this survey: block /wp-admin/, allow admin-ajax.php, and point to the sitemap. You have three clean ways to change it:

1. Upload a physical file. Create robots.txt in your site root through your host’s file manager or SFTP. It overrides the generated one completely, so include the default WordPress rules in it too.
2. Use the robots_txt filter. A short snippet appends your AI rules to the generated output without touching files. Add it with a code snippets plugin, not into a theme file that an update will overwrite:

   add_filter( 'robots_txt', function ( $output, $public ) {
       $output .= "\nUser-agent: GPTBot\nDisallow: /\n";
       $output .= "\nUser-agent: ClaudeBot\nDisallow: /\n";
       return $output;
   }, 10, 2 );

3. Use an SEO plugin’s editor if you already run one. I do not run one on this site, so I cannot vouch for the individual plugins’ behaviour.

Do not confuse any of this with Settings, Reading, “Discourage search engines from indexing this site”. That checkbox changes how the site asks all search engines to treat it, which is a different decision from choosing which AI crawlers may read your content.

After saving, open https://yoursite.com/robots.txt in a private window, confirm your new groups appear and the sitemap line is still there, and check that the file returns status 200:

curl -sI https://yoursite.com/robots.txt | head -n 5

Each crawler reads the groups independently, so the order of groups does not matter; a blank line between groups keeps the file readable.

Do you need an llms.txt file too?

The same survey checked for /llms.txt, a Markdown file some sites publish as a guide for AI tools. 44 of the 63 sites have one, including Shopify, Stripe, WordPress.org and Yoast, and so do this site and Repairpicks. I could not find a statement in the OpenAI crawler page, the Anthropic crawler article or the two Google crawler pages I read that says those companies’ crawlers use it. OpenAI’s own documentation points to an llms.txt as an index of its docs, which is a use of the file, not a promise that its crawlers read yours. Treat llms.txt as an optional convenience for tools that look for it. It does not replace robots.txt and cannot block anything.

What is changing

Two newer signals are arriving, and both are stated in the vendors’ own pages. Anthropic says that since 30 October 2026 it recognises a machine-readable “no AI training” preference based on an IETF draft, and that a robots.txt rule naming ClaudeBot takes priority over that general signal. Google says it intends to support the same shared training-preference signal in early 2027. Until those are widespread, named rules in robots.txt remain the clearest way to say what you want, which makes the table above the part of this page worth bookmarking and rechecking, since vendors add and rename tokens.

A short routine to keep it right

1. Decide first: training only, or training and search.
2. Add the matching groups from the examples above and save.
3. Fetch /robots.txt in a private window and read it once.
4. Once a quarter, re-read the three vendor pages (OpenAI’s crawler page, Anthropic’s crawler article, Google’s common-crawlers page) for new or renamed tokens.

Robots.txt only covers crawlers that choose to listen. For the rest of the surface, How to Check Website Security Online, and What 68 Sites Showed on the Basics shows what a visitor or bot can see without any request for permission.

Checked against the vendors’ crawler documentation and a live fetch of 63 robots.txt files on 8 November 2026.