AI Crawlers and robots.txt: Deciding Allow or Block

A crawler's name usually hints at its job, but every company names things differently. At Anthropic, ClaudeBot collects training data, Claude-User visits a live page on behalf of a user, Claude-SearchBot evaluates content for search-quality answers.

On OpenAI's side, GPTBot crawls for model training while OAI-SearchBot feeds ChatGPT's search feature; a site can block GPTBot while still allowing OAI-SearchBot. Exactly which sub-tasks PerplexityBot splits across separate agents can vary by the situation; the current list always sits on that company's own crawler documentation.

A scene we saw at an e-commerce brand: the technical team blocked PerplexityBot entirely, 'just to be safe', without checking which sub-task it actually handled, when the only thing they wanted to avoid was the training side. The decision criterion is this: before blocking a bot, confirm exactly what it does from that company's own documentation rather than guessing from its name; that five-minute check can save months of lost visibility from an assumption that turned out wrong.

Blocking a training bot, GPTBot or ClaudeBot, stops that company from using a site's content in future model training. In exchange, the risk of never appearing as a source in that model's answers can rise too, since the same content often feeds both training and answer generation.

Blocking a search or user-facing bot, Claude-User or OAI-SearchBot, produces a different result: the site simply stops appearing in that assistant's live answers. For an e-commerce site that is usually an unwanted loss; for a subscription content site it can be a deliberate choice.

A scene we saw at a hosting provider: the technical team proposed blocking every bot outright to cut server load, while the marketing team wanted the brand to keep showing up in assistant answers. The decision criterion is this: blocking is a balance one team alone should not set, it needs someone weighing server load against visibility goals together; deciding on cost alone can quietly switch off the visibility that was actually wanted.

Blocking or allowing a crawler is a few lines of rule inside a robots.txt file.

• Open the robots.txt file at the site's root (for example, sitename.com/robots.txt).
• For a bot to block, write 'Disallow: /' under a 'User-agent: GPTBot' line.
• For a bot to allow, open a separate User-agent block and leave the 'Disallow:' line blank.
• Define each bot in its own block; listing several bot names under one block may not work as expected.
• After the change, check the live file directly in a browser to confirm it displays correctly.

No. Anthropic's own documentation states that blocking ClaudeBot stops only training data collection and does not affect Claude-SearchBot or Claude-User; the three are managed separately in robots.txt.

That means a site can make separate decisions depending on its own priorities: keeping training data collection off while still appearing in live assistant answers is technically possible.

A scene we saw at a news site: after blocking ClaudeBot, the team assumed 'we no longer appear on Claude at all' and, without realising it, blocked Claude-SearchBot under the same blanket rule too. The decision criterion is this: each bot needs its own line and its own purpose stated clearly; applying one rule to several bots at once is the most common way something gets blocked that was never meant to be.

A common one we have seen at a flooring manufacturer is blocking every AI bot in one blanket rule, 'Disallow: /' under 'User-agent: *', which stops the unwanted training bots along with the search and assistant bots the business actually wanted.

Another is updating robots.txt without checking it live; a server cache or a wrong file path can mean the change never actually goes into effect.

The real way to measure the effect of blocking or allowing a bot is reading server log files; those logs show which bot name visits, how often, and which pages it pulls. Without that data, saying 'bot traffic dropped' or 'visibility went up' stays a guess.

A second check is asking the relevant assistant a brand question a few weeks after a robots.txt change and comparing it against the previous state; that shows the user-facing result logs alone cannot show.

The decision criterion is that a change needs at least a month's window before its effect is visible, since bots re-read the file at their own frequency; drawing a conclusion from a few days of observation is misleading.