AI Crawlers and robots.txt: Block or Allow?

For most local businesses, allow them. An AI crawler is a program that visits your pages, and the ones behind AI search, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot, need access if you want to be found. Blocking is an optional choice, mostly about whether your content may be used for training.

This guide is part of the complete AEO guide. The names and purposes below come from each company's own documentation, current in September 2026. Names change, so check the source before you edit anything.

What is a crawler, and what is robots.txt?

A crawler is a program that visits web pages automatically to read them. Google's crawler, Googlebot, builds the search index. AI companies run their own crawlers for search features, for user requests and for training their models.

robots.txt is a plain text file at the root of your website, at yoursite.com/robots.txt. Google's documentation says it tells crawlers which URLs they can access. You do not write code so much as list rules: which crawler, and which parts of the site it may or may not visit. Many website builders create the file for you.

Which AI crawlers exist, and what is each for?

Each company separates its crawlers by job. That separation is what lets you allow search and block training, or the other way around.

• OAI-SearchBot (OpenAI): surfaces websites in ChatGPT's search features. OpenAI says a site that disallows it will not be shown in ChatGPT search answers.
• GPTBot (OpenAI): crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it signals that your content should not be used for training.
• ChatGPT-User (OpenAI): used for certain actions a user takes in ChatGPT and custom GPTs. OpenAI says robots.txt rules may not apply, because a person started the action.
• Claude-SearchBot (Anthropic): analyzes content to improve the quality of Claude's search results. Anthropic says blocking it may reduce your visibility in those results.
• ClaudeBot (Anthropic): collects public web content that may contribute to training. Restricting it signals that your future content should be excluded from training datasets.
• Claude-User (Anthropic): may access a website when a Claude user asks a question that needs it. Blocking it prevents Claude from fetching your pages for user questions.
• PerplexityBot (Perplexity): surfaces and links websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models, and recommends allowing it.
• Perplexity-User (Perplexity): visits pages when a user's question calls for it. Perplexity says it generally ignores robots.txt rules, because a user requested the fetch.
• Googlebot (Google): builds Google's search index. Google says a page must be indexed and eligible to show with a snippet to appear in AI Overviews or AI Mode, and robots.txt directives for Googlebot are the control.
• Google-Extended (Google): a robots.txt token that lets you manage whether content Google crawls may be used to train future Gemini models and to ground responses in Gemini Apps and Vertex AI. Google says it does not affect inclusion in Google Search or serve as a ranking signal.

Should a local business allow or block them?

Most local businesses want to be found, so the default answer is to allow the search crawlers and leave the file alone. If OAI-SearchBot cannot read your site, OpenAI says ChatGPT search will not show it. The same logic applies to the other search crawlers on the list. How Google's own AI features treat your pages is covered in Google AI Overviews and your local business.

Training is a separate question, and it is your call. Blocking GPTBot, ClaudeBot or Google-Extended asks those companies not to use your content for training. OpenAI says each setting is independent, so you can block GPTBot and still allow OAI-SearchBot. The trade-off is inference, not a documented result: a model trained without your pages may know less about you from memory, so you depend more on the live search crawlers finding you.

A typical taquería, dental office or plumber has little reason to block anything. The bigger risk for a small business is blocking by accident. That can happen when a site was built with a hide-from-search setting that never got switched off. WordPress, for example, has a Settings, Reading option called Discourage search engines from indexing this site, which asks search engines to skip your pages. Security plugins and firewalls can also block crawlers without touching robots.txt at all.

How do you check yours, and what do the lines look like?

Type yoursite.com/robots.txt into your browser. If you see a page of text, that is the file. Look for a line reading Disallow: / under User-agent: *, which asks every crawler to skip the whole site. If you find one you did not intend, ask whoever manages your site to remove it.

To allow everything, leave the file as it is, or say so explicitly with these two lines.

• User-agent: *
• Allow: /

To stay findable in search while asking the training crawlers to skip your content, add a group for each one. Each User-agent line starts a new group, and the Disallow line under it applies to that crawler only.

• User-agent: GPTBot
• Disallow: /
• User-agent: ClaudeBot
• Disallow: /
• User-agent: Google-Extended
• Disallow: /

Leave OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot out of that file, or give each an Allow: / line. Under the standard rules, a group that names a crawler takes precedence over the general User-agent: * group for that crawler, although Google's documentation notes that crawlers can interpret rules differently.

Is robots.txt a lock?

No, it is a request. Google's documentation says the instructions in robots.txt cannot enforce crawler behavior, and that it is up to the crawler to obey. Reputable crawlers do. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. Google also says robots.txt is not a way to keep a page out of Google. For that, use noindex or a password.

Anthropic adds a warning about blocking IP addresses instead of using robots.txt: it says doing so may not work correctly, because it stops Anthropic from reading your robots.txt file. Do not put anything private on a public page and expect a robots.txt line to protect it. Put it behind a login.

Changes are not instant either. OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust.

How does robots.txt relate to llms.txt?

They do opposite jobs. robots.txt says which crawlers may visit which parts of your site. llms.txt is a plain-text fact sheet about your business. One controls access, and the other describes you.

Be realistic about the second. Google's guide says Google Search ignores AI text files, and the crawler documentation from OpenAI, Anthropic and Perplexity describes robots.txt controls and does not describe llms.txt. The Growrank dashboard generates an llms.txt file for you as a facts sheet, and it costs nothing to publish, but treat it as a tidy extra and not as a ranking lever.

For the rest of the basics in one place, see the free SEO checklist for small businesses. To see whether any of this changes what assistants say about you, the Growrank dashboard records their answers every week; see plans and the free trial.

Frequently asked questions