AI search

Should you block AI crawlers from your website?

23 September 2026

The worry behind the question is fair. AI companies read the web to build their products, and nobody asked you first. But "AI crawlers" covers several kinds of bot doing different jobs, and a blanket block shuts out the bots that could send you work along with the ones you object to.

This post covers which bot does what, from each company's own documentation, how to check your site and what each choice costs. Copyright law is a question for an attorney.

What do you lose if AI crawlers cannot read your site?

You lose the chance to have your own pages used when someone asks an assistant about your kind of work. ChatGPT, Claude and Perplexity can search the web, read pages and write a reply, often with links to the pages they used.

Say a homeowner in Durbanville asks which roofers can fix a leak before the winter rain. The answer is built from pages the assistant could find and read. If your site turned its search bot away, your leak repair page was probably not one of them.

Nobody pays to read your page on geyser replacements. It exists so that whoever reads it, including software working for a homeowner, ends up with your number. Making pages easy to read and quote is what AI SEO is for, and none of it works on a site that turns the readers away.

Are all AI crawlers the same?

Far from it. The big AI companies run separate bots for separate jobs and document each one, so you can treat them differently. Most fall into three kinds.

Training crawlers collect pages that may be used to train AI models. Blocking one asks the company to leave your content out of future training. On its own, it does not remove you from search results.

Search crawlers build the index an AI search tool answers from, much as Googlebot does for Google. Block one and your pages can drop out of that tool's answers.

User-triggered fetchers, or answer bots, open a page because someone asked a question that needs it. Because a person asked, OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anthropic says blocking Claude-User does stop Claude fetching your pages.

These are the names to look for.

Does blocking Google-Extended take you out of Google?

No. Google's list of its common crawlers calls Google-Extended a standalone product token. It has no crawler of its own. It controls whether content Google has crawled may be used for two things. One is training future Gemini models. The other is grounding, which means feeding content from the Search index to Gemini while it answers, in the Gemini app and on Google's Vertex AI. On Search, Google is direct: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

So blocking it costs you no rankings, but it is not a pure training switch: it covers grounding in the Gemini app too.

Nor does it control AI Overviews or AI Mode. Those are part of Search, and Google's page on AI features and your website says robots.txt rules for Googlebot are the control there. Block Google-Extended and you can still appear in AI Overviews. Block Googlebot to avoid them and Google stops crawling your site for Search altogether. If that has happened by mistake, see why a new website does not show up on Google.

Apple's Applebot-Extended is a similar token. It crawls nothing. Disallow it and, Apple says, your content is opted out of training its models, while your pages can still appear in its search results.

How do you check if AI bots are blocked on your site?

Start with the robots.txt file. It always lives at the same address, and you can read it on your phone.

1. Type your domain followed by /robots.txt into a browser, for example yourdomain.co.za/robots.txt.
2. If the page is not found, there is no robots file and no bot is being asked to stay away. Skip to the next section.
3. If there is a file, read it in groups. Each starts with User-agent lines naming bots, followed by Allow or Disallow lines saying where they may go.

The one to worry about is the blanket block:

User-agent: *
Disallow: /

The star means every bot and the slash means the whole site. Any bot without its own group in the file is asked to stay out, Googlebot included. It can arrive with a template, or stay behind from a test version of the site, and nobody notices because the site still loads for people.

A targeted block names one bot. This is how you block GPTBot in robots.txt:

User-agent: GPTBot
Disallow: /

Look for each name in the table. A name followed by Disallow: / is blocked from the whole site. A Disallow line with a folder after it, like /wp-admin/, only covers that folder, which is normal.

Bear in mind that the file only asks. Google's introduction to robots.txt is plain about it: "While Googlebot and other respectable web crawlers obey the instructions in a robots.txt file, other crawlers might not."

Can something other than robots.txt block AI crawlers?

Yes, and people miss it. Many sites sit behind a CDN or security service, a layer between your visitors and your hosting that speeds the site up and filters bad traffic. Some have their own switch for AI bots, and when it is on, the bots are turned away whatever your robots.txt says.

Cloudflare, one of the biggest of these services, changed its default in July 2025: new sites were asked at sign-up whether to allow AI crawlers, with blocking as the starting point. Its settings now split AI bots into search, training and agent groups, and the defaults have changed over time. What your site has depends on when it was set up and what was clicked. A firewall at your host can block them too, and Perplexity's crawler page gives steps for letting its bots through one.

Google's own advice for its AI features is to make sure crawling is allowed "by any CDN or hosting infrastructure", not only in robots.txt.

When does blocking AI crawlers make sense?

When your content is the product, or when bots are costing you money.

Say you are an electrician who sells an online course for apprentices, or a builder who sells house plans. That material is what you sell. Block the training bots, and put paid material behind a login, because robots.txt protects nothing on its own.

The other case is a server struggling under bot traffic. Anthropic and Common Crawl both say they honour a Crawl-delay line in robots.txt, which slows their crawlers down rather than shutting them out. Bots that ignore robots.txt will ignore that too, so badly behaved bots are a job for the firewall.

If neither describes you, a full block costs more than it saves.

Can you block AI training and still show up in AI answers?

Yes. OpenAI's overview of its crawlers says each setting is independent, so a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot. Anthropic, Google and Apple split theirs the same way. This robots.txt turns away the training bots and Common Crawl but not the search and answer bots:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

With nothing else in the file, every bot not named keeps its access. CCBot is there because Common Crawl's archive is open to anyone. Leave out Google-Extended if you want the Gemini app to use your pages when it answers.

Blocking training costs little you can see, though a model that never read your site learns nothing about your business from it. Leaving everything open costs some server traffic, and your words may end up in training data. Blocking everything costs you the search and answer side, where a homeowner asking who to call might find your name.

Where does Peak Leads stand on AI crawlers?

For a business that wants the phone to ring, our advice is the middle path or no block at all. Whether AI companies train on your pages is up to you. A search bot shut out by a forgotten line is a mistake. That is why our AI SEO work includes checking your robots file and your server, so AI crawlers are not blocked by accident.

We will not promise that letting them in gets you named. Nobody can guarantee a place in an AI answer, and we say so. Letting them in only makes you eligible. Being worth quoting is the rest of the work.

What should you check on your own site?

Open your robots.txt on your phone and look for a star with Disallow: / under it, and for any of the names in the table. Then find out whether your site sits behind Cloudflare or a similar service, and check its AI bot settings or ask whoever manages it.

When someone else runs your site, ask them to walk you through the file and the settings in plain English, the way your monthly SEO report should.