Should You Block AI Crawlers? The 2026 Block-or-Feed Decision

Once you see that AI is scraping your content and barely sending traffic back, the next question writes itself.

So should I just block them?

And the internet gives you the two worst possible answers. Half of it screams "block GPTBot, they're stealing your work." The other half warns "block them and you'll vanish from ChatGPT forever."

Both are wrong. Because "AI bot" is not one thing you allow or deny. It is three different jobs, and blocking each one costs you something completely different.

I run TheGuideX on a tuned crawler setup, robots rules, content signals, the lot, so I have had to make this decision for real. Here is the honest map, with the 2026 facts, so you can make it deliberately instead of copying a robots.txt off a forum.

Blocking isn't one switch, it's three

Get this one distinction right and the whole decision becomes simple.

• Training bots feed the model. GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended, ClaudeBot (Anthropic). Block these and your content stays out of the next model, and your AI citations mostly survive.
• Retrieval bots power the cited answer. OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot, Bingbot (also Copilot). Block these and you disappear from those engines' cited answers.
• Live user-fetch bots grab one page because a human asked for it. ChatGPT-User, Perplexity-User, Claude-User. These are real people pasting your URL, not automated scraping.

This split is not an accident. OpenAI and Anthropic deliberately gave training and retrieval separate user-agents so you can block one without the other. That is the entire game: you can tell GPTBot to go away while letting OAI-SearchBot keep quoting you in ChatGPT.

The most common mistake in this whole topic is a site owner blocking GPTBot and OAI-SearchBot in one angry sweep, protecting their content and deleting their AI visibility at the same time.

The Google-Extended trap

Here is the one that catches even experienced people.

You've heard AI Overviews are eating your clicks (I covered the scale of that in AI dark traffic). So you block Google-Extended, assuming it gets you out of AI Overviews.

It doesn't. Not even slightly.

Google's own crawler documentation is blunt: Google-Extended controls whether your content trains Gemini and Vertex AI, and it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." AI Overviews are served off the live Search index, crawled by regular Googlebot, not by Google-Extended.

So the only way out of AI Overviews is out of Google Search entirely. Blocking Google-Extended stops Gemini training and nothing else. Useful if that's your goal, useless if you thought it was an AIO opt-out.

robots.txt is a request, not a wall

Now the uncomfortable part. Everything above assumes the bot actually obeys your robots.txt. Many do. Some very much don't.

robots.txt is a plain text file that asks crawlers not to visit certain paths. It has no teeth. A well-behaved bot and a bad actor see the exact same file, one complies, one doesn't. Two things make this concrete in 2026:

• Bytespider (ByteDance/TikTok) has been repeatedly documented fetching robots.txt and then crawling the disallowed paths anyway.
• Perplexity got caught red-handed. In August 2025, Cloudflare published evidence that when Perplexity's declared crawler hit a Disallow, a stealth crawler took over, impersonating a normal Chrome-on-Mac browser, rotating IP addresses and switching networks to dodge the block. Cloudflare de-listed Perplexity as a verified bot over it.

So robots.txt is genuinely useful, the reputable AI companies honour it, but understand what it is: a sign on an unlocked door. To actually stop a crawler, you need edge enforcement, a firewall or a CDN like Cloudflare that fingerprints and challenges the traffic. A text file cannot do it.

So what actually happens if you block?

Put the pieces together and the trade-off is clean and real:

This is why the lazy "block all AI bots" advice is actively harmful for anyone doing answer-engine optimization. You would be protecting content nobody was going to pay you for, and throwing away the citations that drive the dark traffic actually worth having.

The economics: your content has a price

The reason this decision suddenly matters is money. In 2026 the web is quietly repricing AI access.

Cloudflare flipped the default. On 1 July 2025, Cloudflare, which carries around a fifth of web traffic, began blocking AI crawlers by default for new domains. The web moved from opt-out to opt-in: AI companies now need permission to scrape a Cloudflare-served site. Cloudflare also launched Pay Per Crawl, which lets a site charge crawlers per request using an HTTP 402 Payment Required response, and in 2026 it says customers send over a billion 402 responses on an average day.

Big publishers are getting paid; small sites are getting scraped for free. The licensing deals tell the story:

• Reddit licensed its data to Google for a reported ~$60M a year (2024).
• News Corp signed with OpenAI in a deal reported at over $250M across five years (2024).
• The New York Times licensed content to Amazon at a reported $20-25M a year (2025).
• Anthropic settled a copyright case over pirated training books for $1.5 billion (2025).

That is the honest asymmetry. If you are The New York Times, your content has a negotiated price. If you run a normal site, it is training data by default, and the crawl-to-referral math is brutal, most AI crawling is extraction with almost no traffic sent back.

New standards are trying to level this. RSL (Really Simple Licensing), launched September 2025 and backed by Reddit, Yahoo and Medium, adds a machine-readable licence to robots.txt so sites can state terms and even pool into a collective, ASCAP-style. It is promising. But like robots.txt, it is a stated licence, not a technical block, it works only if the AI company chooses to honour it or a court makes them.

What about the law?

Short version: do not build your strategy on "AI training is illegal," because in 2025 the courts started saying the opposite, carefully.

Two US federal rulings in June 2025 found that training a model on copyrighted work can be fair use, on the specific facts. In Bartz v. Anthropic, the judge called training "exceedingly transformative", but still held that downloading and keeping pirated copies was infringement (hence the $1.5B settlement). In Kadrey v. Meta, the judge ruled fair use narrowly, effectively saying the authors "made the wrong arguments" rather than blessing AI training in general.

And the biggest case, The New York Times v. OpenAI, is still undecided, with summary-judgment briefing running into 2026. The live question, whether AI answers substitute for the original and cause market harm, has no ruling yet.

Accurate takeaway: courts have found the act of training can be fair use, while punishing piracy in how the data was obtained. The market-harm fight is unresolved. So robots.txt and licensing are your practical levers today, not a settled legal right to say no.

llms.txt: ship it, don't count on it

Because someone always asks: no, llms.txt is not the answer.

It is a proposed standard, a curated Markdown map of your site for LLMs, and the idea is reasonable. But adoption is the problem. Google publicly said it does not support llms.txt and has no plans to; John Mueller compared it to the long-dead keywords meta tag. Server-log studies show AI crawlers almost never request the file.

Ship one if you like, it costs nothing and does no harm. Just don't mistake it for a control that changes how any model sees you today. It doesn't, yet.

The decision framework

So, block or feed? Stop looking for one answer and match the move to the goal.

• Want maximum AI visibility? Feed the retrieval bots. Leave OAI-SearchBot, PerplexityBot, Claude-SearchBot and Bingbot open, and do the AEO work to get cited.
• Want to protect content from training? Block only the training bots, Disallow: GPTBot, CCBot, Google-Extended, ClaudeBot. Your citations survive.
• Want to actually enforce it, or get paid? Move to the edge. Put the site behind Cloudflare (the free tier can block AI bots by default), and look at Pay Per Crawl or RSL if you have the leverage.

For a small site with no CDN, be honest about your options: robots.txt and Content Signals are free but only the compliant bots obey; the only thing that truly enforces is putting a real gatekeeper (Cloudflare) in front of the site. Everything else is a request.

My honest recommendation

For most people reading this, sites that want to be found and cited in AI answers, here is what I would actually do:

Block training, keep retrieval, and stop pretending robots.txt is a lock.

Disallow GPTBot, CCBot, Google-Extended and ClaudeBot so you are not free training data. Leave the retrieval and search bots open so ChatGPT, Perplexity and Claude can still quote you, that is your visibility, and it is the dark traffic worth earning. If your content is genuinely valuable and you can put it behind Cloudflare, enable the edge block so the request has teeth. And ship an llms.txt only because it is free, not because it works.

The one thing not to do is the angry all-or-nothing block. Denying every AI bot to "protect your content" mostly just deletes you from the answers your competitors are winning.

Final take

Blocking AI crawlers is not a moral stand, it is a set of trade-offs you should make on purpose.

The web is repricing AI access in real time, default blocks, pay-per-crawl, licensing collectives, live lawsuits. Most of that leverage belongs to the giants. But the one decision every site owner controls is which bots to let in, and now you know that it is three decisions, not one.

Feed the ones that quote you. Starve the ones that only train on you. And remember that a text file asks, but only the edge enforces.