Should You Block AI Crawlers or Feed Them?

Somewhere in your website's settings is a small text file that decides which automated visitors are allowed in. For most of the web's history that file was uncontroversial: let the search engines crawl, block the badly behaved scrapers, move on.

That decision has become genuinely contested. A growing number of crawlers now collect content to train or ground AI systems, and site owners are being asked, often without realising it, whether to let them. Publishers have gone in both directions, some blocking aggressively, others opening everything.

There is no universal right answer, but there is a way to reason about it that beats copying whatever someone else did.

The two things being confused

Most of the confusion here comes from treating one decision as two, or two as one.

Training crawlers collect content to be used in building or improving models. Your content contributes to a general capability. There is no direct path back to you, no citation, no visit.

Retrieval crawlers fetch content at the moment a question is asked, so it can be used to answer that question. This is where citations come from, and where a business can be named as a source.

Blocking one is a very different decision to blocking the other. A publisher whose business is selling access to its archive has a clear interest in restricting training use. A services business trying to be found has almost no interest in blocking retrieval, because retrieval is the mechanism by which it gets mentioned.

Conflating them produces the worst outcome: a business blocks everything to protect content nobody was going to pay for, and disappears from the answers it wanted to appear in.

Who actually loses by allowing it

Work through your own position rather than the general argument.

If your content is the product, and people pay to read it, allowing training use gives away the asset. That is a real cost, and the case for restricting is straightforward.

If your content is marketing for something else you sell, which describes the overwhelming majority of business websites, it was already free. It was published to be read, shared and quoted. A model trained on your explanation of your service is not depriving you of revenue, because nobody was ever going to buy the explanation.

The second group is much larger than the first, and its members frequently adopt the anxieties of the first without checking whether they apply.

The cost of blocking is not zero

Blocking feels like the safe default. It is not neutral.

If retrieval crawlers cannot reach you, you cannot be cited. Your competitors, who did not block, become the sources for questions in your field. Over time the picture assembled about your industry is built from their material rather than yours.

There is a subtler cost too. Being described accurately depends on your own description being available. If the accessible sources about your business are third-party directories and outdated profiles rather than your own current pages, the description that circulates will be the one you have least control over.

Where a considered restriction makes sense

None of this argues for opening everything without thought.

Content behind a login should stay behind it. Anything you would not publish openly should not become accessible because a crawler asked politely. Material you have paid to produce as a saleable asset, such as research reports, deserves a deliberate decision rather than an assumption.

It is also reasonable to distinguish by section. Public marketing pages, service descriptions and explanatory articles argue for access. A members' library or a paid resource does not. This is a per-directory decision, and treating it as a whole-site switch is what leads to bad outcomes in both directions.

Bear in mind that the file is a request, not an enforcement mechanism. Well-behaved crawlers honour it and badly behaved ones do not. If you genuinely need to prevent access, that requires actual access controls.

The direction of travel

The commercial reality is that these systems increasingly sit between a business and its customers, in the same way that search engines came to. Sitting out is a position, but it is rarely a durable one.

You can already see this settling elsewhere. The debate about whether to use automation in customer contact has largely resolved into a question of where, not whether, and the practical guidance on AI in customer support is now about which interactions suit it rather than whether to permit it at all. Content discovery is following the same curve.

The same is true of how audiences find things generally. The steady movement described in most accounts of current social media marketing trends is towards discovery happening inside platforms rather than through links out of them. Search is not exempt from that pattern.

A workable default

For most businesses whose website exists to generate enquiries rather than to sell access to content: allow retrieval crawlers, make a considered decision about training crawlers based on whether your content has standalone commercial value, protect anything gated, and revisit it annually as the standards firm up.

Then make sure the pages you have opened are actually worth citing, because access without substance achieves nothing. Getting that combination right is the practical work behind AI search visibility, and it starts with a deliberate decision rather than whatever the default settings happen to be.

The worst position is the unexamined one: a blanket block copied from a publisher whose business model is nothing like yours, quietly removing you from the answers your customers are reading.