On 15 September 2026 the defaults change for how Cloudflare lets AI bots reach a site. If you miss it, you may find out from a drop in Search Console – and that is the kind of failure that is hard to trace, because in the log it does not look like a failure at all.
I wrote here in July that AI bots come in three kinds, and that for someone running an MCP server one of them is a customer. Back then it was a framework. Now it is a date in the calendar, so let me go through what actually changes and what follows from it for my own servers.
What changes and who it hits
From 15 September, crawlers in the Training and Agent categories are blocked by default on pages that display ads. Search stays allowed by default.
It does not hit everyone equally. The new default applies to new domains onboarding to Cloudflare, to new customers, and to existing customers on the Free tier. Anyone who does not want it has to turn it off in the Security settings before 15 September. Paid plans are unchanged.
Two details get lost in the summaries, and they are the ones that decide the outcome. First: the condition is that the page displays ads. Neither my blog nor my MCP servers carry ads, so the default change does not reach me directly. Second, and more important: the categorisation exists for everyone behind Cloudflare, whether or not the default touches them. The moment you flip the "block training" switch yourself, everything below applies to you.
Three categories are not three names
The split makes sense because it describes what a bot does, not who it is.
| Category | What it does | What it is to an operator |
|---|---|---|
| Search | collects and indexes content to answer questions later | distribution – I want it |
| Agent | acts in real time on behalf of a specific person | on an MCP server, a customer, not a pest |
| Training | takes content to train or fine-tune a model | arguable – depends what you sell |
For a publisher the reasoning is straightforward: the text is the goods, and training is taking the goods without paying. For someone running an API or an MCP server it inverts. An agent arriving in real time to complete a task for a specific person is exactly the traffic the server exists for. Blocking it is not defence, it is closing the shop.
The trap this article is about
Block the Training category and you also block multi-purpose crawlers – Googlebot, Applebot, Bingbot – even with Search allowed.
The reason is simple and unpleasant: one crawler does both jobs. Googlebot indexes for search and also collects data that models learn from. Say "no training" and there is no half of it to let through. This is the one sentence from the whole change I would want everyone clicking around in those settings to read.
What makes it treacherous is the timing of the feedback. Block a customer by mistake and they tell you. Block Googlebot by mistake and nobody tells you. The server returns correct responses, monitoring is green, no dashboard lights up. You notice weeks later from falling impressions – by which time you are no longer looking for the cause in something you clicked in September.
I walked into the same kind of silence last year in a different shape, verifying hosting from my home network while DNS quietly misled me: everything answered correctly, just from a different machine than I thought. The lesson is the same. A setting whose failure produces no error message has to be verified actively, and from the outside.
What it means for an MCP server
An MCP server has no readers. It has callers. An automated client arriving on behalf of a person who is asking about a company in a register is a served request, not content theft. For that server, the Agent category is a revenue line.
That does not make every automated request useful. I run my own traffic classifier on these servers that sorts each request into a class by behaviour, and its numbers are sobering here.
One third-party monitoring tool was sending thousands of requests a day to all of my MCP endpoints. Roughly two thirds ended in a 400, because it was not valid MCP JSON-RPC – it was an availability ping, not a tool call. It passed through the request counter while consuming no compute and calling no tool.
Hence a rule I keep repeating, because it keeps being forgotten: requests are not usage. Measure interest in your server by request count and you are mostly measuring how many catalogues and monitors know it exists.
The same distinction works in the other direction. One AI crawler went through one of my applications at 77,000 requests a day, and 89,000 the next. Looking only at load, that would be a reason to block it. But those are organisation pages in every language version, and they entered an index that answers questions for people. That is not an attack. It is a distribution channel that showed up on its own.
How to measure it yourself
The basic mistake is classifying by the name in the header. Anyone can set a header. Night traffic that looked like people on one of my domains turned out to be largely scanners with an ordinary browser header, walking non-existent paths and probing for vulnerabilities.
Asking about behaviour is more useful:
Comes at night and never calls a tool? → probe or scanner Calls a tool and waits for the result? → agent, i.e. a customer Walks the sitemap systematically? → index, i.e. distribution Mostly receives 400? → that is not usage, that is a ping
The check before 15 September is short. Find out whether the domain is on the Free tier and whether any page displays ads – only then does the default change act on its own. Look at what is switched on in the Security settings, and leave the training block alone as long as search matters more to you than protecting the text. And after 15 September, watch impressions, not just error rates.
What I will measure, and what would prove me wrong
There are three claims here, and I know how I would find out I was wrong about each.
First: that my domains will not change on their own, because they carry no ads and are not on the Free tier. Any drop in indexed pages or impressions in the weeks after 15 September that nothing else explains would refute it.
Second: that the share of agent traffic on my MCP servers will not fall after 15 September, because agents reach them directly rather than through ad-supported pages. A drop in actual tool calls – not requests – against the August average would refute it.
Third: that a crawler collecting for answers brings people. That one I cannot claim yet. I have its volume, not the visits it produced. Once I have data from both sides I will write up how it turned out, even if the channel brought nothing.