Part of a series on building cz-agents → under the hood
The headline that does not apply to me
Within days of the Cloudflare announcement, the internet had settled on a shorthand: "from 15 September, AI crawlers get blocked." It reads like a sentence you are supposed to act on. When I actually worked through what the change says, I found it does not touch me at all — and that a large share of the advice circulating around it is aimed at the wrong target.
The announcement has two halves, and it is the second one that gets garbled. First half: Cloudflare stops treating "AI bot" as one species and splits it into three categories — Search, Agent and Training. Second half: from 15 September 2026, on newly onboarded domains and for all existing free-tier customers, Training and Agent are blocked by default — but only on pages that display advertising. Search stays allowed everywhere. Anyone who does not want the new defaults can opt out in their security settings any time before the deadline.
That advertising condition is the whole point, and it is exactly what the headlines drop. The logic is coherent: if your page earns its keep by a human reading it and seeing an ad, then a bot that consumes the content and sends you no reader is eating your lunch. If you carry no advertising, that conflict of interest never arises, and the new default does not land on you. Neither my blog nor my MCP hub runs ads. So the first verified fact is that this is a non-event for me — one I still had to work through, because the first half of the announcement matters regardless of any default.
"AI bot" just split into three different animals
The real news is not the blocking. It is the taxonomy — because each of those three categories has a completely different economics, and each implies a completely different decision for whoever runs the site.
Search is a bot that reads your content in order to cite you and link to you. It is the new shape of what search engines have done for two decades. When an assistant names you as a source, you get a visit, or at minimum you get your name in front of someone. Blocking it means disappearing from the place where people increasingly go to look things up.
Training is a bot that carries your content off into a training set. Here the exchange is the least balanced: the model learns from your text, you never find out that it happened, and the feedback loop is zero. This is the one category where it genuinely makes sense to weigh whether to allow it.
Agent is a bot that touches your site or your interface because a human asked it to, right now. Someone puts a question to their assistant, and the assistant goes and fetches data so it can answer. That is not a crawler. That is a user who came in through a different door.
Anyone who lumps all three together — and most of the advice online does — has no route to a correct decision. "Block the AI bots" produces a different outcome in each category, and in two of the three the outcome is self-inflicted damage.
Why "Agent" is the most valuable thing that can knock on my door
I run eight public MCP servers. MCP is the protocol by which an AI assistant connects to somebody else's data and tools — in this case, to mine. When someone connects one of my servers, their assistant then queries my interface for whatever its human needs. And on my side, that query looks like — what else — a bot.
This is the counterintuitive part. The Agent category is precisely the set of requests my product exists to serve. Blocking it would not be a defence. It would be locking the shop from the inside and hanging a "do not disturb" sign on the door. Nothing else would happen: the human at the other end would get an error, would blame the tool, and would go elsewhere.
And here is the trap that made me write this. The advice "block AI bots" has become a reflex over the past year. It sounds safe, it sounds responsible, and nobody asks it what, specifically, it is blocking. Yet those three species can be told apart in your own traffic — it just takes sitting with your logs for a while. When I classified my own, the overwhelming majority of what looks like "bots" turned out to be catalogue indexers and assorted probes: liveness checks, verification that the server still answers, mapping the offering into directories. Useful, harmless, and not customers. And then there is the small, valuable minority where a real person with a real question sits at the other end. A blanket block kills that minority first, precisely because it looks identical to everything else.
Generalised: if your business offers any machine-readable surface at all — not just MCP, but an ordinary API, a structured price list, availability data, a catalogue — then a growing share of your customers will arrive through an agent. Blocking Agent shuts a sales channel at the exact moment it starts to grow.
My decision table
Here is how I translate the three categories into my own interest. It is not a universal recommendation — it is my verdict for my sites, and the value is in how you get to one, not in copying mine.
| Category | What it does | My interest | Verdict |
|---|---|---|---|
| Search | Reads and cites, brings attention | Want it | Open. Being citable is a distribution channel in its own right now. |
| Agent | Fetches data for a live user asking right now | Want it most | Open, emphatically. These are customers. |
| Training | Carries content into a training set | Weigh it | Blog open, product interfaces not. See below. |
For the blog the decision is easy, even if it looks naive at first glance. I write here so that someone wrestling with a similar problem finds me. A growing share of those people no longer ask a search engine — they ask an assistant, and the assistant cites what it has read. For a technical text, closing the door on Training trades hypothetical protection for certain invisibility. That is a bad trade for me. If I ran an ad-funded news site, the arithmetic would be entirely different — which is exactly why the advertising carve-out in the new policy is the sane part of it.
For product interfaces the calculus inverts. What they serve is not prose: it is structured output whose value exists at the moment of the query, not as material to learn from. There I care that a live user is served at the other end, and I do not care to have the whole output pulled down in bulk. The tool for that, in the end, is not bot blocking but plain old rate limits and authentication on the paid parts. Those work regardless of what a bot claims to be — and that is their entire virtue.
What I actually have configured today
If I am going to write about someone else's policy, I should say where I stand myself. I checked this today rather than trusting memory.
My personal site, this blog included, and the MCP hub both sit behind Cloudflare. Neither displays advertising, so the new default block lands on neither — which I had to confirm, because had I only read the headlines I would have gone and changed a setting for no reason.
And I nearly got that confirmation wrong, which is worth admitting, because it is exactly the mistake this piece warns against. On my home network my own DNS points the domain at a machine in my living room. So when I called the blog up to check, a bare Apache answered with no trace of Cloudflare anywhere — and for a moment I concluded the site was not behind Cloudflare at all. It was. I had simply asked from the one spot on earth where nobody else is standing. If you are auditing your own setup, ask from the outside. Otherwise all you confirm is how your site looks from your own sofa. The blog's robots.txt today is a bare Allow: / with not one AI-specific directive in it. I am leaving it that way. Anything else would, in my case, be a fix for a problem I do not have.
One new capability does interest me irrespective of the defaults: robots.txt can now carry a content-use preference at three levels — immediate (use it now, answer from it), reference (cite it and link back) and full (do as you like, training included). That is a shift from "may you enter or not" to "on what terms may you use this". It binds nobody any harder than robots.txt ever did — it is a statement of intent, not an enforcement mechanism. But it is the first sensible vocabulary for what most authors of technical writing actually want to say: read it, cite it, link back — and do not pass it off as yours.
Pay Per Use: a nice sentence, not yet an income
The other half of the announcement is about money. Pay Per Crawl — paying for the download — is evolving into Pay Per Use: you get paid when the content creates value, not merely when a bot fetches it. Ceramic.ai and You.com are named as the first partners.
The shift makes intellectual sense. Charging per crawl rewards volume, which is a bad incentive for both sides; charging per use rewards relevance. But against my own numbers, sobriety wins: I am a small publisher with tens of readers a week. If every use of my text started paying tomorrow, we are talking about a sum that would not cover the hosting. Pay Per Use is, for me, theory rather than revenue, and it would be dishonest to wrap it in the word revolution.
So I read it as a signal of direction, not as a plan. If and when it becomes genuinely collectable, I will need two things in order: knowing who reads me and in which category, and having content worth paying for. Both are worth doing even if Pay Per Use never pays out a cent. That is a decent rule in general: only invest in the preparations that pay off even when the promised future fails to arrive.
What I take from it
Underneath this news sits a shift bigger than one toggle in a settings panel. For twenty years we sorted traffic into "humans" and "bots", and the split made sense, because a bot never bought anything. Now there is a third category — a visitor who is technically a bot and economically a human, because it acts on a human's behalf and in a human's interest. Our tooling does not yet know how to handle that, and our habits know even less.
The practical conclusion is therefore dull, and it holds: before you reach for a block, look at what you are actually blocking. Open your own logs, sort that traffic into crawlers, probes and genuine questions from living people, and only then decide. It takes an afternoon. If it turns out the AI bots are only draining you, shut the door with a clear conscience and you will be right to. If it turns out that some of them are your customers in a new outfit, you have just spared yourself a suicide committed with the satisfaction of a job well done. That is the most expensive mistake available here: defending yourself against something that was, in fact, demand.
Facing a similar decision in your company? → AI consulting