If you clicked “Block AI bots” in your Cloudflare dashboard at any point in 2024 or 2025 and moved on, tomorrow that decision starts doing something you almost certainly did not intend. Starting September 15, Cloudflare changes how it evaluates crawlers that do more than one job. Googlebot does more than one job. If you are blocking AI training crawlers, you may be blocking Google by this time tomorrow.
Here is what is changing, who it affects, and what to check before the end of today.
What Cloudflare Changed on July 1
For years, Cloudflare’s bot settings offered a single toggle: block AI bots, or do not. On July 1, 2026, Cloudflare replaced that with a three-way classification system, available immediately to all customers including the free tier.
The three categories are:
- Search: crawlers that index content to answer questions later and send referral traffic back to your site. The traditional search engine crawl.
- Agent: real-time bots acting on behalf of a user right now. ChatGPT browsing the web, Gemini operating Chrome, that kind of thing.
- Training: crawlers pulling content to train or fine-tune an AI model. GPTBot, ClaudeBot, and similar.
Those controls went live immediately for everyone. What changes tomorrow are the defaults.
The Googlebot Problem
This is the part most guides are glossing over, and it is the part that matters most.
Googlebot crawls for Search. Googlebot also crawls for Training. Google built one crawler that does both jobs simultaneously. Applebot and Bingbot work the same way.
Cloudflare’s new rule, effective September 15, is that multi-purpose crawlers are evaluated under the most restrictive rule that applies to any of their behaviours. So if you have told Cloudflare to block Training crawlers, from tomorrow that rule applies to Googlebot too, because Googlebot trains as well as searches. The most restrictive applicable rule wins. Googlebot gets blocked.
Cloudflare named this consequence explicitly in their July 1 announcement. Multi-purpose crawlers including Googlebot, Applebot, and Bingbot will be blocked for customers who selected to block Training, whether through the new controls or through the old legacy “Block AI bots” one-click toggle.
The block happens at the network layer, before the request reaches your server. It does not show up in your robots.txt. Your server logs record nothing. The only signal you will get is a drop in Google Search Console crawl activity, and that typically takes days or weeks to surface. By the time you notice something is wrong, it has already been wrong for a while.
There is currently no way to block Training crawlers while keeping Googlebot through. Cloudflare has acknowledged the tension and indicated that per-crawler overrides are coming, but that tooling is not live yet. This conflict originates with Google’s decision to bundle both behaviours in one crawler, not with Cloudflare’s implementation. But you are the one who will feel it in your search rankings if you do not act.
What the September 15 Defaults Actually Change
There are two separate changes happening tomorrow. They affect different groups of site owners.
New domains and new sites on existing accounts will have Training and Agent blocked by default on pages that display ads. Search stays allowed. If you are adding a new site to Cloudflare tomorrow, this is where you start.
Existing free-tier accounts that have never changed their bot settings get the same defaults applied automatically. If you are on Cloudflare’s free plan and never touched the bot controls, Training and Agent will be blocked on your ad pages from tomorrow.
Existing paid accounts and accounts with custom bot settings are not automatically changed. Your existing rules stay as they are. But if you enabled “Block AI bots” or “Block Training” at any point, the Googlebot conflict above applies to you regardless of plan type. That is not a tomorrow problem. That is already your situation, and it becomes more consequential tomorrow when Cloudflare formalises the multi-purpose crawler rule.
How to Check Your Settings Right Now
This takes about five minutes. Do it before the end of today.
- Log into your Cloudflare dashboard at dash.cloudflare.com
- Select the domain you want to check
- Go to Security, then Bots
- Look for the AI Scrapers and Crawlers section, or any legacy “Block AI bots” toggle
- If “Block Training” is enabled, or if the old “Block AI bots” toggle is on, you are in scope
- To protect your Google search visibility for now: disable Training blocking until Cloudflare ships per-crawler overrides
Worth knowing: the AI Scrapers and Crawlers section may appear under a slightly different label depending on your plan level and when your account was last updated. If you do not see it immediately, check under Security → Bots → Bot Fight Mode and look for any AI-specific controls below the main toggle.
TSH itself runs on Cloudflare and we ran through this check today. If you use any of the DNS Lookup or SSL Checker tools on this site, those run as Cloudflare Workers, same infrastructure, same dashboard. Worth a check across all your domains while you are in there.
Should You Block AI Crawlers at All?
The case for blocking Training crawlers is legitimate. AI companies are crawling publicly accessible content to train commercial models, generating revenue from that content without compensating the people who created it. A lot of site owners enabled “Block AI bots” in 2024 and 2025 for exactly that reason, and it was a reasonable call at the time.
The practical problem right now is that blocking Training means blocking Googlebot, and losing Google crawl coverage is a worse outcome for most sites than being included in AI training data. Until Google separates its search and training crawlers, or until Cloudflare ships per-crawler overrides that can distinguish between them, the choice is binary. For most small business sites, blogs, and hosting review sites, the pragmatic answer is to allow Search and accept that Training comes bundled with it for the time being. Revisit the setting when the tooling catches up.
For the broader picture on securing your site, bot management is one layer of a larger stack. Cloudflare’s bot controls are useful, but they work alongside your WAF rules, your update hygiene, and your hosting security setup rather than replacing any of them.
Common Questions About Cloudflare’s September 15 Change
Will Cloudflare block Googlebot on September 15?
Not automatically for everyone. Googlebot will be blocked only on sites where the owner has enabled “Block Training” or the legacy “Block AI bots” toggle. If you have never changed your AI bot settings, Cloudflare’s old default was to allow everything, so Googlebot is not at risk on those sites. If you have enabled Training blocking, check your settings today.
How do I check if Cloudflare is blocking Googlebot?
Before the fact: check your Cloudflare dashboard under Security → Bots for any Training blocking or legacy AI bot toggle that is enabled. After the fact: check Google Search Console for a drop in crawl activity under Settings → Crawl Stats. A sudden decline in pages crawled per day is the main signal.
What is the difference between Search, Agent, and Training crawlers in Cloudflare?
Search crawlers index your content to return traffic to your site later, the traditional SEO crawl. Agent crawlers act on behalf of a user in real time, such as an AI assistant browsing the web to answer a question. Training crawlers pull your content to train or improve an AI model. Cloudflare lets you allow or block each category independently.
Can I block AI training crawlers but keep Googlebot?
Not currently. Googlebot bundles Search and Training in a single crawler. Cloudflare’s rule applies the most restrictive setting to multi-purpose crawlers, so blocking Training blocks Googlebot too. Per-crawler overrides are coming from Cloudflare but are not live yet. For now, you have to choose one or the other.
Does this affect sites on Cloudflare’s free plan?
Yes. The new AI crawler controls are available to all customers including free tier. The September 15 defaults apply automatically to free-tier accounts that have not changed their bot settings. The Googlebot conflict also applies to free-tier accounts that have enabled Training blocking.
