Maybe you should not have blocked AI bots crawling your site. But sometimes it’s the right call. What should you do?
You read a headline about AI companies scraping websites to train their models without asking. It feels like theft, so you block AI bots from your site. Or maybe your web host already did it for you. The trouble is that the same switch can also pull your business out of AI search, which is where a growing share of customers now start looking.
Here is the short answer to “should I block AI bots from my website?”: not all of them. Some AI bots learn from your content to help train AI models, and blocking those is a fair choice that does not hurt your Google ranking. Others fetch your page in real time to answer a customer’s question and link back to you. Block those and you disappear from AI search results. Many small business sites are already blocking everything by default, often without anyone choosing to, so the first job is to find out which bots you are blocking.
We build websites for Melbourne businesses at CJ Digital, and over the past year this mix-up has become one of the most common we see. So here is the plain-English version: which AI bots are worth blocking, which to leave alone, and how to check what your site is doing right now.
The instinct to block is fair, and it helps to say so plainly. When an AI company trains its model on your content, it usually pays you nothing and credits you nowhere. Your words help build a product you do not own. Cloudflare, which sits in front of a large share of the web, has reported that some AI crawlers make tens of thousands of requests for every visitor they send back. So you carry the cost of being crawled and get almost nothing for it. If that bothers you, you are not being precious. It is a real concern, and you are allowed to act on it. The mistake is treating every AI bot as the same thing.
“AI bots” is not one thing. Two very different types of bot visit your site, and they want different things.
The first type is a training bot. It copies your content to help teach an AI model, the way a student might copy passages into a textbook they are writing. It sends you almost no visitors. Examples include GPTBot (OpenAI’s training crawler) and Google-Extended (the control for whether your content helps train Google’s Gemini models).
The second type is a search or answer bot. When someone asks an AI tool a question, this bot fetches your page right then, reads the relevant part, and can point the person to you with a link. This is how your business gets named in an AI answer. The clearest example is OAI-SearchBot, the bot behind ChatGPT’s search feature.
This is the key point: these are now run as separate bots. So you can block the training bots and still let the search bots in. You can protect your content from model training and stay visible in AI answers at the same time. You do not have to pick one.
Block a training bot and the cost is small. You stop your content feeding AI models, you lose almost no traffic (these bots barely send any), and it does not affect where you rank on Google. Google’s own guidance is clear that its training control, Google-Extended, is separate from normal search. Publisher network analysis reviewed by the ad-tech firm Playwire found that blocking the main training bot had no measurable effect on Google rankings. So blocking training bots is a clean intellectual-property decision with little downside.
Block a search or answer bot and the cost is the opposite. You remove yourself from AI answers. Cogni, a firm that tracks AI citations, found that sites which blocked Perplexity’s search bot dropped to roughly zero citations in that engine within about two days. For a business that wants to be found, that is a self-inflicted wound. You are not protecting your content. You are hiding your shopfront.
This is the part that catches people out. You might be blocking AI bots without ever having made the choice.
Most of this is controlled by a small file called robots.txt. It is a plain text file on your website that tells bots which parts of your site they are allowed to visit. Whoever built or manages your site can see it.
The bigger factor is Cloudflare, the company that sits in front of roughly one in five websites and helps speed them up and keep them safe. Since 1 July 2025, every new site added to Cloudflare blocks AI crawlers by default. That means a large number of businesses are invisible to AI search not because they decided to be, but because that was the setting on the day their site went live. When we build a site, we set this on purpose rather than leaving it to a default, which is part of what we mean by an “AI Search Ready” build. If you are not sure what your site does, that uncertainty is worth chasing down.
To check, whoever manages your website can look in the Cloudflare dashboard under the AI crawler controls, or read your robots.txt file. It is a setting, not a rebuild.
More and more people now begin a search by asking an AI tool a question instead of typing into Google. Marketers call this the “top of the funnel”, which is jargon for the very first step, before someone has picked who to call. In plain terms: a person asks ChatGPT “who’s a good bookkeeper near Brunswick?” or “reliable conveyancer in the south-east suburbs”, and the tool hands back two or three names with links. If your business is not in that answer, you are not even on the shortlist. The customer never sees you.
In a market as crowded as Melbourne, that shortlist is everything. For most local trades and services there are dozens of competitors within a few suburbs, and the businesses that show up in the AI answer get the first look. We have spent more than 12 years building sites for Australian businesses, and the pattern is the same as it was when Google search took over: the ones who set up early to be found win the easy ground while everyone else catches up.
For most small businesses that want more customers, the sensible default is simple. Let the search and answer bots in so you stay visible in AI answers, and make a separate decision about whether you care about blocking the training bots. That keeps you in the running while you keep control of whether your content trains models.
This is usually a quick settings change, handled by whoever looks after your website. It is not a new site and it is not a big project.
One thing you can skip: Cloudflare’s “pay per crawl”, which lets a site charge AI companies to crawl it. That is aimed at large publishers with huge traffic, not at a local business trying to get found. For a small business it is a distraction.
No. Blocking AI training bots has no measurable effect on where you rank in Google search. Google’s training control is separate from normal search, so you can block it without touching your rankings.
It depends which bot. Training bots use your content to help build AI models, which is the concern most owners have. Search and answer bots only fetch your page to answer a live question and send people back to you. You can block the first and allow the second.
Ask whoever manages your website to check two things: your robots.txt file and your Cloudflare AI crawler settings. If your site is newer and runs on Cloudflare, there is a real chance AI bots were blocked by default.
Generally no. It is built for large publishers who can charge AI companies for access. A local business is better served by being found and cited than by putting up a paywall to bots.
If your goal is to be found, allow the search and answer bots, such as OAI-SearchBot for ChatGPT. Then make a separate, deliberate choice about the training bots, based on how you feel about your content being used to train AI models.
AI search is only going to take more of that first step away from Google, so the gap between businesses that show up and businesses that do not will keep widening. The cheapest time to fix an accidental block is before a competitor fills the space you left empty.
So here is the single thing to do this week. Ask whoever manages your website one question: “Are AI search bots allowed to reach our site?” If they cannot answer quickly, treat that as a no, and get it checked properly.
If you would rather have someone confirm it for you, that is something we do. CJ Digital can check whether your site is reachable by AI search bots and set up to be cited, then tell you in plain terms what to change. The detail is on our AI search optimisation page.
