We pulled fifteen days of raw access logs off this site — 3.1 million lines, 14 to 28 August 2026 — and counted who was actually knocking. The answer was not what we expected, twice over.
First: AI agents now request pages from this site three times more often than Google, Bing, Apple, Baidu, Yandex and DuckDuckGo put together. Second, and more useful: a large share of that “AI” traffic is not AI at all. It is scanners wearing an AI crawler’s name because that name gets waved through.
Both facts change what you should do about it, and neither is visible in Search Console.
What is actually hitting the site
The individual names are where it gets interesting. Googlebot made 2,036 requests. Two AI crawlers each made more: Amazonbot 3,257 and Meta’s meta-externalagent 3,180. GPTBot (1,720) and ClaudeBot (1,630) both out-requested bingbot (996). In bandwidth the gap is wider still — the AI crawler group pulled 626 MB against 183 MB for the search engines.
The smallest group in that chart is the one worth staring at. Those 2,172 requests are not crawlers building an index. They are ChatGPT-User, OAI-SearchBot, Claude-User, Perplexity-User — the fetches an assistant makes because a person just asked it something. Every one of those is a human being, mid-question, whose answer is about to be assembled from whatever your server hands over.
Then we looked at what they got back
A crawler request is not just a name; it is a name, a path and a response code. Splitting the same traffic by what the server actually returned turns the tidy picture above into something much stranger.
Legitimate crawlers behave like legitimate crawlers. Bytespider was served on 97% of its requests, bingbot 96%, Applebot 95%, meta-externalagent 91%. They ask for pages that exist, using the site’s hostname, and they get pages.
Then there is GPTBot, which got nothing back 85% of the time. And “Googlebot”, which sent 56% of its requests to the server’s bare IP address rather than to musthaveplugins.com — nginx answers those with a 444, closing the connection without a response, because a request that does not name the site cannot be Google. Real crawlers resolve your hostname. Scanners sweep IP ranges.
The smoking gun is in the paths
You do not need clever analysis for this part. Just look at what each identity asked for.
OpenAI’s crawler does not want your .aws/credentials, your .git/config or your rclone.conf. Whatever sent those requests picked “GPTBot” for the same reason a burglar picks a hi-vis vest: it is the identity least likely to be stopped. ClaudeBot in the same window asked for the sitemap, then robots.txt, then articles — which is what crawling looks like.
Note that this is not a clean split between good names and bad names. ClaudeBot’s tail includes a handful of /@fs/proc/self/environ probes too. Every popular crawler identity has some impostor traffic riding on it; the ratio is what differs.
How to tell the difference on your own site
Three signals, in increasing order of effort.
- Response codes. Group your log by user agent and count what you returned. A genuine crawler sits above 90% served. Anything below half is mostly not what it says it is.
- Requested paths. Sort each agent’s top paths. Dotfiles,
wp-config, cloud credential files and/proc/self/environare never crawling; they are scanning. - Published IP ranges. The real verification. OpenAI publishes its ranges as JSON for each of its agents; Google is verified by forward-confirmed reverse DNS to
googlebot.com. Check the requesting IP against the list for the name it claims.
A word of warning on that third one, learned the hard way here: it only works if your logs record the visitor’s real IP. This site sits behind a CDN, and for a large share of requests the client-IP field in our own nginx log holds an edge address rather than the originating one — which makes IP verification inconclusive on exactly the requests you most want to check. If you plan to make blocking decisions from logs, fix the real-IP logging first. Everything above about response codes and paths still works regardless.
The distinction that should drive your decision
Most advice on this topic collapses into “block the AI bots”, which is the wrong shape of answer, because two very different things share the label.
| Crawlers that build a corpus | Fetches triggered by a person | |
|---|---|---|
| Examples | GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent, Applebot-Extended, Google-Extended | ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Perplexity-User, DuckAssistBot |
| What it is doing | Reading your site to train or index a model | Fetching your page to answer someone’s question right now |
| What blocking costs you | Little directly — your content is simply not in that corpus | You disappear from the answer. The person asked about you and got nothing |
| Reasonable default | A deliberate choice, either way | Allow, unless you have a specific reason not to |
A single User-agent: * block, or a copied-in list from a blog post, treats those two cases identically. That is how sites end up invisible in AI answers while still being scraped by everything that ignores robots.txt.
What to actually do
1. Measure before you decide
You cannot make a sensible call about traffic you have never counted. The numbers in this article came from raw nginx logs, which is the most direct source and available on any server you control. If you would rather not live in a terminal, Must-Have Analytics keeps a bot traffic report and recognises AI platforms as a traffic source, so you can see both halves — the crawlers arriving, and the human visitors that AI assistants send back to you. The second number is the one that tells you whether any of this is paying off.
2. Write robots.txt as two lists, not one
Decide separately for corpus crawlers and for user-triggered fetchers, and write the file so a future reader can see that you decided rather than copied. If you are blocking training crawlers, name them individually — GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended, Applebot-Extended, meta-externalagent — and leave the user-triggered agents allowed.
For the record, this site currently says nothing about AI crawlers at all: they are all allowed. That is a position, and after seeing these numbers it is one we will revisit — but it is a deliberate one, not an accident.
3. Remember robots.txt is a request, not a lock
This is the part the numbers above make unavoidable. Everything pretending to be GPTBot to look for your AWS keys is, by definition, not reading your robots.txt and honouring it. Politeness only works on the polite.
If content theft rather than model training is your actual worry, that is an enforcement problem and needs an enforcement tool — server or firewall rules, or a plugin like WPACP, which adds source protection and can block bots and AI scrapers at the content level rather than asking them nicely. Decide which problem you have before you pick a tool: “I do not want to be in a training set” and “people are lifting my articles wholesale” call for different things.
4. Be readable to the ones you want
If you have decided assistants should be able to answer questions about you, make that easy. Clean HTML with real headings beats a page that only renders after JavaScript. Structured data still helps. And an llms.txt at your root — a short markdown summary of what the site is and where the important pages live — gives a fetching agent the summary you would have written yourself instead of one it infers from your navigation. We publish one here.
This is the half of “AI SEO” that is actually SEO: not gaming a ranking, just being legible to something that reads quickly and cites what it can parse.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
Not by itself — they are different agents. GPTBot is the corpus crawler; ChatGPT-User and OAI-SearchBot are what fetch a page when someone asks a question. Blocking all three does remove you from the answer. This is exactly why the two lists should be written separately.
Is all this crawling slowing my site down?
On this site, 626 MB over fifteen days is real but not alarming. It is worth checking on a small server or an image-heavy site, where a crawler that ignores your sitemap and walks every variation of every URL can cost more than the content is worth. Response codes tell you quickly whether it is crawling or thrashing.
Should I block the spoofed traffic?
You mostly already are, without meaning to — those requests are getting 404s and dropped connections because they are asking for files you do not have, at a hostname you do not answer to. The useful action is not blocking it but excluding it from your numbers, so you are not making decisions about “AI crawler load” that is really a credential scanner.
Where do I even find my access logs?
On a VPS, usually /var/log/nginx/access.log or /var/log/apache2/access.log, with rotated copies beside them. On shared hosting, look for a “raw access logs” download in the control panel. If your host does not provide them, a plugin-side bot report is your alternative.
Do these numbers apply to my site?
The ratio will not. This is one WordPress site in one niche over fifteen days, and crawler interest varies enormously by topic, size and age. The method transfers exactly, and so does the finding that identity claims need checking. Run the same count on your own logs before acting on anyone’s numbers, including ours.
The takeaway
Two things are true at once, and most advice picks only one. AI agents really have become the majority of non-human traffic — on this site, three requests for every one from a classic search engine, and a growing slice of it is a person waiting for an answer about you. And the log entry that says GPTBot frequently is not GPTBot, which means the first move is never blocking. It is counting, then checking, then deciding.
The whole exercise took one pass over fifteen days of logs. If you have never done it for your own site, that is the highest-value hour of SEO work available to you right now — not because of what you will block, but because of what you will find out you were wrong about.
If you want to keep an eye on it without living in a terminal, the bot traffic and traffic-source reports in Must-Have Analytics cover both directions, and we have written before about connecting those reports to an AI assistant, about what all those 404s do to your crawl budget, and about where WordPress sits in the AI era. For verification, OpenAI documents its crawlers and their IP ranges, and Google documents how to verify Googlebot.
