TL;DR Over 3.9 days, more than twenty different AI agents hit wskpf.com; the busiest, Bytespider, made 763 requests and publishes no way to verify it. The user-agent field is something the visitor fills in itself: of GPTBot's 299 requests, 62 came from IPs that failed OpenAI's own published list. This note shows how to pull every AI agent from your own log, sort training from search from live user fetches, and check which ones are lying. Count lines, never matches, because a case-insensitive name match lets one bot swallow another, and a case-sensitive one quietly drops the odd variant.
I ssh into the box that runs this site the way some people check that they locked the front door - not out of any real suspicion, just an itch that will not settle until I look. I tail four days of the nginx access log, pipe it through a count I have typed so many times my fingers do it without me, and the name at the top of the list is not mine, and it is not Google's. It is Bytespider. ByteDance's training crawler, 763 requests over not quite four days, parked at the top of the heap above Googlebot like it is holding the lease and I am the one squatting.
I did not invite it. There is no version of events in which I invited it. And yet there it is, signed in, bold as brass.
An access log is a guest book. I mean that almost literally, and it took me an embarrassing stretch of this career to feel it in my gut rather than merely know it on a slide. Every request is a line, every line is a signature, and the detail nobody internalizes until a crawler is eating their bandwidth in the small hours is this: the guest writes the signature. There is no clerk at the door checking anything. The visitor walks up to the register, picks up the pen, and writes whatever name it fancies, and the server, being a gracious host with no sense of self-preservation, writes it down exactly as given and serves the page.
The Log Does Not Lie, But the User Agent Might
So before I can decide what to do about anything in that log, I have to do two separate jobs that everyone collapses into one. First, read what each visitor claimed to be. Then, much harder, work out whether the claim is true. The industry mostly does the first, calls it the second, and blocks or welcomes crawlers on the strength of a signature anyone could forge with a one-line change to a header.
Start with the claims, honestly and without flattering them. In a default nginx log the fields are separated in a way that looks hostile until you see it: split each line on the double quote character and the user agent is the sixth field, sitting out at the end where the browser - or the thing pretending to be one - wrote it. The visitor's address is at the very front, but it is not a bare number, it is IP - - [date] all glued together, so to get the actual IP you split that first chunk on spaces and take the first token. Miss that and you will spend an afternoon counting timestamps.
Here is the raw register, filtered to a handful of famous names and counted by the exact string each one signed:
awk -F'"' '{print $6}' /var/log/nginx/access.log | grep -E 'GPTBot|ClaudeBot|Claude-SearchBot|PerplexityBot|Bytespider' | sort | uniq -c | sort -rn | headOutput on wskpf.com, 2026-10-24
586 Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)
259 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
237 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)
153 Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)
80 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
52 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
23 Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
16 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +claudebot@anthropic.com)
13 Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/7.0 Mobile Safari/537.36 (compatible; Bytespider; https://bytedance.sg.larkoffice.com/docx/K5bxdypulop3IIxrJb0lOjLVgFe)
11 Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36Look at what that tells you before you count anything. Bytespider signs more than one way in my log - a toutiao.com URL, a bytedance.com feedback address, a larkoffice.com document link, another variant of that feedback address - and if I had grepped for one of those I would have undercounted it by hundreds. GPTBot signs as both GPTBot/1.4 and GPTBot/1.3 in the same window, because version bumps do not happen all at once. The signatures are messy because signatures are always messy. This is why you count lines, not matches, and why the way you count turns out to matter more than it has any right to.
Say you want a clean tally per agent. The obvious move is a case-insensitive grep for each name, and the obvious move quietly ruins your numbers in both directions. Grep case-insensitively for ClaudeBot and you will also catch every request from Claude-SearchBot, because that bot signs with +claudebot@anthropic.com in its string, and suddenly ClaudeBot's count is inflated by the 16 requests that belonged to a different robot with a different job. Tighten up and match on an exact token and you get the opposite failure, which the next command shows live:
awk -F'"' '{u=$6} u~/Bytespider/{b["Bytespider"]++} u~/meta-externalagent\//{b["meta-externalagent"]++} u~/GPTBot\//{b["GPTBot"]++} u~/ClaudeBot\//{b["ClaudeBot"]++} u~/PerplexityBot\//{b["PerplexityBot"]++} u~/Applebot\//{b["Applebot"]++} END{for(k in b)printf "%5d %s\n",b[k],k}' /var/log/nginx/access.log | sort -rnOutput on wskpf.com, 2026-10-24
763 Bytespider
361 meta-externalagent
299 GPTBot
266 ClaudeBot
103 PerplexityBot
67 ApplebotThat output says 361 for meta-externalagent. The true number of requests from that agent in this window is 368. The seven that went missing did not match the exact pattern my command expected, and a strict match dropped them on the floor without a word. Neither number is a disaster on its own. The disaster is not knowing which way your count is wrong, because then you are making blocking decisions on a figure you cannot defend. The honest tally, variants folded in, is the one in the chart, and 368 is the number I trust.
That chart is four days. Bytespider at 763, meta-externalagent at 368, GPTBot at 299, ClaudeBot at 266, and a long tail under them. More than twenty distinct AI agents hit this quiet consulting site in under four days, out of 54,362 total requests. I run this kind of log pass for traffic I actually want to understand, and the shape of it has changed under me: the register is now half full of guests nobody at the front desk recognizes.
If you want to run this on your own log, the two field positions are the whole game. The user agent is field six when you split on the double quote. The IP is the first space-separated token of field one, not the whole IP - - [date] chunk. Get those two right and every count in this note is a one-line awk away. Get them wrong and you will confidently count timestamps and quotation marks.
A Name Is a Claim, Not a Passport
A signature in the register is a claim, and a claim is not a passport. The uncomfortable follow-on is that there is no single passport office. If you want to confirm that the thing signing "GPTBot" is really OpenAI's crawler and not some scraper in a basement wearing its name, the method you use depends entirely on which operator you are checking, because the operators have not agreed on how to be checked.
Sort them into three families and the whole mess becomes tractable, as long as you let an operator sit in more than one when it offers more than one method. The first family publishes an IP-range file: a JSON document listing every network its crawler is allowed to come from, updated by them, fetched by you. OpenAI does this per agent - separate files for GPTBot and ChatGPT-User, and ranges for its search bot. Perplexity, Bing, Common Crawl and DuckDuckGo publish lists too. For this family, verification is arithmetic: is the visitor's IP inside one of these ranges or isn't it.
The second family tells you to ask the network itself. This is forward-confirmed reverse DNS, and it is older than most of the companies using it. You take the visitor's IP, run a reverse lookup, and check that the hostname that comes back belongs to the operator - Google documents that a real Googlebot address resolves to something ending in googlebot.com, google.com or googleusercontent.com. Then you forward-resolve that hostname and confirm it points back to the same IP. Both directions have to agree, which is the part that defeats a forger, because they can set a reverse record on an address they control but they cannot make Google's forward DNS point back at them. Google, Amazon, Apple and Huawei live in this family, and Bing lives in it too - Bing documents the reverse-DNS check and also publishes an IP-range file, so it belongs to both families at once, which is the point about not forcing these into tidy boxes.
The third family publishes neither. Anthropic is the headline name here: it does not publish IP ranges, and it says plainly that its bots run on public cloud-provider addresses, so blocking by IP may not work reliably and may even block the bot from reading your robots.txt. ByteDance, as far as any primary document I could find goes, publishes nothing you can check Bytespider against at all. For this family there is no verification. There is only the signature, and the signature is worth exactly what you paid for it.
The practical consequence: blocking an AI crawler on the user-agent string alone is theater for every family. For the first two it is unnecessary theater, because you can check the real thing. For the third it is the only move available, which is a polite way of saying the operator has decided its name is not your business to verify. Treat "no published method" as a fact about the vendor, not a gap in your skills.
Make the Robot Prove It
Enough taxonomy. Let me make some of these robots prove it, starting with one from the DNS family. Amazonbot signed into my log 109 times. To confirm it, I take the IPs that claimed to be Amazonbot, run a reverse lookup on each, and see where they land:
awk -F'"' '$6 ~ /Amazonbot\//{split($1,a," "); print a[1]}' /var/log/nginx/access.log | sort -u | head -3 | xargs -n1 dig +short -xOutput on wskpf.com, 2026-10-24
100-24-149-244.crawl.amazonbot.amazon.
100-28-204-82.crawl.amazonbot.amazon.
100-29-107-38.crawl.amazonbot.amazon.Every one of those resolves to a hostname under crawl.amazonbot.amazon, which is exactly where Amazon's documentation says a real Amazonbot should live. That is the first half of forward-confirmed reverse DNS; the honest full check then forward-resolves each of those names and confirms it points back to the IP it came from, which weeds out anyone who has set a convincing reverse record on an address they do not actually own. Done properly across the whole set, 92 of Amazonbot's 93 IPs passed, which worked out to 100 verified requests and 9 that failed.
Now the IP-list family. GPTBot signed in 299 times. First I pull the set of addresses it came from, counted so I can see the shape:
awk -F'"' '$6 ~ /GPTBot\//{split($1,a," "); print a[1]}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | headOutput on wskpf.com, 2026-10-24
216 74.7.227.180
36 185.230.113.52
21 74.7.227.182
16 34.47.8.83
10 45.45.237.65Five addresses, wildly unequal, one of them carrying 216 of the requests on its own. Then I fetch OpenAI's own published list and read off the ranges with my own eyes:
curl -s https://openai.com/gptbot.json | jq -r '.prefixes[].ipv4Prefix' | headOutput on wskpf.com, 2026-10-24
132.196.86.0/24
172.182.202.0/25
172.182.207.0/25
20.125.66.80/28
20.171.206.0/24
20.171.207.0/24
4.227.36.0/25
52.230.152.0/24
74.7.175.128/25
74.7.227.0/25Now compare by hand, which is slower than a script and far more convincing the first time you do it. The command only prints the first ten ranges, but checked against OpenAI's full published file, the two addresses on the 74.7.227 network - 74.7.227.180 with its 216 requests and 74.7.227.182 with 21 - are vouched for, while the other three are not. Those are real: 216 plus 21 is 237, and that is exactly the verified count. But 185.230.113.52 with its 36 requests, 34.47.8.83 with 16, and 45.45.237.65 with 10 appear nowhere in OpenAI's list at all. That is 62 requests signing OpenAI's name from addresses OpenAI does not claim. 237 verified, 62 failed, and the arithmetic falls straight out of the file.
The split is not evenly distributed across the names, and this is the part that surprised even me. Some operators come back spotless. PetalBot, Huawei's crawler, used two IPs, both passed reverse DNS, all 137 of its requests verified, not a single failure. But the two most trusted names in the business carry dirt: Googlebot had 61 requests from IPs that failed verification out of 208, and GPTBot its 62. The failures cluster, and the chart of who fails is its own kind of map:
Read that chart as the signatures you cannot trust even though they spell famous names correctly. Every one of those requests claimed a real crawler and came from an address the real crawler's operator does not vouch for. That does not prove malice - some will be rendering services, proxies, infrastructure nobody updated - but it absolutely proves you cannot treat the name as the identity, because here are hundreds of requests where the name and the identity come apart.
Three Jobs, Three Robots
Once you can tell the verified agents from the unverified ones, a second sort opens up, and it is the one that actually changes what you should do. The verified robots are not all here for the same reason. Lumping them together as "AI bots" and reaching for a block rule is like deciding how to treat a guest based on the fact that they arrived by car.
OpenAI alone runs three different agents for three different jobs, each with its own robots.txt token, so you can allow one and refuse another. GPTBot gathers content that may train the foundation models. OAI-SearchBot indexes the web for ChatGPT's search answers. ChatGPT-User fires when a live person in ChatGPT asks for something that needs your page right now. Three robots, three purposes, one company, and if you block all three because they all say OpenAI you have thrown out your shot at being cited in ChatGPT search to stop a training crawl you could have refused on its own.
Anthropic splits the same way: ClaudeBot collects content that may train its models, Claude-SearchBot works to improve search result quality, Claude-User retrieves a page when a person asks Claude a question that needs it. Perplexity is explicit in its own documentation that PerplexityBot is built to surface and link sites in search results and is not used to crawl content for foundation models at all. The brand on the signature tells you almost nothing. The purpose tells you everything.
And purpose is legible in behavior, not just documentation. ClaudeBot in my log fetched /sitemap.xml 41 times and /robots.txt 41 times - that pairing is the signature of something methodically mapping a site, reading the rules and the index of every page, which is exactly how a crawler that wants coverage behaves. You can see what a robot is for by what it asks for.
So the decision tree is not "AI, yes or no." It is: is this verified, and if so, what is it for. Block training if you have decided your writing should not become model weights without being asked - that is a coherent position and I respect it. But blocking a search indexer is a different act with a different bill: you are removing yourself from the answers people get when they ask an assistant a question your page could have answered. Those are not the same robot and they should not get the same verdict.
The robots.txt You Think Is Working Isn't
Here is where a comfortable assumption falls over. Most people believe robots.txt is a door with a lock. It is not. It is a sign on the door that says please, and the standard that defines it says so in nearly those words.
RFC 9309, which finally wrote the robots exclusion protocol down as an actual specification in 2022 after a quarter century of folklore, states that its rules "are not a form of access authorization." It is advisory. It governs the behavior of robots that have decided to be governed. And several agents in the live-fetch family have decided, out loud, in their own documentation, not to be.
OpenAI says of ChatGPT-User that "because these actions are initiated by a user, robots.txt rules may not apply." Perplexity says of Perplexity-User that "since a user requested the fetch, this fetcher generally ignores robots.txt rules." Meta says Meta-ExternalFetcher "fetches individual links at a user's request" and "may bypass robots.txt rules." These are not rogue scrapers sneaking past you. These are official agents of some of the largest AI companies alive telling you, in their published docs, that when a human presses the button your robots.txt is a suggestion they are free to decline. Not all of them make that choice - Anthropic says its Claude-User honors robots.txt - but enough do that you cannot assume the rule is holding.
My own robots.txt, for the record, allows everything - User-agent: * then Allow: / - so nothing here was blocked and then bypassed on my site. That is the point I want to land. This is about expectation, not enforcement. If you have a Disallow rule and you believe it is keeping every live-fetch agent off a page, open your log and check, because several of the agents whose job is answering a person's immediate question are documented to walk past the sign. robots.txt reliably stops the robots that chose to honor it - Anthropic's bots, Google's crawlers on automatic crawls, the polite training crawlers - and nothing that decided it would not.
The Impostors Go Looking for Your .env
Which brings me to the guests who signed famous names and then went straight for the safe.
There is a tell that separates a probable impostor from a real crawler, and it is not subtle once you know to look. A real crawler, overwhelmingly, asks for your pages. Requests for your secrets fit a different profile: something wants /.env and /.git and key.pem and .pgpass and the dozen other files where a careless developer leaves credentials lying around, it gets a face full of 404s because those files are not there, and it moves on to the next site. No operator in this entire note documents its crawler requesting a secrets file. Not one. So when a request for /.env arrives signed by a famous AI name, the signature and the behavior pull against each other, and that tension is worth a second look rather than a shrug.
Here is the probe hunt in a single pass, filtered to agents that have no business asking for any of this:
awk -F'"' '($6 ~ /ChatGPT-User|cohere-ai|YouBot|Claude-SearchBot|meta-externalfetcher/) && ($2 ~ /\.env|\.git|\.pem|cred|secret|key/){print $2}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | headOutput on wskpf.com, 2026-10-24
2 GET /.env.development HTTP/1.1
2 GET /.env HTTP/1.1
1 GET /userfiles?path=../../../.env HTTP/2.0
1 GET /static//home/user/.env HTTP/2.0
1 GET /static//app/.env HTTP/2.0
1 GET /settings%2F.env HTTP/2.0
1 GET /services/.env HTTP/2.0
1 GET /secrets.yml HTTP/2.0
1 GET /secrets.json HTTP/2.0
1 GET /project/.env HTTP/2.0Those requests came in wearing the names ChatGPT-User, Claude-SearchBot, cohere-ai and meta-externalfetcher. ChatGPT-User's top pages in my log include /.env and /.env.development; it made 14 requests for secret or config files, and zero of them came from an IP that passed OpenAI's verification. Claude-SearchBot returned almost nothing but 404s - 15 of them and a lone 405 - and 10 of its 16 requests were for secret or config files, with no way to verify any of it. cohere-ai asked for /.pgpass and /.s3cfg and a path with /@fs/.env in it, 12 requests, all 404, user-agent the only thing it offered. Even GPTBot's nine secret-file requests came entirely from the unverified 62.
This is where sorting your log stops being hygiene and becomes security. For every one of these agents that I could verify at all, the config-file probes came from IPs that failed that verification - not one of them from a verified IP; for the agents with no published verification method, there is nothing to check against in the first place. That is the pattern to flag: a request for a secrets path, from an address the named operator does not vouch for or cannot be checked at all, returning a 404. It is consistent with impersonation rather than any documented crawler behavior, and reading it as "an AI bot crawled me" is exactly the misreading the forged signature invites.
What to Do With What You Found
So you have the log sorted: verified from unverified, training from search from live fetch, the pages a crawler wants from the secrets nothing legitimate should be asking for. Now the decisions, which are smaller and calmer than the sorting made them feel.
If you want out of model training, the instrument is not a block rule aimed at indexers you actually want. It is the control tokens built for exactly this: Google-Extended, which Google introduced in September 2023, and Applebot-Extended. Neither is a crawler - they issue no requests, which is why not one line in my entire log carried either name - they are switches in robots.txt that tell Google and Apple not to use your crawled content for training while leaving your search presence intact. That is the surgical version of the decision. Block the use, not the visit.
Resist two temptations. The first is llms.txt, the proposed file for telling models about your site. I have one - wskpf.com/llms.txt returns a clean 200 as I write this - and I want to be straight about what it is: a proposed format, not a ratified standard, and no major model provider has formally confirmed that it reads the thing. I keep mine because it costs nothing and the day someone starts consuming it I would like to already be there. I do not keep it under any illusion that it is doing work today.
The second temptation is to believe all this crawling means the traffic is coming. Across these same four days, the number of visits to my site actually referred by an AI assistant, counted off the Referer header, was none. Zero. The robots are here in force - 763 requests from one training crawler alone - and the humans they supposedly feed have sent me nothing back yet. I have written before about making your pages legible to machines, and I stand behind every word, but legibility is a bet on what is coming, not a report on what has arrived. The crawl is real. The referral is a promise.
A repeatable monthly pass, then: count every AI agent by exact token, folding in case variants so your number is honest. Verify each one by its operator's own method - IP list, forward-confirmed reverse DNS, or the admission that there is none. Sort the verified ones by purpose and decide training versus search per bot, not per brand. And flag every request for a secrets path from an unverified or uncheckable IP, because that behavior is consistent with someone trying the doors rather than with any crawler anyone documents.
I keep coming back to the thing I started with and cannot get out from under. The user agent is the one field in the log the visitor writes itself, which is exactly why every request for /.env that came in wearing a famous AI name on my site came from an address that name's owner does not vouch for, or one nobody can check at all.