TL;DR My own wskpf.com logs ran 23 different bot user agents over four and a half days, 21 of them AI, from Bytespider hammering 839 times down to Claude-User managing 7. A user agent string is cheap to fake, so I checked every one my verifier had an IP list or a reverse-DNS method for, and in my log a chunk of those flunked. 62 of GPTBot's 312 requests came from IPs that were not on OpenAI's published list when I checked, which means either OpenAI forgot to list some of its own machines or something out there is wearing a GPTBot costume, and I know which way I'd bet. I'll hand you the exact grep, awk, host and curl commands I used to pull the AI bots out of my log, sort them into training, search and live fetch, and check which of them are who they say they are.
I ran one command, and it told me a lie, which is not the command's fault, the command is a good boy, the command did precisely what I asked, I asked it to count the lines in four and a half days of this website's log that say GPTBot and it counted them and handed me a number with the eager loyalty of a golden retriever bringing you a dead bird, and that number deserves the look you give a man on a regional train who has leaned across the aisle, uninvited, to inform you that he is a federal air marshal and also that the government is in the ceiling.
Everyone Signs Their Own Name
grep -c 'GPTBot' /var/log/nginx/access.logOutput on wskpf.com, 2026-10-25
312Three hundred and twelve requests that say they are OpenAI's training crawler. Says. That word is squatting there holding the entire load on its skinny little back and the whole industry walks past it every day pretending not to see it, the way you walk past a man selling individual cigarettes outside a hospital. A user-agent string is a text field. It is typed by whoever sent the request. There is no gate on it, no checksum, no photo ID, no man in a booth squinting from the string to your face and back. I could open a terminal right now and fetch your homepage calling myself GPTBot, or Googlebot, or Pope Benedict XVI, Emeritus, and your log would write it down with the trusting little heart of a hotel guestbook that has never once suspected that "Mr. and Mrs. Smith" might not be their real names.
Which is the only reason I need to tell you about Francis.
Let me tell you about a guy I invented for the purposes of this note. His name is Francis Dunmore and Francis works the door of a members-only cigar lounge on the second floor, above a vape shop, and Francis has a system, and he would like you to know that it is airtight.
Francis has a laminated clipboard he does not read. He has a velvet rope, except the velvet rope is a retractable dog leash clipped between two orange traffic cones he took from a road. And he has a rubber stamp that says VERIFIED in blue ink, which he bought online for eleven dollars and forty cents including shipping, and which he presses onto the forearm of every human being who climbs the stairs and says a name at him.
That is the whole apparatus. You come up the stairs, you say "I'm on the list, I'm Dave's cousin," and Francis says "welcome back, big guy," stamps VERIFIED on your forearm, and unclips the dog leash. He has never asked for ID. He has never called Dave. There is no Dave. In six years he has stamped, by his own count, which he keeps in a spreadsheet with conditional formatting that turns the busy nights gold, a little over forty thousand forearms, and if you make the mistake of pausing near him he will tell you, with the quiet pride of a surgeon, that he runs the tightest door in the city.
Francis is your analytics dashboard, he's every "AI traffic report" ever emailed to a client, and on a bad day before the caffeine crosses the blood-brain barrier he's me, and 312 is exactly the size of number that ends up on slide four of a deck with a stock photo of a robot shaking hands with a businessman.
Counting, the actual craft of it, is more treacherous than it looks, so let me walk you past the two open manholes before we both drop into one. That grep -c counted whole lines, which is correct. The second you get greedy and reach for grep -o to count every occurrence of the word, you start double-billing yourself, because bingbot writes its own name twice in a single breath, once as the agent and once in the +http://www.bing.com/bingbot.htm it staples to the end like a man who signs his emails with his own name twice, so one request becomes two in your tally and congratulations, you have manufactured traffic that never arrived, you are now reporting ghosts. Go lazy the other direction and match case-insensitively and you fold Claude-SearchBot and Claude-User straight into ClaudeBot, because all three Anthropic agents carry the same contact, claudebot@anthropic.com, baked right into the string, so a careless grep -i claudebot scoops up the search bot and the live fetcher and files all three under training. That is three distinct men in one trench coat and you have counted them as a tall man. Match the token the operator actually uses, count lines and not matches, and when you want many bots at once, stop grepping the whole line and reach into the field where the name actually lives.
Split an nginx combined-log line on the double-quote character and the sixth chunk is the user-agent, the second is the request. So this walks the log once, prints one tidy label per line it recognises, and lets sort | uniq -c do the arithmetic. Case-sensitive on purpose, so the Anthropic trio stays three men.
awk -F'"' '$6~/Bytespider/{print "Bytespider"} $6~/meta-externalagent/{print "meta-externalagent"} $6~/GPTBot/{print "GPTBot"} $6~/ClaudeBot/{print "ClaudeBot"} $6~/Googlebot/{print "Googlebot"} $6~/PetalBot/{print "PetalBot"} $6~/Amazonbot/{print "Amazonbot"} $6~/PerplexityBot/{print "PerplexityBot"} $6~/OAI-SearchBot/{print "OAI-SearchBot"} $6~/ChatGPT-User/{print "ChatGPT-User"} $6~/bingbot/{print "bingbot"} $6~/Applebot/{print "Applebot"} $6~/YouBot/{print "YouBot"}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -12Output on wskpf.com, 2026-10-25
839 Bytespider
413 meta-externalagent
312 GPTBot
302 ClaudeBot
229 Googlebot
149 PetalBot
129 Amazonbot
105 PerplexityBot
97 OAI-SearchBot
74 ChatGPT-User
72 bingbot
71 ApplebotBytespider at the top with 839, ByteDance yanking on this site like it cosigned a loan, then Meta, then OpenAI's training crawler, then Anthropic's, and down the tail the Applebots and YouBots mill around. The chart lays out the eight biggest AI crawlers (Googlebot sits this one out, the run files it under search) so you can feel the shape of the mob. If you have never once opened your own log and this already feels like being handed a chainsaw at a dinner party, start gentler, over here: the log-file analysis shortcut.
While we're down here counting, my field-six tally says 413 for meta-externalagent and the run's own total says 420, and I have not tracked down which seven lines the awk walked past, so the chart uses the run's number and I'm telling you so. I only trust a number when I can say how I got it out loud without blushing, and "I matched the token in the user-agent field and counted lines" survives being said in a meeting while a man from procurement stares at you. If your current AI-bot reporting boils down to "I ran grep on a word", you are Francis, the stamp is out, the dog leash is unclipped, and you don't even know it.
Do not count AI bots with one loose all-purpose grep -ic. Case-insensitivity merges separate agents that share a contact address, and occurrence counting double-bills any bot that repeats its own name in its UA. Match the exact token, count lines, and keep the operators who run multiple agents apart from the first keystroke.
Three Jobs Wearing the Same Logo
Francis walks face-first into the wall he built here, because to Francis a logo is a guy and a guy gets one stamp. OpenAI comes up the stairs and Francis sees OpenAI, singular, one fella, when OpenAI is in fact three separate men in three separate coats who share a surname, a cloakroom, and a lawyer.
awk -F'"' '$6~/GPTBot/{print "GPTBot"} $6~/OAI-SearchBot/{print "OAI-SearchBot"} $6~/ChatGPT-User/{print "ChatGPT-User"}' /var/log/nginx/access.log | sort | uniq -cOutput on wskpf.com, 2026-10-25
74 ChatGPT-User
312 GPTBot
97 OAI-SearchBotGPTBot is the training crawler, the one eating pages to fatten the models. OAI-SearchBot is the one that surfaces and links your site inside ChatGPT's search. ChatGPT-User is the live fetcher that fires the instant a human pastes your URL into the chat or a Custom GPT Action goes and grabs a page in real time. OpenAI's own docs give each a separate robots.txt token, which is the entire point, because it means you can wave one through the dog leash and send another back down the stairs. Anthropic runs the identical three-coat racket with ClaudeBot for training, Claude-SearchBot for search, and Claude-User for the fetch when a human actually asks Claude something. Perplexity splits PerplexityBot, which it swears surfaces and links sites and is not used to train foundation models, from Perplexity-User, which goes and reads a page to answer a live question. Mistral runs three of its own. Each logo is really a small corporate department, with an org chart, and you have been stamping the whole department's forearm as if it were an intern.
And this is the part the slogan crowd never gets within a mile of, the ones who want to BLOCK THE AI BOTS in one triumphant line and move on, because one line in robots.txt cannot govern all three of these men, they don't even obey the same rules. OpenAI rewrote its own documentation to peel the robots.txt-compliance language off ChatGPT-User, so the live fetcher goes where it is pointed whether your robots.txt weeps about it or not. You can disallow the training crawl and keep the search listing, or block the search bot and watch your visibility in that engine's results go wherever unloved websites go, and those are genuinely different decisions wearing the same logo, which is the exact thing Francis never grasped about the thirty-one separate men who all told him they were Dave's cousin.
Before you write one robots.txt rule for a vendor, list every agent that vendor runs and what each does. OpenAI, Anthropic and Mistral each run separate agents for training, search indexing and live user fetches, and Perplexity splits its search bot from its live fetcher.
Ask the Doorman, Not the Guest
Francis finally had the thought that should have detonated in his skull on day one, somewhere around forearm number nineteen thousand, which is that the person shouting the name and the person who could confirm the name are two different people, and he had spent six years asking the wrong one. The guest is a claim with legs and a borrowed tuxedo, and the man who keeps the list is behind the bar, where Francis has never once gone, because the bar is where they keep the lime wedges and Francis has a thing about lime wedges.
Operators hand you one of three ways to check a claim. Some publish their IP ranges as a file you fetch at run time, the real guest list, kept where the guest can't edit it. Some document a reverse-DNS method, where you take the IP, look up its hostname, confirm the hostname is theirs, then look that hostname back up and confirm it resolves to the same IP, which is calling the limo company and asking if the plate matches the booking before you believe the man in the tuxedo. And some publish nothing, which means the claim is unverifiable and you treat it the way you treat a man who says he's a doctor on an airplane and then asks what a spleen is. Google documents all three signals and tells owners flatly to verify, because the Googlebot user agent, in Google's own words, is often spoofed by other crawlers. Google, the guest, is warning you about the other guests, which has never once happened at Francis's door.
Pull a bot's source IPs out of the log first. Take the first space-delimited token of field one, which is the IP, not the whole timestamp lump, then reverse-DNS each. PetalBot documents that its legit IPs resolve under petalsearch.com or aspiegel.com, so it's the clean case, the one guest who actually is Dave's cousin.
awk -F'"' '$6~/PetalBot/{split($1,a," ");print a[1]}' /var/log/nginx/access.log | sort -u | xargs -n1 host | headOutput on wskpf.com, 2026-10-25
86.136.119.114.in-addr.arpa domain name pointer petalbot-114-119-136-86.petalsearch.com.
177.159.119.114.in-addr.arpa domain name pointer petalbot-114-119-159-177.petalsearch.com.Both point at petalbot hostnames under petalsearch.com, so two of two verify, and PetalBot's 149 requests are exactly what the man said they were. Now the IP-list case. GPTBot's ranges live as JSON at a published endpoint, so you fetch it and eyeball the prefixes you'll check your log against.
curl -s https://openai.com/gptbot.json | jq -r '.prefixes[].ipv4Prefix' | headOutput on wskpf.com, 2026-10-25
132.196.86.0/24
172.182.202.0/25
172.182.207.0/25
20.125.66.80/28
20.171.206.0/24
20.171.207.0/24
4.227.36.0/25
52.230.152.0/24
74.7.175.128/25
74.7.227.0/25And here the room gets cold, because against that list only two of GPTBot's five source IPs verified, meaning 250 of its 312 requests came from OpenAI's declared ranges and 62 did not. Googlebot verified 22 of 30 IPs, leaving 76 requests from addresses Google does not claim as its own children. PetalBot came through clean at two of two, Amazonbot at 106 of 107, which is about as honest as anyone gets in this neighbourhood. The chart below ranks the failed requests for six of the big names, and read the roll call, because the names failing are the marquee ones, Googlebot, PerplexityBot, GPTBot, ChatGPT-User, OAI-SearchBot, bingbot. Every one of them dragging traffic in from IPs the operator itself will not vouch for, and for PerplexityBot, ChatGPT-User and OAI-SearchBot the addresses that failed were the majority of the traffic, which is a lot of men in tuxedos who came up the fire escape.
My ego, already a bruised and underperforming organ, would like me to skip this next part, which is how you know it's the most useful thing in the note. My run reported ClaudeBot as unverifiable, filed under "operator publishes no IP list and no DNS check", and that was wrong. It was wrong because my verifier didn't know the right file existed. Anthropic does publish its crawler ranges, as JSON, at claude.com/crawling/bots.json, covering all three coats. I fetched it myself the day of the run and it was dated 2026-10-20 and listed 14 IPv4 prefixes, the first being 216.73.216.0/22, and my code now checks the Anthropic trio against it. Your checker only knows the files you've taught it about, so it stays a little wrong until the next time you trip over an endpoint you didn't know was behind the bar.
Keep a small table of where each operator publishes its check, whether that's a JSON IP file, a reverse-DNS mask, or nothing. Fetch the IP files at run time instead of hard-coding ranges, because these lists move under you while you sleep. When an operator publishes no check, you are stuck with an unverifiable claim, and you write that down as unverifiable, in those words, instead of believing the string anyway.
The Impostors Go Straight for the Keys
Once you can sort verified from unverified, look at what the unverified half actually does, and I need you to brace a little.
awk -F'"' '$2 ~ /\.env|\.git|\.pem|credentials|pgpass/ {print $2}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | headOutput on wskpf.com, 2026-10-25
11 GET /.git/config HTTP/1.1
8 GET /.env HTTP/1.1
6 GET /.git/objects/ec/20d85eb3319ef5fdba268865bff9dda2a033dd HTTP/1.1
5 GET /.git/objects/fd/04813566492b36220f6bfcfb0f70c0f39c710a HTTP/1.1
5 GET /.git/objects/dc/28260e95bd0620290f61e75a55b6ab95f5b0b2 HTTP/1.1
5 GET /.git/objects/c9/06b8e0a2c810c4da646159ac2a832ba6bda4e2 HTTP/1.1
5 GET /.git/objects/c7/a115bc3f5618d867296277bdc0d6bc45ac2686 HTTP/1.1
5 GET /.git/objects/b8/359e1e306cac429996e66c5acac05422527ee9 HTTP/1.1
5 GET /.git/objects/a5/ae39b8caab75bca9706fe006d8d6643ad8d876 HTTP/1.1
5 GET /.git/objects/8a/fcfe5593fea33bc4a9ddf21ae25d5e49cf3205 HTTP/1.1That is a man going through your pockets. Eleven hits on /.git/config, eight on /.env, and a long patient crawl through /.git/objects/ hashes, which is someone trying to read your version-control internals one brick at a time from the alley. Across many of the AI labels in the log it's a similar shopping list, /.env and /.env.production and /key.pem and /.pgpass, like a burglar who has read exactly one blog post. ChatGPT-User's top pages included /.env and /.env.development, fourteen secret-file requests under its name. Claude-SearchBot sent sixteen requests total, every single one a 404 or a 405, poking at /panel/.env and /.env.stage, ten of them straight at secrets, and not one actual page of content. cohere-ai sent twelve, all 404, at /.pgpass and /@fs/.env and /.s3cfg. Meta-ExternalFetcher sent nine, all 404, at /@fs/app/.env.production and /store/.env.
In every bot the run was able to verify, I counted the secret-file probes that came from a verified IP, and the count was a big fat round
Zero.
Where I could check an address, the real operators were reading my pages from their declared ranges like gentlemen, and the credential-rifling arrived under the same famous names from addresses that failed the check, or under names nobody publishes a check for at all.
Careful now, because I would like to keep my lawyer bored. This looks a lot like impersonation riding a famous name, and I can't pin it on OpenAI or Anthropic or Cohere themselves, because these strings are among the most-spoofed on the web and the operators say so out loud. Common Crawl states plainly it is aware of crawlers falsely identifying as CCBot. Cloudflare accused Perplexity of switching to a generic Chrome string to fetch content its declared bots were blocked from. ByteDance lists ten address blocks for Bytespider and the monitors keep catching it arriving from IPs outside them. Because anyone can type GPTBot, I can't pin this scanning on the named lab either. What you can say, and should, loudly, is that a request wearing a billion-dollar AI logo, arriving from an IP that billion-dollar AI does not claim, going straight for your .env file, is in all likelihood a guy in a costume, and the costume is good, and the guy is not here for the cigars.
Treat any secret or config-file probe as hostile regardless of the user agent painted on it, and never allowlist traffic by name. If you are waving "AI bots" through a firewall rule based on the string alone, you have built a door that swings open for anyone who can type, which is everyone, so check the address before you believe the costume. More on bolting this shut in server security for SEO.
What the Verified Ones Actually Read
Turn back to the honest crawlers and read their diet, because it's a completely different animal, a herbivore. GoogleOther sent eighteen requests, five of five IPs verified by reverse DNS, and it read pages, the case studies, the PE section, the how-I-work page. Amazonbot sent 129, 120 verified, grazing on real article pages like /pe/due-diligence/ and /notes/how-to-know-if-you-can-rank. PetalBot and Applebot pulled robots.txt and sitemap.xml and behaved like the search engines they genuinely are. The bots whose addresses almost all checked out spent their visit on pages, assets, robots.txt and sitemap.xml, like customers. Checking that took me one trip behind the bar and a lookup per bot.
Not one request in the entire window carried Google-Extended or Applebot-Extended in its user agent, because those are robots.txt control tokens rather than live crawlers, and neither of them put a single line in this window. Apple says outright that webpages disallowing Applebot-Extended can still appear in search results, since the Extended token only decides whether crawled data trains the models. So if a vendor is selling you "we'll block Applebot-Extended in your logs," point and laugh, because an Extended token marks a preference in robots.txt and has no legs to walk into a log with. And my /llms.txt, which returned a clean HTTP 200, got five fetches from the entire parade of named AI bots in four and a half days, Bytespider once, Meta's agent twice, PetalBot twice, and GPTBot and ClaudeBot and PerplexityBot and the rest of the famous file's supposed audience didn't touch it once. (Humans and nameless scripts may have asked for it too, I only kept one in seven of their requests, so I won't pretend to know their number.) Five fetches from twenty-one AI agents is a sober thing to hold next to whatever you have been told the file is worth, so hold it there for a minute before you spend a sprint on it. If you want the substance of what machines actually read and reward, it's here: make your site legible to machines.
Look at a crawler's source IP and the paths it fetched and the status codes it got back together, because any one of them on its own will lie to you as smoothly as Francis's clientele. A verified operator pulling your pages plus robots.txt and the sitemap is a search engine doing search engine chores, and the same famous name arriving from an address nobody claims to scrape 404s off config-file paths is a man going through your bins who happened to print a lanyard first.
Robots.txt Is a Sign Taped to the Door
Now you can set policy with evidence in front of you instead of vibes and a conference slide. Robots.txt was standardised as RFC 9309 in 2022, and it is advisory, honoured voluntarily and separately by each crawler, which the three-coat problem turns from a blunt club into an actual scalpel. You can disallow GPTBot to stay out of the training set while allowing OAI-SearchBot so you keep surfacing in ChatGPT's search. Block Claude-SearchBot and you can cost yourself visibility in Claude's results, which Anthropic says plainly, so make that trade on purpose.
Except ChatGPT-User's documentation no longer promises robots.txt compliance, and Bytespider keeps getting caught reaching paths its own published rules forbid, from IPs outside its own published blocks, like a man who taped a "no soliciting" sign to his door and then went soliciting. So robots.txt is a sign for the polite ones, and the rest read it the way a raccoon reads a "please close the lid" sticker, which is why you built the verification routine. For the record, this site's robots.txt at run time is a plain User-agent: * and Allow: /, everyone welcome, which is a choice you can now make deliberately, with the log open in front of you, instead of the way Francis ran his door for six years and forty thousand forearms.
Decide robots.txt one agent at a time, block training while keeping search, or the reverse, and do it on purpose. Then pair every allow-or-block with IP verification at the firewall, because robots.txt only moves the bots that bother to read it and the rest need stopping by address.
The last number in the run is the one that finally cost Francis the job. The run checked the Referer header across the full log for the window, all 56,575 requests and not just the sample the commands ran on, hunting for the visitor who arrived because an AI assistant had sent them, a human clicking through from a cited answer, a small thank-you note for the four and a half days of reading the machines just did. None. The machines ate the entire site and tipped nobody, and the only place a single one of them wrote down a name was the log, where the names are worth exactly nothing until you check them, which is a sentence Francis can now recite from memory, out loud, at 2am, phone pressed to his ear, waiting for the limo company to pick up, every single time, like a man who has finally understood what his one job was for six years.