The AI crawlers hitting one site, and which of them were real

The name that cannot exist

Google-Extended has no user agent, so every request carrying it was forged

Google-Extended is a robots.txt control token. It is not a crawler, and Google says so on its own crawler page.

“Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.”

Google, Google common crawlers, read 3 October 2026: developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

The same page carries one more line worth keeping in view: “Caution: The HTTP user agent string can be spoofed.”

So all 1,100 requests in this window whose user agent said Google-Extended were forged by Google's own description of the token. 4 of them asked for a URL this site once had, so the path classifier files them as dead URLs of ours rather than as probes, which is why the catalogue's forged column for this row reads 1,096. By Google's own description of the token they are forged too. That is what makes the name useful here. Its paths are a set of requests already known to be forged, with no argument needed, so every other name's paths can be compared against a known answer instead of against a hunch. That comparison is the forged column in the catalogue below.

Why the string is not the answer

The user agent string cannot settle identity, in either direction

A common piece of advice is to match the vendor's documented user agent string and treat a mismatch as a forgery. This reading says that does not work. Under the name GPTBot alone, 8 distinct user agent strings arrived in the window.

1. The variant that matches OpenAI's documentation character for character sent 1,166 requests, and 99.9% of them were for files that have never existed here. Matching the documented string is therefore not evidence that a request is what it claims to be.
2. It is also the only one of the 8 strings that asked for such a file at all. The other 7 asked for none, the largest of them 327 requests that do not match the documentation at all. Copying the vendor's published string is the forger's default, so failing to match it is not evidence either.

Keep that second point narrow. All that was measured about those strings is that they asked for nothing that has never existed here. It is not a finding that they came from OpenAI, and this page does not claim they did. The claim is only about what the string proves, which is nothing in either direction.

So punctuation, parenthesis placement and version numbers do not identify a forgery. Identity is settled by the source IP address, checked against the vendor's own published prefix file, and by nothing else a site owner has.

The catalogue

18 crawler names, what each one is for, and how a stranger checks it

One entry per crawler family, in the order the reading counts them. The figures in each entry are that name's rows in this one window: requests is served plus dead URLs of ours plus requests for files that have never existed here. The vendor fields are that vendor's own documentation, read on 3 October 2026. Where a vendor publishes nothing, the entry says not documented and the gap is left open rather than filled from somewhere else.

Method and limits

What this reading is, and the questions it cannot answer

• One origin's nginx access log, 19 September 2026 to 3 October 2026, 15 days and 200,515 request rows. A row is attributed to the name its own user agent claims, with this site's own readbacks excluded first.
• A request counts as being for a file that has never existed here when it is a credential path, or carries a self-referring referrer, or asks for a path that 4 or more different vendor names also asked for and this site has never served. A minute counts as crowded when 5 or more different names appear in it. A 404 that is simply an old URL of ours is counted separately, as dead URLs of ours.
• Identity by address is not tested here, and cannot be at this origin: it sits behind a CDN, so the log holds the edge address. That is the trap described above, not a result.
• Nothing here measures robots.txt compliance. This site's robots.txt allows everything, so there was no rule for a crawler to break and compliance was never testable on it. A compliance claim would need a site that disallows something.
• The figures are a dated reading of one origin, not a benchmark. Another site behind another CDN, with another robots.txt and another set of old URLs, will read differently. The useful part is the method and the vendor fields, both of which transfer.
• Read at 2026-10-03T09:01:09+00:00. The reading is kept as data in the repository that builds this page, and the page renders it rather than restating it, so the two cannot drift apart.