Who came by? Reading AI crawlers in your own server log

This blog carries 76 articles before this one. They deal with rankings, with citations in AI answers, with measurement frameworks and visibility scores. The server log appears in exactly one of them, in a subordinate clause. That is worth noting, because it is the only source on the subject that belongs to no one but the operator of the website.

Every other number about AI traffic comes from somebody else and measures somebody else. Fastly measures its own customers, Cloudflare measures its own, Adobe measures US retail. For the German Mittelstand there is no representative survey of how often AI systems come by. What does exist, on every web server, is a text file recording who read that site over the past few days. It is unglamorous, it is not prettily presented, and it is the only number about your own house.

How to control who may read is a separate question; we covered the controls that actually bite in our piece on llms.txt, robots.txt and what really governs AI access. This article is the counterpart: looking up who was actually there.

1. Who is knocking: the tokens in your log

Every line in an access log ends with the user agent the caller sent. With the reputable providers it contains a fixed identifier, a token. These tokens are publicly documented, they rarely change, and they say something about what the request is for. The overview below lists the operators that published their own documentation on this in 2026.

Three points in that table matter more than the rest.

• The Anthropic list does not separate the three bots. There is one shared file of IP prefixes, and it contains no bot names. It lets you establish that a request came from Anthropic — not whether it was ClaudeBot, Claude-User or Claude-SearchBot. That distinction remains available only through the user agent, and the user agent cannot be verified.
• OAI-AdsBot is new. It is still missing from many of the block lists in circulation. Anyone running a robots.txt written in 2025 simply does not have it on the list — neither to block nor to count.
• Filter on the token, never on the full user-agent string. The full string carries version numbers, and those move. Filter on GPTBot/1.2 and you count nothing from the next release onwards. Filter on GPTBot and you keep counting.

One curiosity that puzzles people on their first check: verified Bing requests resolve by reverse DNS to names such as msnbot-157-55-33-18.search.msn.com, even though the user agent says bingbot. The old name is still baked into the infrastructure. It is not an error and not evidence of forgery.

2. Training crawlers and fetch bots: the distinction that carries everything

If you take one distinction away from this article, take this one. The tokens in your log fall into two groups that have almost nothing to do with each other.

Training crawlers such as GPTBot, ClaudeBot or CCBot work to a schedule and in advance. They collect content because collecting is the job. There is no addressee, no question and no time pressure. A request from GPTBot at three in the morning means your page was collected. Nothing more.

Fetch bots such as ChatGPT-User, Claude-User, Perplexity-User or meta-externalfetcher work reactively. Behind every single one of those requests is a person, at that exact moment, with a question, who is about to receive an answer — frequently with a source link. Functionally that is closer to a visitor than to a crawler.

One practical consequence follows, and it is regularly lost in blocking debates: Perplexity and Meta document themselves that their fetch bots as a rule do not obey robots.txt, on the grounds that the user explicitly requested the fetch. The reasoning is coherent — this is a single human-initiated request, not systematic collection.

Block all AI bots wholesale and you block both kinds. You prevent the collection you might not have minded, and at the same time you lose precisely the traffic you wanted: the moment a prospect asks a question and your page is in contention as the answer. That applies all the more now that agentic AI browsers such as Atlas and Comet are out there acting on users' behalf.

3. Into the log: the commands that count

You need no administrator skills for this, only the log file and a terminal. On shared hosting you download the file from the customer portal; on your own server it usually sits at /var/log/apache2/access.log or /var/log/nginx/access.log. Older days are normally archived alongside as .gz; for those, replace grep with zgrep in the commands below and decompress before piping into awk.

A line in the common combined format looks like this — IP address, timestamp, requested path, status code, referrer, user agent:

203.0.113.7 - - [28/Aug/2026:04:11:52 +0200] "GET /services/ HTTP/1.1"
200 18422 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko);
compatible; ChatGPT-User/1.0; +https://openai.com/bot"

A log line in combined format. The user agent sits at the end; the token inside it is ChatGPT-User.

First, an overview of all tokens. The command below counts each token separately and prints a sorted list. It is the sensible first step, because it shows which systems appear on your site at all.

# Requests per token, sorted descending
for t in GPTBot OAI-SearchBot ChatGPT-User OAI-AdsBot \
         ClaudeBot Claude-User Claude-SearchBot \
         PerplexityBot Perplexity-User Googlebot bingbot \
         meta-externalagent meta-externalfetcher facebookexternalhit \
         Amazonbot Applebot CCBot MistralAI; do
  printf '%7s  %s\n' "$(grep -c -- "$t" access.log)" "$t"
done | sort -rn

One pass per token. Note that the filter is on the token, not on the full string with its version number.

Second, the pattern over time. More interesting than a total is the development. The command below takes the date out of the fourth field and counts each day separately.

# Requests per day for one token
awk '/ChatGPT-User/ {print substr($4, 2, 11)}' access.log | sort | uniq -c

The result is one line per day, for example 17 28/Aug/2026.

Third, which pages are being fetched. This is the most valuable extract in practice, because it shows which of your content AI answers actually need.

# The 20 most-fetched pages for one token
awk '/ChatGPT-User/ {print $7}' access.log | sort | uniq -c | sort -rn | head -20

Field 7 is the requested path. For the fetch bots, this list is the single most informative analysis available.

Fourth, the IP addresses behind a token, as preparation for the verification in the next section.

# All distinct IP addresses claiming to be GPTBot
awk '/GPTBot/ {print $1}' access.log | sort -u

You then compare that list against the operator's published IP list.

4. A name badge is not an ID: verifying requests — and where the method ends

The user agent is a freely writable text field. Anyone can put anything in it. It is a name badge, not an identity document. A line containing GPTBot establishes exactly one thing: that somebody wrote GPTBot into it.

There are two dependable ways to check.

4.1 Matching against the published IP lists

Most operators publish machine-readable lists of the IP ranges their crawlers use. You fetch the list, check whether the IP address from your log is in it, and you are done. OpenAI maintains a separate file per bot; Anthropic maintains one shared file for all three tokens. For Bing there is no IP list, so only the second method applies.

4.2 Reverse DNS with a forward confirmation

The method has two steps, and the second is the decisive one. You resolve the IP address to a hostname, then resolve that hostname back to an IP address. Only if both directions agree is the request established.

# Step 1: IP to name
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com

# Step 2: name back to IP - the confirmation
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Only if step 2 returns the original IP address is the request confirmed.

Why the second step counts: anyone can set the reverse record for their own IP address. If you control your own addresses, you can call them anything.googlebot.com. What you cannot do is set the forward record in the googlebot.com zone — that belongs to Google. Only the return leg closes the gap. Do step one alone and you have checked nothing.

4.3 Where the method ends

Here is the limit of what a log file can do. On 4 August 2025 Cloudflare demonstrated that Perplexity, when blocked, switched to undeclared crawlers: a generic Chrome user agent, rotating IP addresses, changing ASNs. Cloudflare put the volume of those undeclared requests at three to six million a day, against 20 to 25 million from the openly declared crawler.

The consequence for your analysis is uncomfortable but clear: anyone who does not want to be identified appears in your log as a visitor using Chrome. Every figure you take from the log is therefore a lower bound, never a full census. It counts the polite ones. That is still more than any external benchmark knows about your website — but it is not a complete answer to the question of who was there.

5. What your host actually keeps

Before you begin, it is worth checking how much history your provider makes available at all. Among the large German hosts the answer is consistently short.

Two points from that table decide what is possible.

IONOS anonymises visitor IP addresses in the log file. The verification described in section 4 therefore cannot be carried out there. You can see that something identifying itself as GPTBot made a request — you cannot check whether it was GPTBot. For the data-protection position that is an advantage; for evidence it is disqualifying. If you host with IONOS and need the check, you have to run it somewhere upstream.

At seven days to six weeks, a year-long analysis from shared-hosting logs is impossible. Anyone who wants to build a time series — and that is the interesting part, because it shows whether fetches are increasing — has to download and keep the logs themselves. A monthly entry in the calendar is enough. Start today and in twelve months you hold a number nobody else has.

For HostEurope and STRATO it is not documented whether the IP address is stored in full. We write that as it stands rather than guessing: thirty seconds and a look at the first column of your own log file will settle it.

6. Other people's benchmarks: what the numbers show and what they do not

Surveys of AI traffic do exist, and they are interesting. They simply do not measure you. Three are cited most often in 2026; here they are, with what each actually covers.

6.1 Fastly: OpenAI dominates the fetches

The Fastly Threat Insights Report of January 2026 analyses roughly 6.5 trillion requests a month across 130,000 applications. Among the fetch bots, OpenAI stands practically alone at 97 per cent; DuckAssistBot reaches 2 per cent. The fetch bots themselves account for about 1 per cent of all bot traffic. Among the crawlers the split is different: Google Other 36 per cent, Meta 28 per cent, PetalBot 18 per cent.

These figures describe Fastly's global customer base, which skews heavily towards large websites. That is not the German Mittelstand. What you can take from the report is the ranking — not the order of magnitude for your own site.

6.2 Fastly, June 2026: the ratio shifts

In an analysis published on 9 June 2026 covering January to May 2026, Fastly reports that AI traffic grew by 30 per cent, roughly 6.5 times faster than human traffic. For May 2026 the report gives a split of 85 per cent crawlers to 15 per cent fetches.

6.3 Adobe: a growth rate from a very small base

In April 2026, on data to March 2026 and more than a trillion visits to US retail sites, Adobe Digital Insights reports a 393 per cent year-on-year rise in traffic from generative AI. The figure is widely quoted and rarely placed in context.

It is a growth rate from a very small base, and it applies to US retail among Adobe customers. No expectation for a German mid-sized company can be derived from it. The more robust finding in the same report concerns quality rather than volume: AI-referred visitors show a 32 per cent lower bounce rate and view 13 per cent more pages. That is the number worth remembering — it describes what those visitors do, not how many of them there were somewhere else.

6.4 Cloudflare: the mismatch between taking and returning

In August 2025 Cloudflare published crawl-to-visit ratios across all sectors: Anthropic 50,000:1, OpenAI 887:1, Perplexity 118:1. For every 50,000 pages Anthropic fetched, one visitor came back. The figures should be labelled as at 2025; no methodologically documented 2026 version exists.

That mismatch is precisely why the debate about zero-click search is being had at all. And it is why your own log analysis should stay sober: a high crawler count is not an achievement.

7. The three fallacies

Look into your log for the first time and you will very likely draw one of these three conclusions. All three are wrong.

7.1 “GPTBot came 4,000 times, so ChatGPT recommends us.”

Wrong, because GPTBot is the training crawler. It tells you your content was collected, and nothing about whether you appear in answers. To measure visibility in answers you count ChatGPT-User, Claude-User and Perplexity-User — the tokens behind which a specific question sits. High crawler counts are attention from the machine, not demand from the market.

7.2 “The user agent says GPTBot, so it was OpenAI.”

Wrong in both directions. Upwards: Common Crawl itself warns about crawlers impersonating CCBot — the token is public, and it is available to anyone who wants to type it. Downwards: in the case Cloudflare documented, Perplexity did the opposite and presented itself as Chrome. A name in the log can claim too much and reveal too little. Hence the checks in section 4.

7.3 “Google-Extended is not in my log, so Google is not training on us.”

Wrong, because Google-Extended cannot appear there. It is purely a robots.txt token with no user agent of its own, and it sends no requests. Applebot-Extended works the same way.

The general case behind the example matters more than the example: nothing follows from absence in the log. It may mean nobody came. It may mean the request wore no name badge of its own. It may mean it wore somebody else's. Or it may simply mean your host's seven-day window overwrote the period in question long ago. An empty result is not a finding.

8. The IP in the log: the data-protection frame

The first column of a log file is an IP address, which opens the data-protection question. On 19 October 2016, in case C-582/14 (Breyer), the Court of Justice of the European Union held that a dynamic IP address can be personal data in the hands of a website operator. The legal basis for keeping server logs is regularly Article 6(1)(f) GDPR, legitimate interests.

There is no statutory retention period for server logs. What governs is the storage-limitation principle in Article 5(1)(e): as short as possible, as long as necessary. In practice seven days is treated as established, and the Bavarian data-protection authority is cited for thirty days. Both are practice and regulatory opinion, not statute.

For the use described here the position is relaxed. Extract aggregated figures — requests per bot, per day, per page — and the result carries no personal data. The IP address is needed only at the moment of verification and need not appear in the analysis afterwards. If you want to keep logs longer in order to build a time series, keep the aggregated table and delete the raw data.

9. Verdict: the most honest number you have about AI

A log analysis is not a large undertaking. Download a file, run four commands, turn the output into a table: a short hour the first time, ten minutes a month after that. What it produces is nonetheless something no benchmark can replace — a statement about your website rather than about the average of other people's websites.

Three things remain after the first pass. You know which systems read you at all. You know which of your pages the fetch bots retrieve — and that is the list your content work should follow. And you hold a baseline to compare against in three months.

The other half of the subject is control: which crawlers you admit and which you do not. How to reflect AI traffic in your web analytics as well is covered in our piece on the AI assistant channel group in GA4. And what a system needs in order to cite you at all is set out in generative engine optimization.

10. FAQ: the most common questions about server logs

Where do I find my website's access log?

On shared hosting, in your provider's customer portal: HostEurope supplies the log files through the KIS customer portal, while STRATO, IONOS and All-Inkl provide them through their own customer menus. On your own server they usually sit at /var/log/apache2/access.log or /var/log/nginx/access.log, with older days archived alongside as .gz files. If you only use a browser-based analytics tool, you see human visitors with JavaScript enabled and nothing else — crawlers appear in the server log alone.

Which token shows that ChatGPT is fetching my page for a user's question right now?

ChatGPT-User. That token marks a fetch at the moment of the question: somebody has just asked something, the system retrieves your page and answers with it, often with a source link. The equivalents at the other providers are Claude-User at Anthropic, Perplexity-User at Perplexity and meta-externalfetcher at Meta. GPTBot and ClaudeBot, by contrast, are training crawlers and say nothing about your visibility in answers.

How do I check that a request really came from GPTBot?

Two ways. First, against the published IP lists: OpenAI keeps one list per bot at openai.com/gptbot.json, /searchbot.json, /chatgpt-user.json and /adsbot.json; Anthropic keeps a single shared list at claude.com/crawling/bots.json. Second, by reverse DNS with a forward confirmation: you resolve the IP address to a name, then resolve that name back to an IP address. Only if both directions agree is the request verified — anyone can set the reverse record for their own IP address.

Why does Google-Extended never appear in my log?

Because it cannot. Google-Extended is purely a robots.txt token with no user agent of its own: it governs whether your content may be used for Gemini and Vertex AI, but it sends no requests itself. Apple's Applebot-Extended behaves the same way. In both cases, absence from the log tells you nothing.

How long do German hosts keep the access log?

Not long. All-Inkl states a maximum of seven days, HostEurope 14 days, STRATO six weeks with the explicit note that older data will not be made available, and IONOS a maximum of eight weeks. A year-long analysis from a shared-hosting log is therefore impossible. Anyone who wants a time series has to download and keep the logs themselves on a regular basis.

Can I run the verification with any host?

No. IONOS anonymises visitor IP addresses in the log file. There you can see that something identifying itself as GPTBot made a request, but you can no longer check whether it was GPTBot — that needs the full IP address. For HostEurope and STRATO it is not documented whether the IP address is stored in full; only a look at your own log file will settle it.

Am I allowed to analyse the IP addresses in the log at all?

On 19 October 2016, in case C-582/14 (Breyer), the Court of Justice of the European Union held that a dynamic IP address can be personal data in the hands of a website operator. The legal basis for server logs is regularly Article 6(1)(f) GDPR. There is no statutory retention period; what governs is the storage-limitation principle in Article 5(1)(e). For this particular use the position is relaxed: if you extract aggregated figures — requests per bot, per day, per page — the result carries no personal data. The IP address is needed only at the moment of verification. This article is research, not legal advice.

Why is a log analysis only a lower bound?

Because it can only count what identifies itself. On 4 August 2025 Cloudflare demonstrated that Perplexity, when blocked, switched to undeclared crawlers: a generic Chrome user agent, rotating IP addresses and changing ASNs, at an estimated three to six million requests a day against 20 to 25 million from the declared crawler. Anyone who takes off the name badge appears in your log as a visitor using Chrome. Every log statistic is therefore a lower bound, never a full census.