Which AI Crawlers Are Reading Your Blog?
GPTBot, ClaudeBot, PerplexityBot and a dozen others fetch your pages every day. Here is what each one actually does, which are crawlers and which are only robots.txt switches, and how to find out who is really reading.
The main AI crawlers are GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), CCBot (Common Crawl), Bytespider (ByteDance) and meta-externalagent (Meta). Google-Extended and Applebot-Extended are not crawlers at all, but robots.txt switches controlling AI use of already-crawled content.
Every public blog now has a second audience it never sees. Alongside search engines and readers, a growing set of AI crawlers fetches pages to train models, to answer questions in real time, or to build a search index for an assistant. Most site owners have no idea which ones visit, how often, or what they take. This guide names them, explains what each actually does, and shows how to find out who is reading.
The crawlers, and who runs them
The list below covers the agents you are most likely to see. Purposes are as each operator publishes them:
| User agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train models |
| OAI-SearchBot | OpenAI | Builds the index behind ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches a page because a user asked about it, in real time |
| ClaudeBot | Anthropic | Collects content for model training |
| PerplexityBot | Perplexity | Indexes pages to cite in Perplexity answers |
| CCBot | Common Crawl | Builds the open crawl corpus many models train on |
| Bytespider | ByteDance | Collects content for ByteDance models |
| meta-externalagent | Meta | Collects content for Meta AI |
| Amazonbot | Amazon | Indexes pages for Alexa and Amazon services |
| YouBot | You.com | Indexes pages for You.com answers |
Two of the famous names are not crawlers at all
This trips up almost everyone. Google-Extended and Applebot-Extended never appear in your server logs, because they do not fetch anything. They are robots.txt switches: Googlebot and Applebot crawl your site as they always have, and these tokens tell each company whether the content they already took may be used for AI training and grounding.
The practical consequence matters. Disallowing Google-Extended does not remove you from Google Search, and it will not reduce your crawl traffic by a single request. It only opts you out of AI training use. Blocking Googlebot itself would remove you from search, which is almost certainly not what you want.
Training, grounding, and live fetches are different things
Lumping every AI agent together leads to bad decisions, because the three jobs have opposite consequences for you:
- Training (GPTBot, ClaudeBot, CCBot, Bytespider) — your content may influence a future model. You get no attribution and no traffic.
- Grounding and indexing (OAI-SearchBot, PerplexityBot, YouBot) — your content can be cited, with a link, when someone asks a relevant question. This is the one that sends readers.
- Live user fetches (ChatGPT-User) — someone pasted your URL into an assistant. That is a real person reading, through a different window.
Blocking the training crawlers while allowing the grounding ones is a coherent position. Blocking everything is also coherent. Blocking the grounding crawlers while hoping to be cited in AI answers is not: you cannot be quoted by an engine you refused to let read you.
You cannot be cited by an answer engine you have blocked from reading the answer.
How to see who is actually visiting
Speculation is unnecessary here, because every visit leaves a line in your server log. Filter the access log by user agent and you have your answer. The catch is that a user agent is self-asserted: anything can claim to be GPTBot, and scrapers frequently do. Treating the string as proof is the common mistake. Verify the important ones by reverse DNS or by the IP ranges each operator publishes.
A second catch is that bot traffic quietly corrupts your analytics. If a crawler increments your view counter, your most-read list stops describing humans. Separate the two, or your content decisions get made on fiction.
Inkwell records this natively. Admin > AI Crawlers reports which engines fetch the site, how often, and which pages draw the most attention, writing to its own table so human analytics stay untouched. It lists every engine it can name including those with zero visits, because "no AI engine has read this site" is the most useful thing that screen can tell you, and it checks claimed identities rather than trusting the header. See the documentation for the details.
What to do with the answer
Once you know who visits, the decisions get concrete. If grounding crawlers are absent, your content is invisible to the assistants people now ask, and the fix is discovery rather than blocking: a clean llms.txt, permissive rules for the engines you want, and content shaped to be quoted. If they are present but you are never cited, the gap is the writing, not the access, and our guide to answer engine optimization covers that. And if a single agent is hammering the site, rate-limit it rather than reaching for a blanket ban.
The wrong move is the default one: never looking, and assuming either that nothing reads you or that everything does.
Frequently asked questions
What are AI crawlers?
AI crawlers are automated agents that fetch web pages on behalf of AI companies, either to collect training data, to build a search index an assistant can cite, or to retrieve a page a user asked about in real time. Common examples include GPTBot, ClaudeBot, PerplexityBot and CCBot.
Should I block AI crawlers?
It depends which job they do. Blocking training crawlers such as GPTBot, ClaudeBot and CCBot costs you nothing in traffic. Blocking grounding crawlers such as OAI-SearchBot and PerplexityBot also removes any chance of being cited, with a link, in AI answers. Decide per crawler rather than blanket-blocking.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended is not a crawler and fetches nothing. It is a robots.txt token controlling whether content Googlebot already crawled may be used for AI training and grounding. Ordinary Search indexing is unaffected. Blocking Googlebot itself is what would remove you from Search.
How do I know if a crawler is really who it claims to be?
You cannot tell from the user agent alone, because it is self-asserted and scrapers routinely impersonate well-known bots. Verify by reverse DNS lookup or by checking the request IP against the ranges each operator publishes. Treat unverified claims as claims, not facts.
Ready to host your own blog?
Inkwell is free, open-source, and self-hosted — your content, your server, your rules. Deploy in minutes on .NET 10.