AI Crawlers and Bots
The user-agent strings that AI crawlers, user-triggered fetchers and AI search bots send when they request your pages, ready to copy for log analysis, analytics filters and firewall rules. The list is updated as new bots appear.
| Vendor | User-agent tokens |
|---|---|
| Anthropic (Claude) | ClaudeBot, Claude-User, Claude-SearchBot, Claude-Agent, claude-code |
| OpenAI (ChatGPT) | GPTBot, ChatGPT-User, OAI-SearchBot, OAI-AdsBot |
| Perplexity | PerplexityBot, Perplexity-User |
| Moonshot AI (Kimi) | Kimi-SearchBot, MoonshotBot |
| Zhipu AI (ChatGLM) | ChatGLM-Spider |
| xAI (Grok) | GrokBot |
| Cohere | cohere-ai |
| Meta | Meta-WebIndexer, Meta-ExternalFetcher, Meta-ExternalAgent |
| Tencent (Hunyuan) | Hunyuan |
| Amazon | Amazonbot |
| DeepSeek | DeepSeekBot |
| ByteDance | Bytespider |
| Apple | Applebot, Applebot-Extended |
| Alibaba (Qwen) | Qwenbot |
| Mistral AI | MistralBot |
| DuckDuckGo | DuckAssistBot |
| You.com | YouBot |
About this tool
Large language model companies send three kinds of traffic to websites: crawlers that collect pages for training or for building an index, user-triggered fetchers that load a page because someone asked the assistant about it, and search bots that index pages so the assistant can cite them. Each identifies itself with a user-agent string, often with a link to the vendor's page describing the bot and how to opt out. This reference gathers the strings seen in real server logs in one copy-ready list, with a table of the tokens by vendor. Use it to recognise the traffic in your access logs, to decide what to allow, and to spot requests that only pretend to be an AI bot (the Reverse DNS Lookup and the IP Blacklist Check help verify the source address).
Related: My User Agent, Track HTTP Requests, HTTP Headers.
How to use it — quick steps
- 01
Copy the list
Press [copy] on the user-agent box and paste the strings into your log filter, WAF rule or bot-management tool.
- 02
Match by token
For reliable matching use the bot token from the table (for example GPTBot or ClaudeBot) rather than the whole string.
- 03
Verify the source
A user agent can be spoofed: check the requesting IP with the Reverse DNS Lookup or the IP Blacklist Check to see whether it really belongs to the vendor.
Key features of this tool
Copy-ready list
One user-agent string per line, exactly as the bots send it, with a [copy] link; no wrapping, so each line is a complete string.
Tokens by vendor
A table of the short bot names (GPTBot, ClaudeBot, PerplexityBot…) grouped by company, the part to match in logs and rules.
Kept up to date
New AI bots and new versions of existing user agents are added as they show up in the wild.
Frequently asked questions
What is the difference between a crawler, a fetcher and a search bot?
A crawler (GPTBot, ClaudeBot, Bytespider…) visits pages on its own to collect data, typically for model training. A user-triggered fetcher (ChatGPT-User, Claude-User, Perplexity-User…) loads a page because a person asked the assistant about it, much like a browser. A search bot (OAI-SearchBot, Claude-SearchBot, PerplexityBot…) indexes pages so the assistant can cite them in answers.
Do these bots follow robots.txt?
The major vendors state that their crawlers and search bots honour robots.txt by their token (GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended…). User-triggered fetchers such as ChatGPT-User or Claude-User generally do not, because a person asked for that specific page, much like a browser request.
How do I find these bots in my logs?
Search the user-agent field for the tokens listed in the table (the name before the slash, such as GPTBot or ClaudeBot) rather than for the full string, since versions change. Use the IP Blacklist Check or the Reverse DNS Lookup to confirm that a request really comes from the vendor and not from someone spoofing the user agent.
Is the list complete?
It covers the bots most often seen in server logs; vendors add and rename bots regularly, and this page is updated as they do. If you see an AI user agent that is missing, let us know through the contact page.