Search Engine Bots
The user-agent strings that the crawlers of Google, Bing, Baidu, DuckDuckGo and Yandex send when they index your pages, ready to copy for log analysis, analytics filters and allow-lists. The list is updated as the engines change their bots.
| Search engine | User-agent tokens |
|---|---|
Googlebot | |
| Microsoft Bing | bingbot |
| Baidu | Baiduspider |
| DuckDuckGo | DuckDuckBot |
| Yandex | YandexBot, YandexMobileBot, YandexImages |
About this tool
Search engine crawlers identify themselves with a user-agent string that contains a bot token and a link to the engine's documentation: Googlebot (with a desktop and a smartphone profile), bingbot, Baiduspider, DuckDuckBot and the Yandex family (YandexBot, YandexMobileBot, YandexImages). Recognising them matters in two directions: to keep them out of visitor statistics and rate limits, and to make sure a firewall or bot-management rule never blocks the traffic that keeps your site in the search results. This reference gathers the strings seen in real server logs in one copy-ready list, with a table of the tokens by engine. Because the strings are easy to spoof, confirm the source with a reverse DNS check before trusting a request that claims to be a search engine.
Other kinds of automated visitors have their own lists: AI Crawlers and Bots for the crawlers and fetchers of AI assistants, Security Scanners UAs for scanning and reconnaissance tools, and HTTP Client User Agents for the default strings of libraries such as python-requests, curl and axios.
Related: Reverse DNS Lookup, My User Agent, HTTP Headers.
How to use it — quick steps
- 01
Copy the list
Press [copy] on the box and paste the strings into your log filter, analytics exclusion list or allow-list.
- 02
Match by token
Use the bot name from the table (Googlebot, bingbot, YandexBot…) rather than the whole string, since browser versions inside it change.
- 03
Verify the source
Before allow-listing an IP that claims to be a search engine, check it with the Reverse DNS Lookup; spoofed crawler user agents are common.
Key features of this tool
Copy-ready list
One user-agent string per line, exactly as the crawlers send it, with a [copy] link; no wrapping, so each line is a complete string.
Tokens by engine
A table of the short bot names (Googlebot, bingbot, Baiduspider…) by search engine, the part to match in logs and rules.
Kept up to date
New crawlers and new versions of existing user agents are added as the engines roll them out.
Frequently asked questions
Why does Googlebot have several user agents?
Google crawls with a desktop and a smartphone profile (the Android string) and updates the Chrome version in the string regularly; Bing and Yandex do the same. Match the token (Googlebot, bingbot) rather than the full string.
How do I know a request really comes from Google or Bing?
Anyone can send a Googlebot user agent. Check the source IP with the Reverse DNS Lookup: real crawlers resolve to googlebot.com, google.com, search.msn.com, yandex.ru or similar, and the forward lookup of that name points back to the same IP. Google, Microsoft and others also publish the IP ranges of their crawlers.
Do search engine bots follow robots.txt?
Yes, all the engines listed here state that their crawlers honour robots.txt and the crawl-delay or rate controls in their webmaster tools; this is what makes them different from scanners and most AI fetchers.
Should I filter them out of my analytics?
Server-side analytics should exclude them, or your visitor numbers include every crawl; JavaScript-based analytics rarely count them. Never block them at the firewall unless you want to disappear from that search engine.