Search Engine Bots

The user-agent strings that the crawlers of Google, Bing, Baidu, DuckDuckGo and Yandex send when they index your pages, ready to copy for log analysis, analytics filters and allow-lists. The list is updated as the engines change their bots.

Search engineUser-agent tokens
GoogleGooglebot
Microsoft Bingbingbot
BaiduBaiduspider
DuckDuckGoDuckDuckBot
YandexYandexBot, YandexMobileBot, YandexImages

About this tool

Search engine crawlers identify themselves with a user-agent string that contains a bot token and a link to the engine's documentation: Googlebot (with a desktop and a smartphone profile), bingbot, Baiduspider, DuckDuckBot and the Yandex family (YandexBot, YandexMobileBot, YandexImages). Recognising them matters in two directions: to keep them out of visitor statistics and rate limits, and to make sure a firewall or bot-management rule never blocks the traffic that keeps your site in the search results. This reference gathers the strings seen in real server logs in one copy-ready list, with a table of the tokens by engine. Because the strings are easy to spoof, confirm the source with a reverse DNS check before trusting a request that claims to be a search engine.

Other kinds of automated visitors have their own lists: AI Crawlers and Bots for the crawlers and fetchers of AI assistants, Security Scanners UAs for scanning and reconnaissance tools, and HTTP Client User Agents for the default strings of libraries such as python-requests, curl and axios.

Related: Reverse DNS Lookup, My User Agent, HTTP Headers.


How to use it — quick steps

  1. 01

    Copy the list

    Press [copy] on the box and paste the strings into your log filter, analytics exclusion list or allow-list.

  2. 02

    Match by token

    Use the bot name from the table (Googlebot, bingbot, YandexBot…) rather than the whole string, since browser versions inside it change.

  3. 03

    Verify the source

    Before allow-listing an IP that claims to be a search engine, check it with the Reverse DNS Lookup; spoofed crawler user agents are common.


Key features of this tool

  • Copy-ready list

    One user-agent string per line, exactly as the crawlers send it, with a [copy] link; no wrapping, so each line is a complete string.

  • Tokens by engine

    A table of the short bot names (Googlebot, bingbot, Baiduspider…) by search engine, the part to match in logs and rules.

  • Kept up to date

    New crawlers and new versions of existing user agents are added as the engines roll them out.


Frequently asked questions

Why does Googlebot have several user agents?

Google crawls with a desktop and a smartphone profile (the Android string) and updates the Chrome version in the string regularly; Bing and Yandex do the same. Match the token (Googlebot, bingbot) rather than the full string.

How do I know a request really comes from Google or Bing?

Anyone can send a Googlebot user agent. Check the source IP with the Reverse DNS Lookup: real crawlers resolve to googlebot.com, google.com, search.msn.com, yandex.ru or similar, and the forward lookup of that name points back to the same IP. Google, Microsoft and others also publish the IP ranges of their crawlers.

Do search engine bots follow robots.txt?

Yes, all the engines listed here state that their crawlers honour robots.txt and the crawl-delay or rate controls in their webmaster tools; this is what makes them different from scanners and most AI fetchers.

Should I filter them out of my analytics?

Server-side analytics should exclude them, or your visitor numbers include every crawl; JavaScript-based analytics rarely count them. Never block them at the firewall unless you want to disappear from that search engine.


Popular tools


From APIVoid, the team behind IPVoid

Enhance your threat detection capabilities

APIVoid provides threat intelligence APIs to automate security workflows, with simple pricing and a 30-day free trial to start, no card required.

What you get with APIVoid

  • IP reputation and blacklist checks
  • Domain and URL reputation scoring
  • DNS lookups and propagation
  • SSL certificate and TLS details
  • Website screenshots and page info
  • Email address verification
  • 30+ cybersecurity APIs
30-day free trial