HTTP Client User Agents
The default user-agent strings that HTTP libraries, SDKs and command-line clients send when a script or program requests your pages, ready to copy for log analysis, analytics filters and rate-limiting rules. The list is updated as new clients appear.
About this tool
Every HTTP library and command-line client sends a default User-Agent header unless the developer overrides it: the library name and version, often followed by the language runtime and the operating system. Because most scripts, integrations, monitors and scrapers keep that default, these strings are the quickest way to tell programmatic traffic from real visitors in access logs and analytics. This reference gathers the defaults seen in real server logs in one copy-ready list: Python (requests, urllib, httpx, aiohttp), Go, Node.js (axios), Java (Apache HttpClient, okhttp), Ruby, curl, Wget, Windows system components and scraping frameworks such as colly and grub.
These clients are neutral tools: the same python-requests string comes from a company's uptime monitor and from someone copying your catalogue. Look at the request pattern and the source address (the Reverse DNS Lookup and the IP Blacklist Check help) before filtering, and prefer rate limits to blanket blocks, which also break legitimate integrations.
Related: Search Engine Bots, AI Crawlers and Bots, Security Scanners UAs, My User Agent.
How to use it — quick steps
- 01
Copy the list
Press [copy] on the box and paste the strings into your log filter, analytics exclusion list or rate-limiting rule.
- 02
Match by client name
Use the name before the slash (python-requests, curl, axios, Go-http-client…) rather than the full string, since versions change constantly.
- 03
Judge by behaviour
Combine the user agent with request rate, paths and source IP before deciding whether the traffic is an integration, a monitor or a scraper.
Key features of this tool
Copy-ready list
One user-agent string per line, exactly as the clients send it, with a [copy] link; no wrapping, so each line is a complete string.
Libraries and tools
Python (requests, urllib, httpx, aiohttp), Go, Node.js (axios), Java (Apache HttpClient, okhttp), Ruby, curl, Wget and Windows services, each with the format it uses by default.
Kept up to date
New clients and new versions of existing ones are added as they show up in server logs.
Frequently asked questions
What do these user agents tell me?
That the request came from a program, not a browser: a monitoring script, an integration, a scraper, a security scanner built on a library, or a developer testing with curl. The string is the library's default, which most authors never change.
Are these requests harmful?
Usually not by themselves: uptime monitors, API clients, feed readers, webhooks and your own scripts all send them. The pattern matters: hundreds of requests per minute, probing of login pages or systematic downloading of every page is scraping or abuse; a few requests to an API endpoint is normal integration traffic.
Why is the version in the string useful?
It identifies the library version and often the language runtime and operating system (python-requests/2.34.2, CPython/3.7.2 Windows/10), which helps when debugging an integration or when a very old client breaks on a modern TLS configuration.
Can I block them?
You can, in Nginx, Apache or a WAF rule, but blocking by user agent alone also stops legitimate integrations and is trivial to bypass by setting a browser-like string. Rate limiting and per-path rules by behaviour work better; verify the source IP with the Reverse DNS Lookup and the IP Blacklist Check when in doubt.