User Agents for Web Scraping

What is a User Agent?

This term refers to any software that facilitates end-user interaction with web content. The user agent string (UA) is the text that a client computer program sends in a request.

The user agent string helps the target server determine which browser, device type, and operating system are being used. For example, the string tells the server that your computer is using Chrome and Windows 10. The server can use this information to tailor the response to the device type, OS, and browser.

Most browsers send the user agent header in the following format, although there is no specific consistency in the choice of user agents:

syntax for choosing a user agent - User-Agent: Mozilla/5.0 (<system-information.) <platform> (<platform-details>) <extensions> white text on grey background

Each browser adds its own comment components, such as platform or RV (release version). Mozilla provides examples of strings that should be used for crawlers:

Crawler string example - white text on grey background - Mozilla/5.0 (compatible; Googlebot/2.1; +http::/www.google.com/bot.html

Below are examples from the Chrome developer site showing the format of the UA string for various devices and browsers:

Chrome for Android

Phone UA:

white text on grey background - user agent to collect data as a android linux device or some other mobile device such as an apple iphone

Mozilla/5.0 (Linux; ; )AppleWebKit/ (KHTML, like Gecko) Chrome/Mobile Safari/

Tablet UA:

White text on grey background - syntax for user-agent to appear as either android tablet or apple ipad

Why is it necessary to use a User Agent?

When web scraping, you may sometimes find that the web server blocks certain user agents. This is mainly because it identifies the source as a bot, and some websites do not allow scraping bots. More sophisticated sites do the opposite—they allow user agents they consider legitimate to perform scraping tasks. Truly advanced sites check whether the browser behavior actually matches the claimed user agent.

You might think that the right solution is to omit the user agent from requests. However, this forces tools to use a default UA. In many cases, the target web server blacklists and blocks it.

How can you keep your user agent from being blocked?

Tips to Avoid UA Blocking When Scraping

#1: Use a Real User Agent

If your user agent is not from one of the major browsers, some sites will block its requests. Many bot-based web scrapers skip the UA setting step, leading to detection and banning due to a missing or incorrect default UA.

You can avoid this problem by setting a widely-used UA for your web scraper. A large list of popular user agents can be found here. You can compile a list of popular strings and rotate them while making cURL requests to a website. However, we recommend using your browser's user agent, as your browser's behavior is more likely to match what the user agent expects, unless you change it too much.

#2: Rotate User Agents

When making numerous requests during web scraping, you should rotate them. This minimizes the chance that the web server will detect and block your UAs.

How can you randomize requests?

One solution is to change the request IP address using rotating proxies. This way, a different set of headers is sent each time. On the web server side, it will appear as if the request is coming from different computers and browsers.

Tip: The user agent is a header, but headers include much more than just user agents. You cannot simply send random headers; you must ensure that the user agent you send matches the headers you are sending.

You can use botcheck.luminatio.io to check whether the headers you are sending match what is expected from the user agent.

How to rotate user agents

First, you need to collect a list of user agent strings. We recommend using strings from real browsers, which can be found here. The next step is to add these strings to a Python list. Finally, have each request pick a random string from the list.

An example of rotating user agents with Python 3 and Selenium 4 can be seen in this stack overflow discussion. Example code:

white text on grey background coding environment, python code on how to rotate user agents

Whichever program or method you choose to rotate UA headers, to avoid detection and blocking, you should follow the same practices:

#1: Rotate the full set of headers associated with each UA.
#2: Send headers in the order that a real browser typically sends them
#3: Use the previous page you visited as the "referrer header".

Tip: When using the referrer header, ensure that the IP address and cookies do not change. Ideally, you should actually visit the previous page so that the target server has a record of it.

#3: Rotate User Agents via a Proxy Server

You can avoid the headache and hassle of manually defining lists and rotating IP addresses by using a rotating proxy network. Proxies allow automatic rotation of IP addresses and UA strings. This makes your requests appear as if they come from various browsers. It significantly reduces blocking and increases the success rate of requests, as it seems the requests come from real users. Keep in mind that only specialized proxies using Data Unlocking technology can properly manage and rotate user agents.

List of User Agents for Scraping

There is a large number of browser/phone/device/bot/search engine/developer tool user agents that can be used to emulate various browsers when using tools like wget and cURL. These include:

Lynx: Lynx/2.8.8pre.4 libwww-FM/2.14 SSL-MM/1.4.1 GNUTLS/2.12.23
Wget: Wget/1.15 (linux-gnu)
Curl: curl/7.35.0

HTC: Mozilla/5.0 (Linux; Android 7.0; HTC 10 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.83 Mobile Safari/537.36
Google Nexus: Mozilla/5.0 (Linux; U; Android-4.0.3; en-us; Galaxy Nexus Build/IML74K) AppleWebKit/535.7 (KHTML, like Gecko) CrMo/16.0.912.75 Mobile Safari/535.7
Samsung Galaxy Note 4: Mozilla/5.0 (Linux; Android 6.0.1; SAMSUNG SM-N910F Build/MMB29M) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/4.0 Chrome/44.0.2403.133 Mobile Safari/537.36

Samsung Galaxy Note 3: Mozilla/5.0 (Linux; Android 5.0; SAMSUNG SM-N900 Build/LRX21V) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/2.1 Chrome/34.0.1847.76 Mobile Safari/537.36
Samsung Phone: Mozilla/5.0 (Linux; Android 6.0.1; SAMSUNG SM-G570Y Build/MMB29K) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/4.0 Chrome/44.0.2403.133 Mobile Safari/537.36
Bing’s Search Engine Bot: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Google’s Search Engine Bot: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Apple iPhone: Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1

Apple iPad: Mozilla/5.0 (iPad; CPU OS 8_4_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12H321 Safari/600.1.4
Microsoft Internet Explorer 11 / IE 11: Mozilla/5.0 (compatible, MSIE 11, Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
Microsoft Internet Explorer 10 / IE 10: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; MDDCJS)

Microsoft Internet Explorer 9 / IE 9: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0; Trident/5.0)
Microsoft Internet Explorer 8 / IE 8: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Microsoft Internet Explorer 7 / IE 7: Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 6.0; en-US)
Microsoft Internet Explorer 6 / IE 6: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Microsoft Edge: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393
Mozilla Firefox: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0
Google Chrome: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36

Conclusion

Since most sites block requests that do not contain a valid or recognizable browser user agent, learning proper UA rotation is essential to avoid site blocking. Using a correct user agent tells the target site that the request came from a legitimate source, allowing you to collect data from the desired sites without hindrance.