5 Tips When Using a Bot to Avoid Proxy Server Blocking

If you regularly engage in web scraping, you know that there are two aspects to consider in this process: legal issues and IP blocking. Although extracting data from a website without the owner's consent is not directly illegal, it is frowned upon, which is why IP addresses are constantly blocked.

Besides the fact that this data can be used to gain a business advantage, using bots on a website can reduce its performance and ultimately lead to a crash.

Therefore, if you want to do web scraping, make sure you can complete the process before you even start, so you don't waste your resources. One way to ensure success is to prevent IP blocking, and in this article we will look at ways to reduce the risk of proxy server blocking.

Choosing a proxy server is another factor that needs careful consideration before starting the actual scraping. While all proxies provide anonymity, some are more cautious and harder to detect than others.

Bot Identification and Blocking

Before moving on to preventing proxy site blocking during scraping, it is necessary to understand how these sites identify a bot and distinguish it from a real user.

It's safe to say that sites and those who extract data play a cat-and-mouse game. While sites improve their methods of detecting and blocking such activity, those who engage in it seek ways to hide their presence while performing automated processes.

You might think it's very easy to tell a real user from a bot, but it's not. Suspicious activity must be detected, then flagged, and only after further monitoring can the site block it.

The most common methods used by websites to identify bots involved in web scraping are as follows:

• If a large number of requests to a URL come from a single IP address, it is assumed to be a bot because a human cannot work that fast.
• Websites can also detect bot usage if WebRTC transmits your real IP address to the site's servers.
• When a request sent to the site's server has various attributes that do not correlate with each other. To avoid bot detection, always ensure the request matches the time zone and selected language.
• When suspicious browser configurations are detected, the site may associate them with bot usage and block the IP. An example is disabled JavaScript. Different browsers use different JavaScript versions, and based on supported features and other criteria, the site can recognize your browser.
• Connecting to a site without cookies is suspicious and indicates bot usage. However, having cookies does not allow you to use a bot unnoticed, since cookies are used to track you.
• Websites also notice non-human behavior on the web page. Mouse and keyboard actions are difficult for a bot to imitate, making it easy to detect. Unlike a human, bots are easily predictable, and when this happens, the site's security measures become suspicious.

Detecting bot activity during web scraping is the first reaction of sites to you. Suspecting your activity, they may respond in various ways: track you, display an error page, or provide false data. Ultimately, access to the site may be blocked.

Sites disapprove of bot usage due to spam on review and comment pages. A large number of requests also affects site performance and can slow it down.

This can lead to site crashes or degraded user experience. No one likes to lose in competition, and many sites, especially e-commerce sites, compete with other brands.

Data extraction is one way a competitor can outperform you by exploiting your weaknesses and improving your strengths. This is another reason sites block bots.

Web Scraping Scripts in Python: Reducing the Risk of Your Scraper Being Blocked

User Agent
When creating a web scraper, you need to set a user agent to access the target site. The user agent header provides the site with information about you so you can receive the requested data in a convenient format.

You can get your user agent string by searching Google for "what is my user agent?". If you use the requests library, you can set your user agent as follows:

headers = {

\ 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',

\ }

r = requests.get('example.com',headers=headers)

To make the specified user agent look real, select a random agent from a database or text file.

The following function returns a random user agent that can be selected when sending requests to reduce the chance of being blocked:

import numpy as np

def get_random_ua():

\ random_ua = ''

\ ua_file = 'ua_file.txt'

\ try:

\ with open(ua_file) as f:

\ lines = f.readlines()

\ if len(lines) > 0:

\ prng = np.random.RandomState()

\ index = prng.permutation(len(lines) - 1)

\ idx = np.asarray(index, dtype=np.integer)[0]

\ random_proxy = lines[int(idx)]

\ except Exception as ex:

\ print('Exception in random_ua')

\ print(str(ex))

\ finally:

\ return random_ua

With the get_random_ua function, a random user agent from the text file will be used.

Proxy Server
One of the main reasons bots are blocked is the use of static proxies, or proxies that follow a specific sequence. Use multiple random proxies so the site cannot detect any pattern.

If rotating proxies are not available, use IP rotators so you can use one proxy at a time, possibly throughout the day. Also pay attention to all proxies that are blocked by the target site.

Delay
When your requests come in quickly, the site will block your bot because this behavior is not organic or human-like. You can delay requests using numpy.random.choice():

delays = [7, 4, 6, 2, 10, 19]

delay = np.random.choice(delays)

time.sleep(delay)

Referrers
Setting a referrer is another important part of preventing scraper blocking. Typically, if you are scraping a listing page or homepage, you should set the Google homepage for that country. If you plan to scrape individual product pages, you have the option to either set the URL of the relevant category or find a backlink to the domain you are scraping.

Request Headers
Some sites are more sophisticated than others and will require more effort if you want your parse to not be blocked. Such sites, as part of their bot detection strategy, look for certain entries in the request headers, and if those headers are not found, they either block or substitute the requested content.

By inspecting the page, you can determine which headers are required. Implement them or implement headers one by one after testing.

Reduce the Risk of Your Proxy Server Being Blocked: Web Scraping Tips

Below are recommendations for preventing proxy site blocking by websites.

Respect Site Policy
Before working with a website, find out what its policy is. It is generally accepted that the best results come from being polite and following the site's policy on crawling.

Most websites store a robots.txt file in the root directory that contains details on what can and cannot be scraped. It also contains information on how frequently you can scrape.

You should also review the site's terms of service, as they contain information about the data on the site. You will learn whether this data is public or protected by copyright, and the best way to access the target server and the data you need.

Scrape at a Lower Speed and Frequency
Bots are used because they are more efficient and faster than a human, and this speed is their downfall as it is one way sites detect and block them. This action, like any other that is not natural or human-like, is suspicious, and to avoid drawing attention, you need to regulate the number of requests sent at a time. Too many requests also negatively affect the target server, overloading it and making it slow and unresponsive.

Reconfigure your scraper and slow it down by making it sleep randomly between requests. Additionally, take longer sleep breaks of varying lengths after scanning a different number of pages. To avoid raising suspicion or red flags, it is best to make this as random as possible.

IP Rotation
Anyone who has done web scraping is familiar with the common warning not to send too many requests using the same IP address. This guarantees you will be blocked, so you need to use multiple proxies before starting scraping. Data extraction requires sending multiple requests to the web server, and the number depends on the volume of data needed. Normal human behavior has a limit on the number of requests that can be sent at once, so a higher number will be considered bot activity.

To use multiple proxy servers for data collection, you will need an IP rotator program. This software takes one IP per session or for a certain time and sends requests through it. Thus, the target server will believe that the requests are coming from the same device, which will avoid blocking.

Randomize Crawling Pattern
Website bot protection features can detect bot usage by tracking their activity and identifying patterns in their actions and the way they navigate to other sites. This is especially true if you have a fixed pattern, so randomness is good.

To reduce the risk of proxy blocking, configure the bot to randomly perform some actions, such as mouse movement, mouse clicks, or mouse scrolling.

People are unpredictable in these actions, and what you aim for is human-like behavior. So the more random your actions, the more human-like you will appear.

User Agents
The user agent HTTP request header transmits information to the target server such as the type of application used, operating system, software, software version, and allows the target server to decide which type of HTML layout to send: desktop or mobile.

If the user agent is empty or unusual, it can be a red flag, as the site server may consider it coming from a bot. To avoid this, use common configurations to avoid suspicion.

Below are typical user agent configurations for various browsers:

• Apple iPad

Mozilla/5.0 (iPad; CPU OS 8_4_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12H321 Safari/600.1.4

• Apple iPhone

Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1

• Bing Bot (Bing search engine bot)

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

• Curl

curl/7.35.0

Google Chrome

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36

• Google Nexus

Mozilla/5.0 (Linux; U; Android-4.0.3; en-us; Galaxy Nexus Build/IML74K) AppleWebKit/535.7 (KHTML, like Gecko) CrMo/16.0.912.75 Mobile Safari/535.7

• HTC

Mozilla/5.0 (Linux; Android 7.0; HTC 10 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.83 Mobile Safari/537.36

Googlebot (Google search engine bot)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html

• Lynx

Lynx/2.8.8pre.4 libwww-FM/2.14 SSL-MM/1.4.1 GNUTLS/2.12.23

• Mozilla Firefox

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0

• Microsoft Edge

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393

• Samsung Phone

Mozilla/5.0 (Linux; Android 6.0.1; SAMSUNG SM-G570Y Build/MMB29K) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/4.0 Chrome/44.0.2403.133 Mobile Safari/537.36

• Samsung Galaxy Note 3

Mozilla/5.0 (Linux; Android 5.0; SAMSUNG SM-N900 Build/LRX21V) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/2.1 Chrome/34.0.1847.76 Mobile Safari/537.36

• Samsung Galaxy Note 4

Mozilla/5.0 (Linux; Android 6.0.1; SAMSUNG SM-N910F Build/MMB29M) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/4.0 Chrome/44.0.2403.133 Mobile Safari/537.36

• Wget

Wget/1.15 (linux-gnu)

Sending too many requests to the server from a single user agent is bad and not human-like. So instead, simulate real human behavior by switching between different request headers being sent...

Using bots is very necessary when you need to extract data from a target site. This process can be done manually, but it is very tedious, hence the need for automation. Using bots allows you to quickly send many requests to the site's servers and obtain as much data as you need in a short time.

However, this speed can be detrimental to the site, and along with the fact that data can be stolen by competitors, site owners do not approve of data extraction. Therefore, they implement anti-bot measures to detect and block proxy servers.