Read and Respect the Robots.txt File
Robots.txt is a file used by websites to let "search bots" know whether the site should be crawled and indexed by search engines. Many sites simply forbid crawling, meaning the site should not be crawled by search engines or other bots. When you try to extract data from the web, it is very important to understand what robots.txt is and how to read and respect robots.txt to avoid legal consequences.
Why should you read and respect the robots.txt file?
Respecting robots.txt should not be about violators facing legal trouble. Just as you must obey lane discipline while driving on a highway, you should respect the robots.txt file of the site you are crawling. This is considered standard behavior on the web and is in the interest of web publishers.
Many websites choose to block bot crawlers because their content is sensitive and of little use to the public. If that's not a good enough reason to follow robots.txt rules, note that crawling a site that forbids bots can lead to litigation and end badly for a firm or individual. Let's now move on to how you can follow robots.txt rules to stay in the safe zone.
Robots.txt Rules
1. Allow full access
User-agent: *
Disallow:
If you find this in the robots.txt file of the site you are trying to parse, you are in luck. It means that all pages of the site are accessible to bots.
2. Block all access
User-agent: *
Disallow: /
You should stay away from sites with this entry in robots.txt. It says that no part of the site should be visited by an automated spider script, and violating it could lead to legal trouble.
3. Partial access
User-agent: *
Disallow: /folder/
User-agent: *
Disallow: /file.html
Some sites disallow crawling of only certain sections or files on their site. In such cases, you should configure your bots to leave the blocked areas untouched.
4. Crawl rate limit
Crawl-delay: 11
This parameter is used to restrict too frequent visits by parsers. Since frequent visits can cause unwanted server load and slow the site for visitors, many sites add this line to the robots file. In this case, the site can be crawled with an 11-second delay.
5. Visit time
Visit-time: 0400-0845
This tells crawlers the hours when parsing is allowed. In this example, the site can be crawled from 04:00 to 08:45 GMT. Sites do this to avoid bot load during peak hours.
6. Request rate
Request-rate: 1/10
Some sites do not welcome bots trying to fetch multiple pages at once. Request rate is used to limit such behavior. The value 1/10 means the site allows bots to request one page every 10 seconds.
Being a good bot
Good bots follow the rules set by websites in their robots.txt file and follow best practices when parsing. It goes without saying that you should examine the robots.txt file of each target site to ensure you are not violating any rules.
Confused?
It is not uncommon to feel intimidated by the complex technical jargon and rules associated with web parsing.
If you find yourself in a situation where you need to extract data from the web but are confused about compliance issues, we would be happy to become your data collection partner and take responsibility for the entire process.