Overview of Web Scraping Best Practices

At ESK Solutions, we ensure that our services respect the rights of websites and companies whose data we scrape.

We often hear that web scraping is a legal "gray area," but in reality, scraping itself is not illegal. The gray area lies in how and what you use.

This article provides a set of recommendations for finding information on the internet, helping you know when to be cautious about the method and type of data you scrape.

Disclaimer: We are not your attorney, and the recommendations in this guide do not constitute legal advice. Our head of legal is not providing you with legal services, so none of his opinions or recommendations in this guide should be taken as legal advice for you. The comments and recommendations below are based on the experience of ESK Solutions in helping our clients comply with legal regulations when scraping data. If you need assistance with your specific situation, you should consult a lawyer.

Don't Be a Burden

The first rule of working with web pages: do no harm to the site. The second rule of scraping: DO NOT harm the website. This means the volume and frequency of your requests should not overload the site's servers or interfere with its normal operation.

1. Limit the number of concurrent requests to the same site from a single IP.
2. Respect the delay that crawlers should maintain between requests by following the crawl-delay directive specified in the robots.txt file.
3. If possible, schedule scraping during off-peak hours of the site.

An important aspect of this rule is providing web administrators of the sites you scrape with an easy way to contact you. If you ever receive a complaint from a site you are scraping, you should either stop scraping the site or limit your activities to address the issue.

Don't Infringe Copyright

When scraping a site, you must always consider whether the data you plan to extract is protected by copyright.

Copyright is defined as an exclusive legal right over a tangible work—such as an article, painting, film, etc. Essentially, if you created it, you own it. For a work to be copyrighted, it must be original and tangible.

Common types of online materials that may be copyrighted include:

• Articles
• Videos
• Photographs
• Music
• Databases

Thus, copyright is highly relevant to scraping because most data on the internet (e.g., articles and videos) are copyrighted works.

However, there are situations where exceptions may apply to some or all of the data, allowing lawful use without infringing the owner's copyright.

Fair Use

Fair use is an exception that allows limited use of copyrighted materials. Typically, fair use includes categories such as criticism/parody, commentary, news reporting, teaching, scholarship, and research. An example of fair use is publishing short excerpts of articles with links, which generally falls under the fair use exception due to its transformative and limited nature.
The factors commonly used to determine whether fair use applies are:

1. The purpose and character of the use (i.e., whether it is transformative in some way);
2. The nature of the work (i.e., fact vs. fiction or published vs. unpublished); 
3. The amount of material used—the less you copy, the better; 
4. The effect on the potential market—the extent to which your use may deprive the owner of income or potential market opportunity.

Transformative Use

One factor in determining fair use is whether the use is transformative. Instead of distributing and storing exact duplicates or long excerpts of a scraped website, transform the content and its use in some way to avoid copyright infringement.

Facts

Facts contained in copyrighted materials are often not covered by copyright laws, so if you limit scraping to factual data, such as product names, prices, etc., such scraping is permissible.

Please note that different countries have different exceptions to copyright law, and you should always ensure that the exception applies in the jurisdiction where you operate.

Don't Violate GDPR

The introduction of GDPR completely changes how you can scrape personal data or personal data of EU citizens (and sometimes non-EU citizens). 

In this section, we briefly outline best practices when it comes to scraping personal data. Personal data is any data that can identify an individual:

• Name
• Email
• Phone number
• Address
• Username
• IP address
• Bank or credit card information
• Medical data
• Biometric data

If you do not have a "lawful basis" for scraping and storing this data, you will violate GDPR if any of the scraped data belongs to citizens. In the context of scraping, the most common lawful bases are legitimate interest and consent.

Consent

For consent to be a lawful basis for scraping a person's data, you need to obtain explicit consent from that individual to scrape, store, and use their data according to your intentions. This means that you or a third party must have been in direct contact with that person, and they agreed to terms that allow you to scrape their data.

Examples include companies like Mint.com, where users grant Mint consent to log into their online bank accounts and retrieve their transactions, so they can be tracked and displayed in a more user-friendly format on Mint.com.

Legitimate Interest

Most companies will find it very difficult to prove that they have a legitimate interest in collecting someone's personal data.

In most cases, only governments, law enforcement agencies, etc., have a legitimate interest in collecting citizens' personal data, as they typically collect it for the public good.

Beware of Login and Website Terms and Conditions

When you log in and/or explicitly agree to a site's terms and conditions, you are entering into a contract with the site owner, thereby agreeing to their rules regarding scraping. These terms may explicitly state that scraping any data on the site is prohibited.

This means you need to carefully review the terms and conditions you agree to if your crawlers must log in to scrape data, as they may specify that scraping is prohibited. You must always comply with the terms of any contract you enter into, including a website's terms and conditions and privacy policy.

Looking for data extracted from the internet? We scrape the data you need and deliver it in the format you want. Just tell us what you need.

Learn More About Web Scraping

ESK Solutions has been working in the scraping industry for 5 years. During this time, we have gained extensive experience and knowledge in data scraping.