Solution Architecture: Conducting Legal Review of Web Scraping

In this article, we will share with you the step-by-step process of legal review for every project related to web scraping that we undertake.

To ensure that your solution architecture complies with both legal regulations and best industry practices, we have compiled a checklist for convenience and to protect the reputation and integrity of web scraping as a practice. The regulation of personal and commercial data is constantly changing worldwide, and given the international nature of the Internet, establishing a clear legal practice in your solutions should be a top priority.

In this article, we will cover three critical legal checks that must be performed when analyzing the legal feasibility of any web data collection project, as well as questions to ask yourself when planning data collection work.

Preliminary Check: Define the Purpose of Use

Data comes in many forms and sizes. However, before starting to extract this data, it is necessary to determine the exact status and legality of extraction for each project.

There are three forms of data whose extraction may pose legal risks:

1. Personal data
2. Copyrighted data
3. Data behind a login

However, the first step in the legal review process is to determine the data usage scenario—i.e., what it will be used for and whether you have explicit consent from the data owners for its extraction, storage, and use.

The ultimate purpose of data use can significantly affect the legal status of scraping data from a website, especially in the case of personal data, which we will discuss later.

Therefore, the first step of any legal review process is to determine:

1. What will you use this data for?
2. Who owns this data? The site, an individual, no one, etc.
3. Do you have permission from the data owner to extract it?

Once all this is determined, you can proceed with the legal review.

Check #1 – Personal Data

Personal data, also known as personally identifiable information (PII), is any data that can be used to directly or indirectly identify a specific individual. With growing awareness of how personal data is used, the extraction of personal data has led to increasingly strict data protection regulations—a prime example being the General Data Protection Regulation, or GDPR.

First and foremost, you need to check whether you plan to extract personal data in any form. Common examples include:

• Name
• Email
• Phone number
• Address
• Username
• IP address
• Date of birth
• Employment information
• Bank or credit card information
• Medical data
• Biometric data

If you are not extracting any personal data, you can move on to the next stage of legal review. However, if you are extracting any of the above types of personal data, then you need to examine the data protection regulations related to that data.

Check #2 – Copyrighted Data

Copyrighted data typically describes content owned by legal entities or individuals who have clear control over its reproduction and capture. The fact that web data is publicly available on the Internet does not mean anyone can extract and store it.

In some cases, the data itself may be copyrighted, and depending on how/what data you extract, you may be found to infringe the owner’s copyright, creating additional risks for users of that data.

First, you need to check whether you plan to extract data in any form that may be copyrighted. Common examples include:

• Articles
• Videos
• Images
• Stories
• Music
• Databases

If you are extracting any of these forms of web data, then you need to determine whether extracting and using this data in your projects would violate copyright.

Such cases must be evaluated on a case-by-case basis, as copyright issues are often not black and white like personal data, and sometimes they can be overcome if your case has a valid copyright exception. Methods for doing so include:

• Fair use: For example, instead of extracting all the data from an article, you extract short excerpts, which may be considered fair use.
• Facts: Facts are generally not subject to copyright laws, so if companies limit themselves to only factual data—i.e., product names, prices, etc.—this may be permissible without copyright infringement.

Database Rights

Database rights are a subset of copyright that requires separate explanation. A database is an organized collection of materials that allows a user to search and access individual pieces of information contained in those materials.

Database rights can create additional risks when using web data in your projects if that data was not extracted appropriately.

In the United States, a database is protected by copyright if its selection or arrangement is original and creative. Copyright protects only the selection and organization of the data, not the data itself.

In the EU, databases are protected under the Database Directive, which provides much broader protection for EU databases. The directive has two objectives: (1) IP protection, as in the US, and (2) protection of the labor and risk involved in creating the database.

If you believe the data source may be subject to database rights, decision-makers should definitely consult with their legal counsel before proceeding with data scraping and ensure they either:

• only scrape a portion of the available data;
• scrape only the data itself and do not copy its organization; and
• try to limit the amount of scraped data to factual or other non-copyrighted data.

Copyright can be a complex topic, so it is always best to consult with a qualified legal professional before using potentially copyrighted data in your projects.

Check #3 – Data Behind a Login

Extracting data from a website that first requires logging in to access the data can raise potential legal issues. In most cases, logging in requires agreeing to the site’s terms and conditions, which may explicitly state that automated data scraping is prohibited.

In this case, you need to review the terms and conditions to determine whether you are violating them by scraping data from the site. Since the terms of use of some sites can be quite confusing, it is recommended that an experienced legal professional review them before scraping data from behind a user login.

Our Recommendation

To comply with modern data protection requirements, it is critical to keep the information provided by the legal department up to date and ensure regular oversight by data protection professionals. Legal reviews are an integral part of regulatory compliance and demonstrate good faith. Additionally, consistent legal evaluation of projects helps streamline the scraping process and provides complete confidence that scraping remains respectful and productive.