Web Scraping: Skill or Crime?

Business success no longer depends solely on staff and employee skills. Although they remain important, today’s businesses primarily rely on data. No matter how good your product or service is, without data on customers or clients, you cannot stay informed about market trends.

•     Desired features
•     Competitive pricing
•     Product updates

High-quality data has unprecedented applications. Since web scraping is legal, it’s no surprise that companies collect data from both internal and external sources. Among external sources, a small percentage of data is collected through surveys, while the rest is sourced from the internet. Data taken from the internet is publicly available—anyone can view and copy it. However, if you attempt to copy that same data automatically, certain rules may apply to you.

For example, you can scrape text content from websites. But if you attempt to scrape data from too many pages on a single site, causing the site to slow down, it could lead to legal issues. The difference here is that in the second scenario, your actions negatively affected accessibility for other users of that site.

Is It Legal to Use Data?

There is nothing illegal about collecting data, storing it in databases, and reusing it for business decisions, data analysis, and market research. However, if you wish to display that same data on your website or application, you may be bound by various terms and conditions. These terms will depend on the actual source of the data.

For content creators who take figures, graphs, and images from other online articles, simply providing attribution may suffice. In cases where product information or original content is taken and largely reused, you may need to request permission from the website owners. Some important factors to keep in mind when searching for data:

•     The website’s terms and conditions.
•     Personal data collection (especially from social media sites).
•     Regional intellectual property rules (since not all countries may have the same regulations).

It’s also worth noting that there is much talk about web scraping being in a legal gray area—nothing could be further from the truth. This activity is regulated and has its own rules, but as seen in numerous court rulings, it is not illegal!

Recent Court Rulings and Their Implications

1. Data relating to individuals is now protected by laws such as GDPR in the EU and CCPA in California. A Polish court was the first to rule, fining a data analytics company approximately €221,000 for scraping publicly available data on 6 million people. Even though personal data may be public, it is necessary to review existing laws and ensure compliance. In most cases, this may require sending a notice to each individual whose data will be collected or used, and only scraping data from those who do not object.
2. The LinkedIn vs. HiQ Labs case is arguably one of the most famous and lengthy legal battles over web scraping. HiQ Labs is a data analytics company that consumes public data from LinkedIn profiles. In the main ruling, the Ninth Circuit Court stated that authorization does not apply to public websites. It also indicated that selectively banning companies from accessing publicly available data could be considered unfair competition and have antitrust implications. This decision creates a level playing field where all parties collecting data from similar sources will be judged by the same criteria.
3. One of the earliest web scraping lawsuits was filed by eBay against Bidder’s Edge, an auction listing site. Although eBay won the case, arguing that Bidder’s Edge frequently scraping auction data from their site was damaging their systems and that other companies might follow suit, increasing server load, several judges later stated that this case cannot serve as a precedent in subsequent cases, and the same argument may not be persuasive.

Pros and Cons

Regardless of your end goal, there are some basic rules and best practices you can follow when collecting data from the internet.

• On sites like Twitter that provide APIs for developers, try to use them instead of scraping the website, even if you have to pay when API consumption exceeds the set limit.
• Aim for a healthy time interval of a few seconds when scraping multiple pages from the same site to avoid overloading the system.
• Make sure you know the regional rules and read the website’s terms and conditions before using scraped data for commercial purposes.
• When scraping data that is behind a login page, ensure you follow the rules of that site.

Web scraping is used not only by companies but also by researchers, students, data professionals, and virtually anyone trying to collect data. This often precedes testing machine learning models and algorithms.

If the data is not used in its original format but only the results or trends—as is the case most of the time—you will rarely encounter any obstacles. This holds true as long as the data was obtained correctly.

However, using scraped data for commercial purposes can be a different ballgame. If it is proven that you are using scraped data without any modification, exactly like the original webpage, you may be fined and possibly required to compensate the site from which you collected the data.