What is Web Scraping?

Definition of Web Scraping

Web scraping, also known as screen scraping, generally refers to the process of extracting, copying, saving, and reusing content from third-party websites on the Internet. Along with manual scraping, where content is copied manually, a number of tools for automatic copying of websites have emerged. A positive example of web scraping is the indexing of sites by Google and other search engines. In most cases, such indexing is welcomed since it is the only way for users to find the company pages they are looking for on the Internet. On the other hand, malicious scraping aimed at illegal appropriation of intellectual property violates copyright law and is therefore illegal.

How Does Web Scraping Work?

Web scraping uses various technologies and tools:

Manual scraping. The fact is that sometimes both the content and part of the source code of websites are copied manually. Cybercriminals resort to this method, especially when bots and other copying programs are blocked by the robots.txt file.
Software tools. Such web scraping tools as Scraper API, ScrapeSimple, and Octoparse allow creating web scrapers even for those with little programming knowledge. Developers also use these tools as a basis for building their own scraping solutions.
Text pattern matching. Automated matching and extraction of information from web pages can also be done using commands in programming languages such as Perl or Python.
HTTP manipulation. Using HTTP requests, it is possible to copy content from static or dynamic sites.
Data mining. Web scraping can also use data mining. To do this, web developers rely on the analysis of patterns and scripts into which site content is embedded. They identify the desired content and use a "wrapper" to display it on their site.
HTML parser. HTML parsers, familiar from browsers, are used in web scraping to extract and parse the desired content.
Copying microformats. Microformats are a commonly used component of websites. They can contain metadata or semantic annotations. Extracting this data allows conclusions to be drawn about the location of special data fragments.

Usage and Applications

Web scraping is used in a wide variety of fields. It is always used to extract data, often for completely legitimate purposes, but abuse is also common.

Web scrapers for search engines
Indexing websites is the foundation of search engines like Google and Bing. Only through web crawlers that analyze and index URLs can search results be sorted and presented.

Web scrapers are "bots," which are automated programs that perform specific and repetitive tasks.

Replacement of web services
Screen scrapers can be used as a replacement for web services. This is particularly interesting for companies that want to provide their clients with specific analytical data about a site. However, using web services for this is costly. Therefore, a more economical option is to use screen scrapers that extract data themselves.

Remixing
Remixing or mashups combine content from various web services. The result is a new service. Remixing is often done via APIs, but if such APIs are unavailable, screen scraping techniques are used here as well.

Misuse
Abuse of web scraping or web harvesting can have various goals:

Price scraping: One special form of web scraping is called "price scraping." In this case, retailers use bots to obtain information about competitors' product prices in order to intentionally undercut them and acquire customers. Due to the high price transparency on the Internet, customers quickly move to the next cheapest seller, leading to increased price pressure.
Content/product scraping: Instead of prices or pricing structure, in content scraping bots target the site's content. Attackers copy carefully crafted product pages from online stores, indistinguishable from the original, and use the expensive content for their own e-commerce portals. Online marketplaces, job boards, and classifieds are also popular targets for content scraping.
Increased load time: Web scraping wastes valuable server capacity: A large number of bots constantly refresh product pages in search of new pricing information. This slows down page loading for users, especially during peak times. If loading desired web content takes too long, customers quickly switch to competitors.
Phishing: Cybercriminals use web scraping to obtain publicly published email addresses and use them for phishing. Additionally, for phishing, criminals may recreate the original site, making a deceptively realistic copy.

How Can Companies Block Web Scraping?

There are several measures to prevent website scraping:

Bot management: Using bot management solutions, companies can precisely determine which bots are allowed to access information on the site and which should be considered malicious.
robots.txt: With the robots.txt file, site operators can specify which areas of the domain can be crawled and exclude certain bots from the start.
Captcha prompts: To protect against bot requests, Captcha prompts can also be integrated into websites.
Appropriate integration of phone numbers and email addresses: Site operators protect contact data from being harvested by placing them behind a contact form. Data can also be integrated using CSS.
Firewall: Strict firewall rules for web servers also protect against unwanted scraping attacks.

Scraping as Spam

In many cases, sites containing scraped content without a link to the source violate copyright law. Additionally, search engines like Google classify them as spam. For sites with original content, such spam sites also pose a danger because, in case of doubt, search engines may consider the legitimate site a duplicate of content and impose corresponding penalties. This leads to a significant drop in SEO ranking. To actively combat web scraping, companies and webmasters use, for example, special Google Alerts that provide information about suspicious content on the Internet.

Legal Framework: Is Scraping Legal?

Many types of web scraping are subject to legal regulation. This applies, for example, to online portals that compare prices from various retail chains. A corresponding decision was made by the German Federal Court in 2014: As long as technical protection preventing scraping is not overcome, there is no anti-competitive obstacle.

However, web scraping becomes a problem when it violates copyright law. Anyone who integrates copyrighted material on a website without linking to the source is acting illegally.

Furthermore, if web scraping is used for illegitimate purposes, such as phishing, the scraping itself may not be illegal, but subsequent actions may be.

Web Scraping: What You Need to Know

Web scraping is an integral part of the modern Internet. Many popular services, such as search engines or price comparison portals, would be impossible without automatic extraction of information from websites. However, abuse of this method poses a serious danger to companies, for example, when dishonest competitors extract valuable content from an online store and copy it to their own. The load from traffic generated by autonomous bot activity is also significant. Currently, bots generate about half of all website traffic. Therefore, effective bot management is one of the most important factors in protecting a company's website from scraping attacks.