Data Parsing

What is Data Parsing?

Data parsing, or web scraping, is the process of importing data from websites into files, databases, or spreadsheets. It is used to extract data from the Internet, either for the personal use of the scraping operator or for reusing data on other sites. There are many software applications for automating data parsing.

Data parsing is typically used for:

• gathering business intelligence for website content
• determining prices for booking or travel comparison sites
• finding potential customers or conducting marketing research using open data sources
• sending product data from e-commerce sites to online marketplace platforms such as Wildberries.

Data collection has legitimate uses, but it is often abused by malicious actors. For example, data scraping is frequently used to collect email addresses for sending spam or conducting fraud.

Website parsing can also be used to obtain copyrighted content from one site and automatically publish it on another site.

In some countries, it is prohibited to use automated email address collection methods for commercial gain, and it is generally considered an unethical marketing practice.

Data Parsing and Cybersecurity

Data collection tools are used by a wide variety of companies, not necessarily for malicious purposes. These include marketing research and business intelligence, web content and design, as well as personalization.

However, data parsing also creates problems for many companies, as it can be used to disclose and misuse confidential data. The website being scraped may not know that its data is being collected or what exactly is being collected. Similarly, legitimate data parsing may not ensure secure data storage, allowing attackers to access it.

If attackers gain access to data collected via web scraping, they can use it in cyberattacks. For example, attackers can use the collected data to carry out:

• Phishing attacks - attackers can use scraped data to refine their phishing methods. They can find out which employees have the necessary access permissions, or if someone is more susceptible to a phishing attack. If attackers manage to obtain personal data of executives, they can conduct targeted spear-phishing attacks.
• Password cracking attacks - attackers can crack credentials to infiltrate authentication protocols, even if passwords were not directly transmitted. They can study publicly available information about your employees to guess passwords based on personal data.

Data Collection Methods

Here are several methods commonly used to collect data from websites. In general, all website scraping methods extract website content, process it using a parsing engine, and generate one or more data files with the extracted content.

HTML Parsing

HTML parsing involves using JavaScript to process a linear or nested HTML page. It is a powerful and fast method for extracting text and links (e.g., nested links or email addresses), scanning pages, and extracting resources.

DOM Parsing

The Document Object Model (DOM) defines the structure, style, and content of an XML file. Typically, a DOM parser is used for in-depth examination of web page structure. DOM parsers can be used to access nodes containing information and collect web pages using tools like XPath. For dynamically generated content, parsers can embed web browsers such as Firefox and Chrome to extract entire web pages (or parts of them).

Vertical Aggregation

Companies with significant computing power can build vertical aggregation platforms focused on specific verticals.

These are data collection platforms that can operate in the cloud and are used to automatically generate and monitor bots for specific verticals with minimal human intervention. Bots are generated according to the information needed for each vertical, and their effectiveness is determined by the quality of the data they extract.

XPath

XPath stands for XML Path Language, which is a query language for XML documents. XML documents have a tree structure, so parsers can use XPath to navigate them, selecting nodes according to various parameters. A parser can combine DOM parsing with XPath to extract entire web pages and publish them on a target site.

Google Sheets

Google Sheets is a popular tool for data collection. Scrapers can use the IMPORTXML function in Sheets to save data from a website, which is useful if they want to extract a specific pattern or data from a website. This command also allows checking whether a site can be scraped or is protected.

How to Mitigate Website Scraping

For a website's content to be viewable, it typically must be transmitted to the machine of the user viewing the site. This means that any data accessible to a viewer is also accessible to a parser bot. You can use the following methods to reduce the amount of data that can be scraped from your site.

Rate Limiting User Requests

The speed of human visitors navigating a website is relatively predictable. For example, a human cannot browse 100 web pages per second, while machines can make multiple simultaneous requests. The number of requests can indicate the use of data scraping methods that attempt to collect your entire site in a short time.

You can limit the number of requests an IP address can make within a certain period. This will protect your site from exploitation and significantly reduce the speed at which data collection can occur.

Protection Against High Request Volume with CAPTCHAs

Another way to slow down data scraping efforts is to use CAPTCHAs. They require site visitors to complete a task that is relatively simple for humans but prohibitively difficult for machines. Even if a bot can pass a CAPTCHA once, it cannot do so in multiple instances. The downside of CAPTCHAs is their potential negative impact on user experience.

Regularly Change HTML Markup

For a data scraping bot to navigate a site and effectively analyze useful information, it needs consistent formatting. You can disrupt the bot's workflow by regularly changing HTML markup elements.

For example, you can nest HTML elements or change various aspects of the markup, making it difficult to scan consistently. Some websites implement random modifications on each render to protect their content. Alternatively, websites may change markup code less frequently to prevent long-term data scraping efforts.

Embedding Content in Media Objects

This is a less popular protection method that involves using media objects such as images. To extract data from image files, optical character recognition (OCR) must be used, since the content does not exist as a string of characters. This significantly complicates the content copying process for data scrapers, but can also become an obstacle for legitimate internet users who cannot copy content from the site and will have to retype or memorize it.

However, the methods listed above are partial and do not guarantee protection against scraping. To fully protect your site, deploy a bot protection solution that detects scraper bots and can block them before they connect to your site or web application.