Tips for Correctly Extracting Web Data Even from the Most Complex Sites Using Best Parsing Practices
Searching for data on the internet comes with certain difficulties, and that should come as no surprise if you've ever tried to find data online. Data available on the web does not follow any rules, structure, or standards, and this alone makes it hard to predict the problems you might encounter when searching for data online. The complexity multiplies many times when complex parsing needs to be performed at scale.
Web data, despite carrying invaluable business information, remains a tough nut to crack for many. This is where a specialized parsing service like ours comes to the rescue. At ESK Solutions, we handle tasks of all kinds, and each data extraction task is a challenge in itself.
However, the complexity of web data extraction varies greatly depending on several factors. Here are some of the most challenging tasks we have handled to date.
Project 1: Data Analysis of the Telecommunications Sector
Target Sites: Mobile operator websites
Required Data Points: All offers available for different customer segments
A company wanted to collect data on offers available on various mobile operator websites to provide a competitive advantage to their clients in this field. The requirement was feasible, even though it was extremely challenging. Below are the problems that made this project exceptionally difficult.
Project Challenges
1. Too Many Steps to Reach the Data
Offer information on the source sites was displayed only after entering specific variables such as the customer's zip code and offer type. This meant a long path to actually displaying the data. As a result, the parser had to be programmed to select every possible combination of inputs to effectively force the site to show all available data.
2. Frequent Site Changes
Since the mobile industry is fast-moving, the data on these sites tends to change very often. Mobile network providers frequently modify existing offers, discontinue some rate plans, and introduce new ones. This required careful monitoring and implementation of automated parsing to address site change issues.
3. Character Encoding Problems
Character encoding on a website is usually declared in its HTML code. However, some websites may have incorrect character encoding declarations or use more than one encoding across the site. This can greatly complicate parser setup and continue to cause problems if the site does not adhere to a single encoding.
4. Duplicate Data on the Site
Duplicate data can be a real issue, especially when the scale of data extraction from websites is large. Although we have a cleaning system designed to find and remove duplicate records from the dataset, duplicate data on the site further complicates the extraction process.
Project 2: Extracting Data from Hotel Search and Price Comparison Sites
Target Sites: Online travel portals and hotel websites
Required Data: Hotel listings and reviews
The client wanted to obtain hotel data from hundreds of travel sites worldwide to build a comprehensive hotel search system. Each target site required its own parser customization, and the individual issues that had to be resolved when setting up crawlers for 100+ sites made this project complex.
Project Difficulties
1. Blocking
Some sites in the target list had various blocking mechanisms aimed at automated parsers. This had to be handled by using an optimal frequency of GET requests and requesting only a nominal number of pages at a time. We avoided blocking mechanisms by following best practices for web parsing.
2. Discovery
Discovering URLs for extraction is a critical stage in the parsing and data extraction process, and the poor navigation structure of some target sites made it difficult for parsers to move seamlessly through pages. We solved this by setting several fallback rules for the URL discovery operation.
3. Character Encoding Problems
Character encoding issues were a challenge. We had to manually ensure that the encodings we used matched those of each target site. In cases where sites showed mismatched character encoding, we also set up some automated measures to address the problem.
4. Duplicate Data on Target Sites
Duplicate data present across multiple sites added complexity to this project. We let our cleaning system handle the deduplication of extracted data, and this approach proved beneficial for the client as well.
Project 3: Consulting Firm Needs Product Data for a Price Intelligence System
Target Sites: Popular e-commerce portals
Required Data: Product information
The client wanted to help one of their clients with price analysis and needed a service that could not only deliver product data but also perform matching. Although we typically do not handle processes beyond data extraction and delivery, we decided to take this on given the scale and interesting nature of the requirement.
Project Challenge
1. Product Matching
Product matching is a very complex aspect that goes beyond data parsing expertise. A strong matching system is necessary because each e-commerce portal will have slight variations in product descriptions, including product name and brand name.
Nevertheless, we developed an algorithm that can perform matching after data extraction and indexing on our side to meet the requirements of this unique project.
Data Parsing Services Are a Solution to Problems
Given the lack of standardization when it comes to data displayed on websites, parsing has been and always will be a challenging task that requires skill, experience, and knowledge to solve. That is why we emphasize the importance of using a fully managed solution when it comes to web data needs for businesses, regardless of their size or industry.