Solution Architecture: Designing a Well-Optimized Web Scraping Solution

Components of Web Scraping

There are several essential components that every web scraping project needs if you want to reliably extract high-quality data from the internet at scale:

• Parsers
• Scraping Hosting
• Proxy Management
• Data Quality Monitoring and Control
• Scraping Maintenance

However, depending on project requirements, other technologies may be needed to obtain the necessary data:

• Headless Browser
• Intelligent Parsers
• Data Post-processing

The amount of resources required for project development and maintenance will be determined by the type and frequency of data usage, as well as project complexity.

Web Scraping Design

Discussing the components of web scraping projects is good, but the best way to understand how to build a solution is to look at real-world examples.

Product Monitoring

In the first example, we'll look at one of the most common web scraping use cases: product monitoring. Here, we'll examine a typical scenario:

Project Requirements: The client wanted to extract products from specific product pages on Amazon.com. The client was to provide a package of search queries, and parsers had to find these keywords and extract all products related to them (~500 keywords per day).

• Product URL
• Product Title
• Manufacturer/Publisher
• Category
• Image URLs
• Description
• Product Description
• Product Information
• ASIN
• Rating
• Up to 20 one-star reviews
• Up to 20 five-star reviews
• Canonical URL
• Title
• Article Image (top image)
• Text
• Media URLs
• Authors
• Publication Date
• Summary

The extracted data would be used in a product analysis tool designed for consumer brands that want to track their products and those of their competitors.

Legal Assessment: Generally, extracting product data does not pose significant legal issues provided that the parser (1) does not have to perform a search behind a login (which is often not the case), (2) only extracts factual information or non-copyrighted information, and (3) the client does not intend to recreate the entire target site's store, which could raise database rights concerns. In this instance, the project had virtually no legal issues.

Solution: After project evaluation, the solution architect designed a custom solution to meet the client's requirements. The solution consisted of three main parts:

• Data Discovery - the decision to manually develop data detection crawlers for the site to automatically enter product keywords into the search box, navigate to the corresponding "shelf page", and extract product URLs for each product. These parsers would then page through each "shelf page" until all product URLs were extracted.
• Data Extraction - after extracting product URLs, they would be passed to data extraction parsers that would visit each individual product page and extract the required data.
• Scale and Reliability of Data Extraction - given the volume of requests the parsers would make to the same site each day, it was obvious that a proxy solution was necessary.

Result: Currently, the developed parsers extract ~500,000 products from the site daily, which the client feeds directly into their product monitoring application.

Media Monitoring

In the next example, we'll look at a more complex web scraping project that required using artificial intelligence to extract article data from over 300+ news sources.

Project Requirements: The client wanted to develop a news aggregator application that would collect news content for specific industries and interests. The client provided an initial list of 300 news sites to scrape, but this number could increase as the company grew. The client needed to extract every article within specific categories from all target sites, crawling them every 15 minutes or every hour depending on the time of day. The client required the following data from each article:

• Canonical URL
• Title
• Article Image (top image)
• Text
• Media URLs
• Authors
• Publication Date
• Summary

After extraction, this data had to go directly into the client's application, so ensuring high data quality and reliability was a critical requirement.

Legal Assessment: When extracting articles, one must always remember that articles are copyrighted material of the target site. You must ensure that you are not simply copying the entire article and republishing it. In this case, since the client aggregated content internally and only republished titles and short excerpts, it was determined that this project might fall under the fair use doctrine under copyright law. When extracting articles, various copyright considerations need to be addressed, so it's always best to consult with a lawyer first.

Typically, developing a robust and scalable parser for a single site takes an experienced parser 1-2 days. A rough calculation quickly shows that manually developing 300+ parsers would be a very costly project if each parser takes one working day.

Solution: After conducting a technical feasibility assessment, the solution architect designed a custom solution to meet the client's requirements. The solution consisted of three main parts:

• Data Discovery - the decision to manually develop data detection parsers for each site, which would survey the target site and find articles to extract. Since these parsers were searching listing pages, they were much easier to develop and highly portable across sites, significantly reducing development time.
• Data Extraction - after extracting article URLs, they were sent to us, where we already used AI-powered parsers to extract article data without needing to manually develop data extraction parsers for each site.
• Scale and Reliability of Extraction - as with the previous project, given the number of sites being scraped, this project required a sophisticated proxy management system and a data quality assurance layer. For this project, the solution architect recommended using proxy solutions and using open-source projects for data quality monitoring.

Result: This project was successfully delivered for the client, who can now extract 100-200 thousand articles per day from target sites for their news aggregation application.

Conclusion 

This is what the four-step process for developing solutions for web scraping projects looks like.