Solution Architecture: Determining Technical Feasibility of a Web Scraping Project

After completing the legal review of a potential project, the next step in every project should be an assessment of technical feasibility and complexity for successful implementation.

Preliminary testing and planning will help save countless hours that could be wasted if you start developing a full-fledged solution and hit a brick wall.

The Technical Analysis Process

The technical analysis process focuses on four key components of the web scraping development process:

• Data Discovery
• Data Extraction
• Extraction Scale
• Data Delivery

We will examine each one individually, breaking down the technical questions you should ask yourself at each stage.

Using the project requirements gathered during the requirements gathering process, you should have all the necessary information to begin assessing the technical feasibility of the project.

Step 1: Data Discovery

The first step of the technical feasibility process is to examine the possibility of crawlers accurately discovering the required data, as defined in the project requirements.

For most projects, when you know the exact websites from which to extract data and can manually navigate to the required data, there are usually no technical issues with developing crawlers to discover that data. The crawler simply needs to be designed to reproduce the sequence of user actions required to access the data.

Generally, the main reasons for technical issues at the data discovery stage are that you don't know how to manually discover all the data:

• You don't know which sites contain the data you need; or
• You cannot filter data on the target sites to extract only the needed data.

For example, the most common technical issue we encounter during the data discovery stage is when a client wants to extract a specific type of data but either doesn't know which sites to extract it from, or has a massive list of over 100 sites they would like to scrape.

In such cases, we can explore obtaining the necessary data using a number of approaches at our disposal:

• Data Aggregators - Thanks to our extensive experience extracting data from the web, we can sometimes work with the client to identify existing sites that aggregate data from the internet and, if scraped, could meet their data needs. This is an ideal solution if the client knows what data they want to extract but doesn't have a clear idea of which sites contain the required data.
• Broad Search - In some cases, Zyte can configure a search within a specific region of the web (e.g., a country domain) to find the client's needed data. This solution can be useful for certain types of data (corporate or personal), but because a generic crawler is used for extraction, data accuracy will be much lower than with custom-developed crawlers. We often recommend this solution only if data quality requirements are low.
• Artificial Intelligence - If a client has a list of over 100 sites from which they want data but lacks the budget to develop custom spiders for each, we often recommend considering AI-powered data extraction technology, such as Zyte Automatic Extraction API. This solution allows data extraction from known sites without the need to develop custom code for each site, significantly reducing project costs. Currently, Zyte Automatic Extraction API is only available for extracting product and article data, but coverage for other data types is planned in the coming months.

Each of these approaches has its limitations, primarily regarding data quality and coverage. Therefore, we generally recommend developing custom crawlers for each site if data quality is a priority.

Step 2: Data Extraction

The next step is to verify the technical feasibility of the data extraction stage.

Here, the focus is on verifying the ability to accurately extract data from the target sites and assessing the required complexity. Our solution architecture team will conduct a series of tests to design the extraction process and ensure the technical feasibility of extracting data with the required quality. These tests will check:

• JavaScript/AJAX - Since modern websites increasingly use JavaScript and AJAX to dynamically display data on web pages, it may be necessary to use a headless browser to render this data for crawlers or develop custom code to execute parts of JavaScript without a headless browser. The solution architect will run a series of tests to determine whether the target data is rendered using JavaScript or AJAX.
• Number of steps required to extract data - In some cases, all target data is not available on a single page, and the crawler needs to make multiple requests to obtain it. In such cases, the solution architect determines the number of requests required, which defines the amount of infrastructure needed for project implementation.
• Complexity of iterating over records - Some sites have more complex pagination (infinite scroll, etc.) or formatting structure, which may require the use of a headless browser or complex crawling logic. The solution architect will determine the type of pagination and crawling logic needed to access all available records.
• Data validation - A key component of any web scraping project is maintaining high data quality and coverage. Therefore, even before the project starts, our architect will assess the complexity of ensuring high data quality and coverage as the project scales.
• Maintenance complexity - The solution architect not only assesses the complexity of extracting target data from the current website but also studies the site's history and industry trends to determine the likelihood of site changes that could break the crawlers.

After completing this stage, the solution architect examines the feasibility of extracting data at the required scale...

Step 3: Extraction Scale

After the feasibility of data discovery and extraction at a small scale has been verified, the next step is to ensure that the project can be executed at the required scale and speed to meet project requirements. This is often the most complex and problematic area when it comes to web scraping, and it is where the experience and knowledge of a good crawling engineer come into play.

Provided there are no legal or serious data discovery/extraction issues, a small parser can usually be developed to extract data without encountering any problems. However, when scaling up the parser's speed and scale, issues can quickly arise:

• Test scrapes - Our solution architect usually conducts a series of test scrapes to identify any bottlenecks regarding maximum scraping speed. This is often relevant for small sites or when the project requires tight deadlines to complete data extraction, but there is a risk of overloading the site's servers.
• Anti-bot countermeasures - Our solution architect will run target sites through our internal analytical tools to detect the presence of anti-bot countermeasures, captchas, or CDNs, which will increase project complexity and limit the ability to extract data at the required scale or frequency. The presence of these technologies can increase the risk of bans or reliability issues.

Armed with this information, our solution architecture team can gain a deep understanding of the project's complexity and the challenges involved in implementing and maintaining it at scale.

Step 4: Data Delivery

The final stage of technical feasibility is verifying the viability of the data delivery format and method. In most cases, very few problems arise at this stage because we have a wide variety of data formats and delivery methods available to meet any client requirements.

However, some options may be more complex than others, as they require more development time (e.g., developing a custom API, etc.).

Most often, project complexity increases if post-processing of data or data science is required. This can significantly increase project complexity.

So there you have it—the four steps to conducting a technical feasibility analysis for a web scraping project.