Solution Architecture: How to Determine the Scope of a Web Scraping Project

As we have already mentioned, the ultimate goal of the requirements gathering phase is to minimize the number of unknowns, ideally reducing assumptions about any variable to zero, so that the development team can build an optimal solution for business needs.

Thus, precisely defining project requirements is the most important part of any web data scraping project.

In this article, we will look at four crucial steps required to define requirements for each web data scraping project, as well as questions to ask yourself when planning data extraction work.

The Requirements Gathering Process

The requirements gathering process can be divided into two parts: 1) understanding business needs and 2) defining technical requirements to meet those needs.

Business Need

Business need is at the core of any web data scraping project, as it clearly defines what goal it aims to achieve.

• Do you want to optimize your pricing strategy by monitoring competitors' prices?
• Do you want to track online sentiment towards specific companies?
• Do you want to grow your business by generating new leads for the sales department?
• Etc.

Once you have clearly defined what you want to achieve, you can begin gathering technical requirements.

At the start of every solution architecture process, our team pays great attention to understanding the client's core business needs and goals. Because when we truly understand your goals, we can help you select the optimal data sources for your project and best optimize the project if repurposing or compromises are needed due to technical or budget constraints.

You need to ask yourself the following questions:

• What is the goal of your business, what are you trying to achieve?
• What role can web data play in achieving this goal?
• What type of web data do you need?
• How will you use this data to achieve your business goals?

At first glance, these questions seem very simple, but we have found that in many cases clients come to us without a clear idea of what web data they need and how they can use it to achieve their business goals. Therefore, the first step is to clearly define what type of web data you need to achieve your business goals.

Technical Requirements

Once everyone clearly understands the business goals, it is time to define the technical requirements for the project, i.e., how we will extract the necessary web data.

Every web data scraping project has four key parts:

1. Data Discovery
2. Data Extraction
3. Extraction Scale
4. Data Output

We will examine each of these stages separately, explain why each is important, and what questions to ask yourself at each stage.

Step 1: Data Discovery

The first step of the technical requirements gathering process is to determine what data needs to be extracted and where it can be found.

Without knowing what web data you need and where to find it, most web data scraping projects become unfeasible. Therefore, a clear understanding of these basic factors is crucial for moving the project forward.

You need to ask yourself the following questions:

• Do you know which sites contain suitable data for extraction?
• How will the parser navigate to the data? I.e., how will the parser locate the data on the site.
• Is it necessary to log in to access the required data?
• Is it necessary to input any data to filter data on the site before extraction?
• Is it necessary to access the site from a specific location to see the required data?

By asking yourself these questions, you can realistically determine the scope of the project.

The next step in the requirements gathering process is data extraction...

Step 2: Data Extraction

At this stage, we need to clearly define what data we want to extract from the target web pages. Often, a page contains a huge amount of data, so the challenge is to focus specifically on the data the client needs.

One of the best methods to clearly define the scope of data to be extracted is to take screenshots of target web pages and mark the fields that need to be extracted. Often during meetings with our solution architects, we discuss this process with clients to ensure everyone understands exactly what data needs to be extracted.

You should ask yourself the following questions:

• What data fields do I want to extract?
• Is it necessary to extract any images on the page?
• Is it necessary to download any files (PDF, CSV, etc.)?
• Is it necessary to take a screenshot of the page?
• Is it necessary to reformat data into a different format (e.g., remove currency symbols from product prices)?
• Are all required data available on one web page?

A general rule is that the more data extracted from a page, the more complex the web data scraping project. Each new data type will require additional extraction, data quality assurance, and possibly additional technical resources (e.g., a headless browser if the data type is rendered using JavaScript, or if multiple requests are needed to access the target data).

After completing this stage, you can assess the project scope...

Step 3: Extraction Scale

So, by this stage you should have a very good understanding of the type of data you want to extract and how your parsers will locate and extract it. Next, we need to determine the scale of the web data scraping project.

This is a very important step because it allows you to estimate the amount of infrastructure resources (servers, proxies, data storage, etc.) needed for the project, the volume of data you are likely to obtain, and the project complexity.

When evaluating the scale of web data scraping projects, three main variables must be considered:

1. Number of sites
2. Number of records extracted from each site
3. Frequency of data extraction operations.

After completing steps 1 and 2 of the requirements gathering process, you should know exactly how many sites you need to extract data from (variable 1). However, estimating the number of records to be extracted (variable 2) can be somewhat more difficult.

Number of Records Extracted

In some cases (most often in smaller projects), it is sufficient to simply count the number of records (products, etc.) on a page and multiply by the number of pages. In some situations, the site even displays the number of records per page.

However, there is no one-size-fits-all solution for this issue. Sometimes it is impossible to estimate the number of results you will extract, especially if the scraper extracts hundreds of thousands or millions of records. In such cases, the only way to know for sure how many records the program will return is to actually run it. You can estimate, but you will never know for sure.

Data Extraction Frequency

The last factor to consider is how often you want to extract this data? Daily, weekly, monthly, one-time, etc.?

Additionally, does the data extraction need to be completed within a specific time frame to avoid changes in the source data?

It goes without saying that the more frequently you extract data, the larger the extraction scale. If you move from monthly data extraction to daily, you effectively increase the extraction scale by 30 times.

In most cases, increasing the scale from monthly to daily will not have a significant impact other than increasing the required infrastructure resources (server bandwidth, proxies, data storage, etc.).

However, this also increases the risk that your parsers will be detected and blocked, or that excessive load will be placed on the site's servers. Especially if the site does not typically receive a large amount of traffic each day.

Sometimes the scale of scraping is simply too large to perform at the desired frequency without crashing the site and requiring huge infrastructure resources. Typically, this problem only arises when extracting huge volumes of data at intervals of once per hour or less, for example, when tracking every product in an e-commerce store in real time. During the technical feasibility stage, we usually check this.

Deltas and Incremental Views

Another factor to consider when moving from a one-time data extraction to a regular project is how you plan to handle incremental queries?

Here you have several options:

• Extract all available data with each visit