Evolution of Web Scraping as a Market Segment

Data scraping from websites and its structuring as a market segment has come a long way from an emerging technology to an integral part of many businesses. The first parsers were developed for a much smaller Internet (around 100,000 web pages), but today some popular sites contain millions of pages. The evolution of scraping has come a long way.

Although early parsers could collect text and images, modern parsers are much more robust because, in addition to data collection, they can also monitor web applications for vulnerabilities and availability. But not all parser libraries are designed for the entire Internet. Here is how some web parsers differ from each other:

Different Parsers Have Different Focus Areas

Parsers are created to solve specific tasks, and therefore some of them focus more on at least one of the following three parameters:

1. Quality - such parsers have broad coverage, good resources, detect new (related) pages, and close the loop.

2. Representativeness - ensuring completeness of target data copies; such parsers are usually focused on specific sites and crawl deep areas of the Internet.

3. Freshness - in some situations, having the latest data copies is vital, for example in the case of price scraping from marketplaces and their updates. Such parsers are typically suitable for areas where there is a need for the freshest data, perhaps with near real-time frequency.

First Parsers

Here is a brief history of the first generation of web scraping scripts:

1. RBSE spider - developed and used as part of NASA's Repository Based Software Engineering (RBSE) program in 1994 at the University of Houston. It was created by David Eichmann from NASA using Oracle, C, and wais.

The main purpose of this parser was indexing and statistics collection. At the time this parser was created, the size of the Internet was only about 100,000 web pages.

2. WebCrawler - created by Brian Pinkerton from the University of Washington and launched on April 20, 1994, WebCrawler was the first search engine that operated based on a website parser. According to Wikipedia, WebCrawler was the first search engine to provide full-text search.

3. Archive.org - The Internet Archive, also known as The Wayback Machine, used Heritrix as a parser to archive the entire Internet. Written in Java, it is licensed as free software, accessible either via a web browser or a command-line tool.

It is also worth noting that Heritrix is not the only crawler used in building the Internet Archive. In fact, most of the data was contributed by Alexa Internet, which collects data for its own purposes using a parser called ia_archiver.

Second Generation Web Parsers

Second generation parsers are either a) specialized parsers or b) large-scale parsers. Focused parsers are those targeted at specific sites, personally configured and movable, such as SPHINX and Mercator.

Search engine providers like Google, Lycos, and Excite developed parsers capable of global scaling and data indexing.

From In-House Development to DIY Tools

As soon as businesses began to take scraping seriously, they started developing in-house solutions. The development team was tasked with collecting relevant data from the Internet.

Developers used open-source technologies and customized them to their needs. However, this required dedicated teams for end-to-end monitoring of the ingestion pipeline, and it turned out to be quite challenging to manage at scale across the company.

Moreover, development teams could not focus on their actual solution that the scraping setup was supposed to complement.

Given the market volume where such solutions were offered, there was a sharp increase in the number of "do-it-yourself" (DIY) tools. They served a specific purpose: since the web still had a relatively consistent structure, it was easy to extract data using these tools.

Several DIY tools, such as Apache Nutch and Scrapy, were the most popular. Development teams began using these tools to access web data faster with minimal technical barriers.

However, as the Internet grew in complexity, the limitations of these tools quickly became apparent. DIY tools had entirely different limitations—they operated on a limited scale and provided much less control over fixing failures.

Evolution of Data Scraping as an Outsourcing Model

Subsequently, specialized data scraping solutions emerged—a model where an external provider handles all the work of finding and extracting data. Clean and structured data is typically provided on an ongoing basis through a pipeline.

This means that companies can focus on their core business and let someone else take care of providing the data they need.

Although early web pages had limited functionality and mostly uniform structure, the rapid expansion of the web and the increasing complexity of web page elements made data retrieval somewhat challenging. Specialized scraping services operate either as a SaaS product or under a Data-as-a-Service model.

Both models have typical advantages and disadvantages, and achieving the right fit depends on the user's business scenario, i.e., the nature of requirements, availability of time, resources, labor, and know-how.

Moreover, today there are significant tactical challenges in the world of scraping. Here are some of them:

• IP address blocking by target websites,
• risk of DDoSing target site servers,
• non-uniform web structures,
• AJAX elements,
• need for real-time delays,
• other anti-scraping tactics and tools such as ScrapeShield (provided by the popular CDN Cloudflare).

Nevertheless, solutions have evolved accordingly with the problems, and most of the above issues are easily resolved at ESK Solutions through various mechanisms.

As the complexities associated with crawling websites increase, in-house teams may not be the most efficient way to perform crawling. In such a situation, an outsourcing partner can be a useful ally, as they have specialized expertise in this area and have overcome most common problems.

Thus, the return on investment from using outsourcing services is usually much higher than developing in-house capabilities.