Parser Scaling: Queues, Workers, Distributed Collection

Challenges of Scaling Data Parsers

Modern businesses increasingly rely on up-to-date structured data from external web sources. Price monitoring, competitor analysis, aggregation of news and product feeds require regular collection of millions of pages. A single parser script quickly hits physical limitations: network bandwidth, limits on the number of simultaneous connections, and computational resources. Attempting to increase throughput through vertical scaling (a more powerful server) provides a temporary effect and sharply increases costs.

Industrial-scale data collection requires a fundamentally different architecture—distributed, fault-tolerant, and easily scalable. This is where message queues and isolated workers come to the forefront. This approach allows horizontal scaling of the system by simply adding new nodes without affecting already running components. For companies that need development of site parsing, building a scalable architecture from the start is profitable—it saves time and resources as load increases.

Architecture with Queues and Workers

Why Queues Are Needed

A message queue acts as a buffer between the task generator and the executors. The generator (scheduler) forms tasks—URLs, required selectors, request parameters—and places them into the broker. Workers independently read the next assigned task, perform parsing, and save the result. The queue addresses several key challenges at once:

• Decoupling producers and consumers. The scheduler does not know how many workers are active and does not wait for their response. Handlers can be added or removed on the fly.
• Managing peak loads. If the rate of incoming tasks exceeds processing speed, the queue accumulates them, preventing request loss.
• Guaranteed delivery. With proper broker configuration, a message will not be lost in the event of a single worker failure and will be handed off to another.
• Prioritization and routing. Depending on urgency or topic, tasks can be distributed into different queues processed by dedicated worker pools.

Workers: Unit of Execution

A worker is a process (less commonly a thread) that polls the queue in a loop, receives a task, performs it, and confirms successful processing. In the context of parsing, a single worker can either download HTML pages or extract data (often both phases are combined). Key properties of a well-designed worker:

• Isolation. A failure within a worker should not affect others; they are usually launched in separate containers (Docker) or even on different physical machines.
• Repeatability. The same URL given to two different workers should yield identical results (ceteris paribus).
• Fault tolerance. The worker must handle transient network errors, timeouts, response codes 429 and 5xx with exponential backoff before retrying or discarding the task to a dead letter queue.

Such an architecture fits perfectly into microservices-based solutions, including web development of complex portals with aggregation functionality or SaaS application development, where collecting data from external platforms becomes a key part of the product. Isolated workers simplify testing, deployment, and monitoring, and the use of queues naturally integrates with modern CI/CD pipelines.

Horizontal Scaling: Redis and RabbitMQ

Redis as a Queue Broker

Redis is an ultra-fast in-memory data store that supports data structures like List and Pub/Sub, which can easily be adapted for simple queues. Popular worker frameworks (e.g., Celery in Python) can use Redis as a broker out of the box. Advantages of Redis:

• Sub-millisecond latency for inserting and retrieving messages, which is critical when tasks are arriving at a very high rate.
• Ease of deployment — a single process with no extra dependencies.
• Flexible memory usage. It can be combined with disk persistence (RDB/AOF) to protect against data loss on restart.

However, the cost of speed is the lack of built-in guaranteed delivery. If a worker extracts a message and crashes before completion, the task may be lost. For many scraping scenarios where occasional re-crawling of a page is acceptable, this is acceptable. In schemes requiring strict one-time processing, additional mechanisms are used — for example, blocking pop with timeout and manual acknowledgment.

RabbitMQ for Reliable Delivery

RabbitMQ is a full-featured broker that implements the AMQP 0-9-1 protocol. It is designed for guaranteed delivery even in the event of node failures. Key features:

• Acknowledgements. The message is removed from the queue only after an explicit signal from the worker of successful processing.
• Durable queues and messages. Data can survive a broker restart.
• Complex routing. Exchanges, bindings, routing keys allow building topologies with stream segregation by task types, sources, or priorities.
• Dead Letter Exchange (DLX). Tasks not processed after N attempts are automatically moved to a separate queue for analysis or manual intervention.

The choice between Redis and RabbitMQ is determined by the requirements for reliability and infrastructure complexity. For high-frequency collection of public data, Redis with manual retry handling is often sufficient. When scraping is critical to the business and the loss of even one relevant page is unacceptable, for example when collecting real-time prices for financial applications, RabbitMQ is preferred. In large projects, a hybrid is often used: a fast buffer on Redis, from which tasks are moved to RabbitMQ for guaranteed processing of long-lived instances. To manage clusters and automatically scale computing resources, such solutions are often wrapped with cloud services, allowing elastic addition of capacity during peak hours.

Horizontal Scaling of Workers

As collection volumes grow, adding performance becomes trivial: just increase the number of worker nodes. They all connect to the same queues, and the broker distributes tasks according to its strategy (round-robin, by number of unacknowledged messages, etc.). To coordinate and prevent duplicate crawling of the same URLs, distributed data structures such as Redis Bloom filters or a centralized state store are used.

Horizontal scaling is especially effective in a cloud environment: you can deploy dozens of low-cost virtual machines or containers, each with several workers, and dynamically change their number depending on the queue depth. Orchestration systems (Kubernetes) handle auto-scaling based on metrics, such as queue length in RabbitMQ, very well.

Frequently Asked Questions

How many workers are needed to scrape 1 million pages per day?

The exact number is determined by the complexity of the target sites, the duration of HTTP responses, and the requirements for pauses between requests. As a rough guide — with an average page load time of 500 ms and one worker performing sequential requests with a pause, a single thread can process no more than ~170k pages per day (not counting the time for parsing itself). For a million pages, around 6 workers would be needed under ideal conditions. In practice, taking into account network delays, proxy rotation, and error handling, 20–50 workers are typically budgeted. This is just an example: real performance is always checked through load testing on similar sources.

How to avoid blocks when mass scraping?

Anti-blocking techniques include: rotating a pool of residential or datacenter proxies, dynamically changing the User-Agent and headers, rate limiting, random delays between requests, using headless browsers only where necessary, and timely updating of cookies/sessions. Many clients who come to us for ordering website parsing additionally ask for seamless integration of the collected data into their accounting systems, including CRM systems for automatically filling product cards or contacts.

If task processing speed is critical and the acceptable loss of a small percentage of messages is not a disaster, Redis will be optimal due to minimal overhead. If, however, each message has commercial value and requires guaranteed processing, the choice is clearly RabbitMQ. In practice, both tools are often combined: Redis as a fast buffer in front of the main broker, RabbitMQ for managing long-lived and retry tasks. This approach, for example, is used in enterprise systems such as corporate intranet portals, where news and market analytics are aggregated from external sources.

Can a parser be scaled without queues?

Theoretically yes — through load balancing across multiple script instances managed by an external scheduler on a schedule. However, this approach lacks the flexibility that queues provide: prioritization is difficult, retrying failed tasks is hindered, and there is no ability to flexibly scale the worker pool up or down in real time. Once the data volume exceeds the scale of a single server, the architecture with queues and workers becomes the de facto industry standard.