Scraping Real Estate Aggregators: Maps, Filters, and Anti-Scraping Bypass Using Geo-Proxies

The real estate market is one of the most data-rich segments, and aggregators like Cian, Avito Real Estate, and DomClick have become the main sources of information for analysts, real estate agents, and investors. Attempting to obtain this data automatically runs into serious layered protection: dynamic maps, session-bound filters, geolocation redirects, and anti-scraping mechanisms. In this article, we will examine in detail why scraping real estate is difficult, how geo-filters and maps work, and why geo-proxies are the key to stable ad collection.

Why Real Estate Aggregators Are a Difficult Target for Scraping

Aggregator sites combine hundreds of thousands of listings from developers and private individuals, providing filtering by dozens of parameters and visualization on a map. Behind the seeming simplicity of the frontend lie powerful headless backends that were designed from the start to protect the most valuable data from automated collection.

• Dynamic content. The main search results and map data are loaded via internal APIs with tokens generated on the fly. Direct HTML copying will not work.
• Geo-dependent behavior. The same URL may return different results depending on the user's IP address: algorithms shift the output toward the local market, and when a request comes from an atypical location, an additional check is triggered.
• Multi-layered anti-scrape. In addition to classic request rate limiting, JavaScript challenges, captchas, behavioral pattern analysis (mouse movement, timing), and dynamic DOM structure changes are employed.

Our experience in developing parsing and price monitoring systems shows that without a preliminary analysis of each platform's architecture, you can spend months banging your head against closed doors. Below, we will uncover the most critical nodes—maps and filters.

Maps and Geo-Filters: Key Obstacles

Most aggregators use their own map implementations based on libraries like Mapbox GL JS or closed APIs on top of Yandex.Maps / Google Maps. When trying to automate map interaction, several key points must be considered:

Tile Loading and Bounding Box

Object data is not transferred all at once for the entire area—it is loaded in chunks as the map moves. Each pan or zoom generates a request with viewport parameters (latitude, longitude, zoom, bounding box boundaries). A simple parser emulating such requests will quickly hit a limit if bounding box coordinates are repeated from a single IP. Moreover, platforms impose restrictions on the area size: you cannot request "all of Moscow" with one bounding box—the server will simply truncate the result set or return an empty result.

Geocoding and Session Bindings

When filters are used (radius from metro, district, coordinates), a geocoding chain is triggered—converting an address to coordinates. These requests are cached on the aggregator side, but per session: if the script runs without preserving cookies and without IP rotation, the cache does not work, and each call generates an expensive request to the geocoder. Protection systems quickly react to an abnormally high frequency of such calls from a single identifier.

Localized Results and Prices

Price values, contact data, and property availability may change depending on the region determined by IP. For example, a property in Kazan viewed from an IP address registered in Moscow may display with Moscow coefficients or not appear in the results at all due to regional restrictions. Without using geo-proxies, it is impossible to get a representative picture of the market nationwide.

To reliably overcome these barriers, we at ESK Solutions recommend an architecture with a distributed network of residential geo-proxies integrated at the parser session level. Such solutions are also in demand in niche areas—for example, when creating SaaS platforms for real estate analytics, where data needs to be updated across thousands of locations.

Geo-Proxy: Operating Principle and Selection Criteria

Ordinary datacenter proxies with IP ranges belonging to hosting providers are practically useless for real estate aggregators: major platforms maintain blacklists of such subnets. Residential proxies and mobile proxies yield a much higher percentage of successful requests but require fine-tuning to bypass geo-dependent filters.

Types of Proxy Servers

• Residential. Addresses from pools of real internet service providers. High trust, but limited availability for target cities.
• Mobile. IPs from cellular operators. Often provide maximum local geo-targeting and can bypass CAPTCHAs.
• ISP proxies. Static addresses registered as belonging to ISPs but hosted on datacenter equipment. A compromise option if IP consistency for session retention is important.

Proxy Geo-targeting

For parsing Kazan real estate, the proxy must be located in Kazan or at least in the same region (Tatarstan). Modern proxy providers allow rotation selection at the city level. However, it's important to understand that even with an exact city match, an IP may be deemed suspicious due to abrupt location changes within a single session. Therefore, the parser architecture should support long-lived sessions with sequential IP changes only within the same city.

Example of selection: to cover 50 cities with a daily update frequency, a pool of 150–200 residential IPs with geo-binding capability will be required, along with a task orchestration platform — the kind we often incorporate when designing cloud services for parsing. Without centralized proxy and request queue management, the project risks becoming an unmanageable zoo of scripts.

Parser Architecture with Geo-Proxying

Let's move on to technical implementation. We typically organize data collection from real estate aggregators into a multi-tiered architecture:

Task Management Layer

Coordinates which cities and property categories (residential, commercial, new developments) are currently being processed. The scheduler distributes tasks among workers considering proxy geographical affinity. This reduces the number of location switches and extends the pool's lifespan.

Session and Browser Emulation Layer

Since almost all map interfaces run on JavaScript, we use headless browsers (Playwright/Puppeteer) with geolocation spoofing via Sensor API emulation and injecting coordinates into viewport parameters. Each worker launches an isolated session using its own proxy address. It's critical to manage the browser fingerprint: WebGL vendor, Canvas, screen resolution, and time zone must match the target region. This is exactly the kind of problem that comprehensive custom web development under non-standard integrations solves.

Anti-scraping Bypass Layer

In addition to proxies, we implement:

• Intelligent delays. Requests emulate real user behavior: pauses between map zooms, random order of viewing listings, scrolling to the end of the page.
• Header spoofing and rotation. Each session gets a unique User-Agent and Accept-Language matching the region.
• CAPTCHA solving. Integration with automated recognition services (if CAPTCHA does appear), but the main goal is to prevent its occurrence through high-quality traffic.
• Parsing internal APIs. Map tiles and object lists are often served in JSON format with proper token emulation. We analyze the browser's network traffic and reproduce the necessary requests directly, bypassing rendering, which speeds up collection significantly.

Storage and Post-processing

Collected data is cleaned, deduplicated, and enriched with geo-coordinates for subsequent loading into analytical systems. Here also proxy success monitoring is deployed: if the failure rate for a specific city exceeds a threshold, the proxy manager requests address rotation.

The final scheme, although requiring initial investment in design, pays off with the stability of the data flow. For businesses considering launching their own product based on such data, we offer a cycle from a parser prototype to a ready-made service for collecting and monitoring prices with a fully managed cloud infrastructure.

Frequently Asked Questions

Can you scrape aggregators without geo-proxies, using only datacenter IPs?

Technically, yes, if the volume of requests is minimal and you are not interested in correct regional association. However, in practice, after just a few dozen requests to the map from a hosting IP, most major aggregators enable additional verification or a captcha. Such an approach is unsuitable for regular monitoring.

How often do you need to rotate proxies for a single city?

This depends on the aggressiveness of the specific site's protection. On average, when collecting once a day for a single city, you can use the same IP for several days if the requests are naturally distributed over time. With a higher frequency (once an hour), rotation should occur at least every 5–10 sessions to avoid accumulating address "fatigue".

What to do if the map requires authorization or an API key?

Many platforms require registration to access extended data. In this case, session proxies must support long-lived cookies and not break the session after each request. Sometimes it's easier to emulate a registered user through a headless browser and replace authorization tokens. When designing corporate solutions, we additionally integrate CRM integrations for automatic loading of received leads.

Is scraping real estate aggregators legal?

The legislation of different countries interprets the automated collection of publicly available data differently. In most jurisdictions, the collection of factual information (prices, property characteristics) is permitted provided that robots.txt is respected, no circumvention of technical protection measures, and no overloading of servers. We always recommend that clients conduct a legal audit of the target resource and obtain a written opinion before starting the project.

There is no universal "silver bullet" solution for scraping all aggregators — each project requires a combination of geo-proxies, environment emulation, and intelligent request routing. It is this comprehensive approach that we build into the foundation of each implementation so that you receive clean data without blocks.