Legal Aspects of Web Scraping in Russia and the EU: robots.txt, ToS, and Personal Data

Web scraping is a powerful tool for price monitoring, competitor analysis, database population, and business process automation. But alongside its obvious benefits, collecting data from public sources carries legal risks, especially when used commercially. Ignorance of legal norms can lead to IP blocks, claims from website owners, and even administrative fines. In this article, we will thoroughly examine how web scraping is regulated in Russia and the European Union, what "lawful" data collection means, and how to build a safe process that respects robots.txt, terms of service (ToS), and personal data processing requirements.

What is robots.txt and why does it matter?

The robots.txt file is a widely used exclusion protocol for search engine crawlers, placed in the root of a website. It provides instructions for search engines and other automated agents about which sections may be indexed and which may not. Although robots.txt itself is not a legally binding document, ignoring it is often treated by courts as a violation of the site's usage rules. This applies primarily in common law countries, but in Russia as well, an explicit prohibition in robots.txt can be interpreted as circumvention of technical information protection measures.

When planning scraping, you should always check the target site's robots.txt. If it says Disallow: / for all robots or specifically for your parser, then automated data collection will almost certainly lead to a conflict. Moreover, some sites explicitly state in robots.txt that scraping for commercial purposes is forbidden. Even if the site owner has not gone to court, ignoring robots.txt can serve as grounds for a pre-trial claim and subsequent lawsuit under articles concerning unlawful access to computer information (e.g., Article 272 of the Russian Criminal Code) or violation of website usage rules.

Experience shows that proper handling of robots.txt is one of the key criteria of developer integrity. When building custom parsers, our specialists always integrate a robots.txt compliance module and configure request frequency so as not to place excessive load on the server. To learn more about how we build safe solutions for price collection, visit the page price parsing for marketplaces and retail.

Terms of Service (ToS) and explicit bans on scraping

Terms of Service (ToS) or user agreement is the primary document in which the site owner defines acceptable ways to interact with the resource. Typically, ToS contains an explicit ban on automated data collection, content copying, and creation of derivative works without written permission. From a legal standpoint, such an agreement can be considered a contract of adhesion. By visiting the site and using its services, the user automatically agrees to the ToS. Therefore, scraping in violation of an explicit ban may be classified as breach of contract.

In Russia, case law on disputes involving ToS violations through parsing is still less extensive than in the US, but the trend is already noticeable. Site owners often cite Article 1252 of the Russian Civil Code (infringement of exclusive rights to content) and Article 10 of the Federal Law "On Information." In the European Union, however, key precedents are based on Directive 96/9/EC on the legal protection of databases, which protects non-trivial databases from extraction and reuse without the rights holder's permission. Courts assess whether the database was created with substantial investment, and if so, scraping may be prohibited even in the absence of technical restrictions.

To minimize risks, you must analyze the ToS before starting data collection. If the ban is explicit, do not rely solely on the fact that the content is technically accessible. An alternative may be to enter into a direct agreement with the resource owner or use public APIs if provided. Custom development allows you to embed ToS and robots.txt processing into the scraper's logic as early as the design stage. For example, when building a comprehensive web solution, we combine parsing with legal compliance—learn more about this on turnkey web application development.

Personal data: GDPR and Federal Law No. 152-FZ

One of the most sensitive areas is the inclusion of personal data in scraping results. Both Russia and the EU have strict laws regulating its processing. Even if the data is publicly available, its automated collection, storage, and use may require legal grounds. In the European Union, such a basis could be a legitimate interest, but it must be carefully balanced with the rights of data subjects, and in Russia the list of grounds is even more limited.

GDPR: Risk of fines up to 20 million euros

The General Data Protection Regulation (GDPR) has been in effect in the EU since 2018 and applies to any organization processing data of EU citizens, regardless of its location. If scraping captures names, email addresses, phone numbers, geolocation, or any other identifiers that can directly or indirectly point to a person, it constitutes processing of personal data under GDPR. In such case, the company must have legal grounds (consent, necessity for contract performance, legitimate interest), comply with principles of data minimization, transparency, and the right to erasure. Fines for violations can reach 20 million euros or 4% of annual global turnover.

Scraping data from social networks, forums, or directories containing personal information without proper justification almost certainly violates GDPR. Therefore, when designing parsers, we recommend immediately excluding fields that may contain personal data from the output, or immediately anonymizing them in memory. Our experience in cloud services development shows how important it is to design secure data storage — all temporary files and logs that may contain personal data are processed in accordance with GDPR, using isolated environments and encryption.

Federal Law No. 152-FZ: Requirements of Russian Legislation

In Russia, the main act is Federal Law No. 152-FZ "On Personal Data." It obliges the operator to obtain the subject's consent for processing their personal data, except when processing is necessary for contract performance or achieving socially significant goals. Collecting data from public sources does not in itself relieve the obligation to comply with 152-FZ. Moreover, since 2021, amendments have been in effect that tighten requirements for consent and database localization. If during scraping you collect personal data of Russian citizens, the database must be physically located on servers in Russia.

In practice, this means that even monitoring product prices can carry a risk if product descriptions contain data about individual sellers. To eliminate risks, our specialists implement automatic filtering of personal data right at the content collection stage. And for corporate solutions with huge volumes, we offer an architecture with local storage in a protected environment — such projects we implement as part of SaaS application development for data processing and analytics.

Best Practices for Legal Web Scraping

To minimize legal risks, it is necessary to build a process where technological implementation goes hand in hand with compliance. Below are recommendations based on analysis of Russian and European regulatory frameworks, as well as many years of experience in custom parser development.

• Check robots.txt and ToS. Configure an analyzer that, before starting data collection, reads the robots.txt file and checks for a direct prohibition of scraping in the terms of service. If a Disallow is found for the required section — either refrain from collecting or seek a legal compromise.
• Use official APIs. Many sites provide APIs for data access. This is the safest route because the resource owner has already given explicit permission for such interaction and set limits.
• Minimize the volume of collected data. Do not pull more than needed for a specific business task. Avoid full duplication of databases; collect only slices and aggregates. This reduces risks regarding copyright and database regulations.
• Filter personal data. Implement personal data detectors at an early stage of the pipeline. Everything that resembles email, phone, passport data, or names of individuals should be removed before saving to the database or masked.
• Follow rate limiting. Excessively frequent requests not only put a load on the server but can also be perceived as an attack (DoS). This adds the risk of administrative and even criminal liability. Set delays of at least a few seconds between requests and respect the Retry-After header.
• Document the legal basis. Document the basis on which data collection is carried out. For example, if scraping of publicly available information is done under legitimate interest (Legitimate Interest Assessment — LIA for GDPR), prepare a corresponding Data Protection Impact Assessment (DPIA).
• Involve a lawyer in the project. Each case is unique. No general recommendations can replace consultation with an IT law specialist familiar with the legislation of specific jurisdictions.

At ESK Solutions, we believe that legal security is an integral part of the technical specification. Our parsers are built from scratch, tailored to the client's individual requirements, including full compliance with robots.txt, PII filtering, and built-in rate-limiting mechanisms. If your business needs a reliable and legal tool for data collection, we recommend looking into the service price parsing for marketplaces — we will help build the entire cycle from idea to implementation.

Frequently Asked Questions

Can robots.txt be completely ignored if the data is technically accessible?
No. Although the legal force of robots.txt varies from country to country, ignoring it is easily proven and serves as a strong argument against you in court. This is especially dangerous in commercial scraping, as it adds a violation of the website's terms of use to the existing risks.

Are open data from the internet considered personal?
Yes, if they allow direct or indirect identification of a natural person. First and last name, social media profile, photo, email — all of these are personal data. The fact that they were published by the user themselves or became publicly available does not eliminate the need to comply with legal requirements, including obtaining consent for processing (with rare exceptions).

Is it possible to include direct permission for scraping for specific purposes in the ToS?
Legally, this is possible, and some website owners deliberately grant such permission by specifying whitelisted bots or rules in the user agreement. If you plan to scrape your own site or a partner's site, it is worth explicitly stating the permission — this will remove risks for both parties.

What to do if the site is not protected but I fear legal consequences?
The best solution is to order development from professionals who will immediately incorporate mechanisms for respecting robots.txt, filtering PII, and configurable request frequency. At ESK Solutions, we support the project from the legal analysis stage, helping to choose a minimally risky strategy for data collection.