XML and JSON Supplier Feeds vs HTML Parsing: When Is One More Beneficial?

Introduction: Two Paths to Supplier Data

For online stores, aggregators, and price comparison platforms, regularly loading product listings is a critical process. There are two fundamentally different ways to obtain structured data from suppliers: using ready-made feeds (XML, JSON, YML, CommerceML) or applying HTML parsing (scraping) of catalog pages. The choice affects update stability, development and maintenance costs, as well as legal risks. Below, we examine when relying on contractual feeds is justified, and when parsing websites and marketplaces is unavoidable.

Stability of Contractual Feeds: YML, CommerceML, and API

The main advantage of structured feeds is their predictability. A supplier providing YML (Yandex Market Language) or CommerceML typically fixes the format in a contract or technical specification. This means:

• A clearly described data schema that rarely changes without notice.
• No need to "guess" the page structure and fight with layout.
• Guaranteed scheduled delivery: file via FTP/HTTP or API endpoint.
• Built-in validation — you can automatically catch inconsistent records before they reach the storefront.

CommerceML is especially valued in conjunction with 1C: it describes not only products and prices but also group hierarchy, properties, and units of measurement. Integrating CommerceML via an exchange module reduces export time and minimizes manual labor. YML, in turn, remains the de facto standard for price aggregators and is well supported by most CMS.

JSON feeds (often via REST API) add flexibility: you can get incremental updates, filter the selection, and integrate with modern microservice architectures. The investment in developing a connector for such an API pays off through zero operational costs for parsing in the future.

HTML Parsing: When It's Unavoidable

Scraping remains a forced solution when suppliers do not provide feeds or their format is chaotic. Typical situations:

• Small regional suppliers with a basic website where the only data source is the public catalog.
• Marketplaces without an open API that require monitoring competitor prices.
• Dynamic pages with JavaScript rendering where data is loaded asynchronously.
• Irregular sources for which building a full gateway is economically unfeasible.

The main risk is instability. A change in a single CSS selector or DOM structure completely breaks data collection. Therefore, industrial parsing requires cloud infrastructure with proxy farms, browser farms, retry mechanisms, and alerting. Without this, IP blocks and CAPTCHAs will quickly make the process unprofitable.

Nevertheless, modern scraping can be reliable if designed as a fault-tolerant pipeline. In our practice, we use containerized collectors, headless browsers, and message queues, allowing us to serve hundreds of sources. More details on the approach can be found in the comprehensive parsing and monitoring service.

Cost and ROI Comparison

The financial aspect often outweighs technical arguments. Let's provide a rough calculation for a store with 50 suppliers (all figures are simplified examples for illustration).

Option A: Feeds (YML/JSON/CommerceML)

• Connector development: 20–40 person-days per typical module × 20 suppliers with feeds. The remaining 30 are without feeds.
• Maintenance: 2–4 hours per month for adapting to schema changes (rarely).
• Total cost of ownership over 3 years: ~60–80% of one-time development costs plus minimal support.

Option B: Total HTML Parsing

• Parser development: 10–25 days per source due to the need to bypass protection, configure browsers, proxies. For 50 suppliers, many times more.
• Ongoing support: 5–15 hours per month per source to fix broken parsers, update proxies, deal with CAPTCHAs.
• Infrastructure: renting servers, proxy pools, CAPTCHA solvers.
• Total cost over 3 years: comparable or 1.5–2 times higher than Option A, especially considering maintenance effort.

Conclusion: if at least 30–40% of suppliers agree to provide a feed, it's more profitable to invest in automated turnkey connector development, and for the remaining ones use managed parsing as a temporary measure. Gradually transitioning suppliers to feeds provides maximum contract stability.

Recommendations for choosing a strategy

The optimal route for a growing e-commerce project looks like this:

1. Audit current suppliers: identify those who already have YML, CommerceML, or API. Assess the costs of integrating with them.
2. Develop a universal feed intake gateway capable of parsing XML/JSON, validating, and loading into your database. This can be part of a SaaS solution for internal automation.
3. For suppliers without feeds, implement pilot parsing with stability monitoring. If the supply is critical, include a contract requirement for providing a structured catalog in the future.
4. As you scale, move parsing to cloud infrastructure to avoid performance degradation of the main site, and engage a professional team for maintenance.

HTML parsing should not be considered as the primary method if there is an opportunity to negotiate a feed. Contract stability directly affects the store's reputation: customers see current prices and availability, and search engines do not detect discrepancies.

Frequently Asked Questions

Can you completely abandon parsing if all suppliers provide feeds?

Theoretically, yes. But in practice, there will always be sources for which a feed is not provided—for example, one-time purchases from non-core counterparties. In such cases, it's convenient to keep a lightweight parser as a backup channel. The main data flow goes through feeds, and the parser covers exceptions.

What to do if a supplier changes the YML structure without notice?

First, include a clause in the contract requiring notification N days before changes. Second, build the intake module to check the XSD schema (or JSON Schema) and send an alert to the manager if there is a mismatch. Then even a sudden change won't break the storefront—the data simply won't enter the catalog until manual verification.

How long does a typical CommerceML integration with 1C take?

With a ready-made exchange module in the CMS, it takes from a few hours to a couple of days to set up field mapping. If customization of non-standard properties is required, it may take up to 5–7 business days. We perform such work as part of our CRM and inventory system integration service.

How to protect parsing from blocks?

Use residential proxies, User-Agent rotation, emulation of real user behavior, distributed sessions via headless browsers, and captcha services. It's better to entrust the setup of such a pipeline to experts in industrial parsing—it's cheaper than maintaining your own infrastructure.