How to Write a Technical Specification for Parser Development: Checklist, Timelines, Budget, and KPIs

Why a Proper Technical Specification for a Parser Matters

Data parsing is a technically complex task, and without a clear technical specification, it almost inevitably turns into a series of reworks, budget overruns, and missed deadlines. A weak specification is the main reason why a finished parser becomes unusable just a month after delivery. A good specification, on the other hand, removes uncertainty, provides a common understanding of goals, and becomes a legally significant document for acceptance. Below, we have compiled a checklist to help you create a technical specification for parser development, as well as how to estimate timelines, budget, and measurable KPIs.

What Should Be in the Specification: A Checklist for the Client

The technical specification for a parser should cover seven mandatory sections. Even if you are ordering a service for parsing websites and marketplaces from a studio, this checklist will help you correctly describe the project and compare offers.

Target Data Sources

• Full URLs of sites or sections from which data will be collected.
• Access type: open pages, personal account, API.
• Need for authorization and method of execution (login/password, cookies, OAuth).
• Expected content dynamics: static pages, infinite scroll, AJAX loading.

Structure of Collected Data

• List of fields for each page type (product, category, article).
• Data types: text, number, date, image, link, nested object.
• Cleaning rules: remove HTML tags, normalize case, replace special characters.
• Handling multilingual, multi-currency, and regional pricing.

Update Frequency and Volumes

• How often data needs to be updated: once a day, once an hour, in real time.
• Collection depth: only new items or a full recrawl each time.
• Maximum number of pages/records per cycle.
• Peak loads (e.g., seasonal sales).

Output Formats and Delivery

• Where to output: CSV, JSON, XML, Excel, direct write to database (PostgreSQL, ClickHouse).
• Transfer method: FTP, cloud storage, API, email notification.
• File naming structure and export frequency.

Anti-Blocking Measures

• Use of proxies (residential, datacenter) and their rotation.
• Browser emulation: headless Chrome, Puppeteer, Playwright.
• Request rate limiting, random delays.
• CAPTCHA handling (manual or through recognition services).

Infrastructure and Environment

• Where the parser will run: on the client's server, in the cloud, on VPS.
• OS, runtime environment, resource requirements (CPU, RAM, disk).
• Need for monitoring and alerts.

Error Handling and Logging

• What to do when source is unavailable: retry N times, send notification.
• Log format and retention period.
• Recording incomplete or incorrect entries.

How to Define KPIs for the Parser

Without numerical metrics, it is difficult to evaluate the effectiveness of invested funds. We recommend including the following minimum set of KPIs in the specification:

• **Percentage of successfully collected pages** — the ratio of fully processed URLs to the total planned number. Lower bound: 95–99% depending on anti-blocking conditions.
• **Collection speed** — number of pages per minute/hour. Allows forecasting the full cycle time.
• **Data integrity** — the proportion of records where all required fields are correctly filled. Acceptable value ≥ 98%.
• **Failure response time** — maximum time from source failure to automatic restart or alert.
• **Monitoring false positive rate** — no more than once per week.

These metrics then become part of the acceptance criteria and allow for objective evaluation of the work. If you plan to integrate the parser into a broader ecosystem, such as a CRM system or SaaS platform, add KPIs for integration time and API stability.

Timelines: What They Depend On and How to Calculate

Actual parser development timelines are determined by three groups of factors:

• Source complexity. A static HTML site can be parsed in a few days. Dynamic SPAs with WebSocket, complex authentication, and browser emulation may require 3–6 weeks. Add time for selecting and testing proxies if the site is heavily protected.
• Volume and frequency. A one-time collection of 10,000 items takes one week. Streaming collection from dozens of sources with updates every 15 minutes takes 2–3 months, including load testing and optimization.
• Post-processing and integration. If the data needs not only to be collected but also cleaned, enriched, categorized, and delivered to a BI system, add 20–30% to the baseline estimate. Timelines also depend on infrastructure readiness; cloud deployment, which we handle as part of cloud development services, speeds up launch.

For reference: a typical online store parser with 5–10 fields, proxy rotation, and CSV export with weekly updates takes about 3 to 6 weeks (example). Complex monitoring systems can take up to 4 months (example). We recommend building a 15–20% buffer into your specification for unexpected changes in the source’s markup.

Budget: what affects the cost

The final budget consists of three components:

1. Parser core development. Depends on the number of sources, crawl depth, and anti-blocking requirements. One simple source — from 80,000 RUB (example). Adding each subsequent similar source can be 30–50% cheaper due to code reuse.
2. Infrastructure costs. Proxy pools, servers, cloud resources, CAPTCHA API fees. These can range from 10,000 to 100,000 RUB per month (example) depending on volume. We help select the optimal configuration when ordering cloud services.
3. Maintenance and support. Source websites change, so you should budget for regular upkeep — typically 15–25% of the development cost per year. For critical projects, we recommend hosting the parser on the client’s infrastructure using web development services for the admin panel and monitoring.

To stay within budget, clearly define the scope in the specification as either Time & Material or a fixed price with a precise list of user stories. Any expansion of sources or fields after approval must be handled as a separate request.

Acceptance criteria and testing

Parser acceptance should rely on predefined acceptance criteria that translate KPIs into verifiable test cases. Add a section like this to the specification:

• Test data set: 100 to 500 pages manually verified by the client. The parser must collect at least 99% of mandatory fields with accuracy matching the reference.
• Stress test: Successful run at the maximum stated volume (e.g., 100,000 items) with speed degradation no greater than 20% and no record loss.
• Anti-blocking resilience: A 24-hour run with blocked request rate no higher than 3%.
• Output format compliance: Automatic validation of CSV/JSON structure against the defined schema.
• Documentation: Delivery of deployment instructions, configuration file descriptions, and operational procedures.

Testing should be performed on a staging environment identical to production. If the parser later becomes part of a larger product, acceptance criteria should account for interaction scenarios, for example via APIs designed as part of SaaS applications development.

Frequently asked questions

Is a specification necessary if the parser is very simple?

Yes. Even for collecting a few fields from a single page, a specification eliminates ambiguity: what exactly counts as a "price," how to handle discounts, whether currency should be considered. Spending an hour documenting requirements saves days of back-and-forth and rework.

How long does it take to write a good specification?

On average, 2 to 6 hours of work from the client. If you use the checklist above and provide sample data, the time is reduced. A studio can help formalize the requirements into a document as part of a pre-project investigation.

Can requirements be changed during development?

Yes, but this should be governed by a change scope procedure (Change Request). Minor adjustments (adding one field) often fit into an agile process. A radical change of sources will require a revision of deadlines and budget.

What guarantees are there that the parser won't break when the source site changes?

No parser is immune to a complete layout change or the introduction of new protection. However, risks can be reduced: use semantic selectors instead of rigid XPath; set up alerts for a drop in the percentage of successful data collections; allocate hours for adaptation in the maintenance contract. This is exactly how we build long-term projects as part of the website and marketplace parsing service.