Observability for Parsers: Success Metrics, Alerts, and Data Collection SLA
Setting up observability for industrial parsers: key success metrics, automatic alerts, and data collection SLA using the Prometheus stack.Why Parsing Without Observability Is a Business Risk
Modern data collection systems from web sources have evolved from throwaway scripts into continuous pipelines that drive pricing, competitive analytics, and supply chains. A failure in an unnoticed parser can distort catalog prices, and losing data from marketplace monitoring can lead to missed revenue. This is where observability takes center stage—the ability to understand what's happening inside the system without looking at the code.
Parsing is a distributed and finicky process: sources change their layouts, networks go down, proxies get blocked, servers throw captchas. Without metrics, logs, and configured alerts, engineers only learn about problems from business complaints. At ESK Solutions, when developing industrial parsers, we build observability into the architecture from the start, using a stack of Prometheus, centralized logging, and Grafana. This approach enables not only rapid response but also guarantees a data collection SLA.
Key Parsing Success Metrics
To manage data collection, you need to measure not just "works/doesn't work" but also quality, speed, and stability. Metrics are typically divided into SLIs (Service Level Indicators), which later serve as the foundation for SLA targets. For parsing, we identify several mandatory groups.
Success Rate
The most straightforward indicator. A successful request is one that returns an HTTP response in the expected range (usually 2xx or 3xx for redirects) and extracts the target data from the response body. We measure at each step: page retrieval, processing, and storage. Metric providers include the proxy layer, the parsing engine, and the storage write layer.
- Prometheus metric:
parser_requests_total{status="success|fail"} - Typical SLA target: ≥98% under average load.
Data Completeness and Integrity
Even a successful HTTP response doesn't guarantee we've collected all required fields: the price might be missing, the description truncated, or a numeric value might come in the wrong format. For each source type, we define a reference structure and calculate the percentage of records that pass validation.
- Metric:
parser_data_completeness_ratio— the fraction of fields retrieved relative to the expected model. - Counter for rejected records:
parser_invalid_records_total.
Latency and Throughput
Businesses often care how quickly data becomes available after it's published on the source. We measure the time from task receipt to valid record storage, as well as overall throughput (records per second).
- Latency histogram:
parser_task_duration_secondswith quantiles p50, p95, p99. - Instantaneous speed:
parser_records_per_second(gauge).
Example alert rule: if p95 latency exceeds 10 seconds for 5 minutes, the system notifies the team.
Errors and Specific Exceptions
We track errors by type separately: timeouts, blocks (CAPTCHA/403), page structure changes (selector not found), proxy limit exceeded. Each type is its own time series, allowing us to spot trends before they become critical.
parser_errors_total{type="captcha|timeout|structure|proxy_limit"}
Alerts and Incident Escalation
Metrics without alerts are analytics, not operational readiness. An alerting system turns parsing into a service with predictable failure response.
Typical Triggers
We recommend combining threshold-based and dynamic alerts. Threshold alerts fire when success rate drops below a set level, the task queue grows, or the error rate becomes anomalous. Dynamic alerts, such as those based on deviation from a moving average, help catch sudden spikes that don't violate a hard threshold but signal a developing issue.
Example triggers:
- Success rate below 95% over a 5-minute window.
- Number of pending tasks > 1000 for 10 minutes.
- CAPTCHA error share exceeds 20% of all requests.
Configuring Alerts with Prometheus and Alertmanager
Alerting rules are described in Prometheus Rule Files, while routing, grouping, and notifications are managed by Alertmanager. We configure mandatory channels: corporate messenger (Slack/Telegram), email for non-critical warnings, and integration with an on-call system (PagerDuty/Opsgenie) for failures that threaten SLA.
It's important to set up "silence periods" and deduplication so that the same problem doesn't turn into a notification storm. All alerts are automatically logged for post-mortem analysis.
SLA and Commitments to the Customer
When data collection is a service, SLA parameters are specified in the contract. Typical SLA metrics include: availability (percentage of time the pipeline successfully processes tasks), data timeliness (maximum delay from publication), and completeness (share of covered positions). Based on SLIs and alerts, we build dashboards that show real-time SLA compliance, and monthly reports are generated automatically.
Logs and Tracing for Fast Diagnostics
Metrics tell you there is a problem; logs tell you where and why. In distributed parsing without a unified log, an engineer spends hours correlating events.
Structured Logs
All parser components write logs in JSON format, which is easy for systems like Loki or Elasticsearch to parse. Each entry contains a trace ID, task ID, severity level, and context (source URL, proxy, parser version). This allows quick filtering by a specific task or error type.
Centralized Collection and Storage
Logs from all pods, workers, and scrapers flow into a central repository — typically Grafana Loki integrated with Grafana, or Elasticsearch. This approach lets you correlate Prometheus metrics and logs in one interface and build complex queries like "show all errors in the last hour for source X."
Linking Metrics and Logs Through Tracing
By adding a trace ID to each task, we can trace the entire path from a browser request to record storage. Prometheus collects durations and error counts, while logs contain details of each step. When an alert fires, the on-call engineer clicks on the chart in Grafana and jumps to the list of related logs — reducing response time from tens of minutes to a few seconds.
Dashboards as a Single Control Center
A good dashboard is not just a collection of charts but a decision-making tool. For parsing, we develop several visualization layers accessible to all stakeholders: engineers, analysts, and product owners.
Real-Time Visualization of Prometheus in Grafana
The main "control panel" is built in Grafana with panels for all metric groups: overall success statistics, latencies, queues, errors by type, worker load. A separate section covers business metrics: number of products in the catalog updated in the last 24 hours, missed items. All panels have preconfigured drill-down links to logs and traces.
SLA Dashboards for Customers and Management
Based on the same metrics but aggregated by week/month, SLA compliance reports are created. These are often embedded in a client portal or sent automatically. This builds trust and enables rapid adjustments to parser settings.
If you need a full-featured web interface for managing parsers with personalized dashboards, the ESK Solutions team will develop it to your requirements.
Infrastructure and Deployment
Observability doesn't exist in a vacuum — it must be deployed, scaled, and maintained. We use containerization and Kubernetes, deploying the monitoring stack as a standard add-on. Prometheus and Grafana are installed via Helm charts, alert configurations are versioned in Git, and the cloud infrastructure allows flexible resource allocation for peak loads. For customers needing an isolated environment, we provide a SaaS monitoring platform with a dedicated tenant and access control.
Conclusion
Observability transforms parsing from a black box into a transparent and manageable service. Prometheus metrics provide an objective picture of success, alerts reduce reaction time, logs provide context for investigation, and dashboards make SLA fulfillment measurable and provable. Implementing these practices at the start of a project costs tens of times less than eliminating the reputational and financial consequences of poor-quality data collection. If you are planning to build or improve a data collection system with guaranteed quality, the experts at ESK Solutions will help design, implement, and maintain observable parsing, relying on best engineering practices.Frequently Asked Questions
What minimal metrics should be implemented in a parser?
The minimal set: a counter of successful and failed tasks, a histogram of processing duration, and a counter of error types (timeouts, blocks, validation errors). This is enough for basic alerts and initial SLA. As the system grows, metrics are added for each source, data integrity, and proxy load.
How to link Prometheus and logs for quick diagnostics?
The simplest way is to add a unique task identifier (trace ID) to the logs and expose it in the metric labels. Then in Grafana you can configure a data link that goes from any Prometheus chart to Loki/Elasticsearch with a pre-filled query for that identifier. You can also use Grafana Tempo for distributed tracing if the parser is split into microservices.
Should different alerts be configured for different sources?
Yes, because sources behave differently: one can tolerate a 10-second delay, for another 3 seconds is already critical, some have frequent CAPTCHAs, while others rarely encounter them. We recommend setting up alerts based on the 'source profile': acceptable error rate, delay threshold, etc. This prevents false positives and focuses the team on real problems.
Can SLA for data collection be guaranteed when using free proxies?
Free or public proxies cannot provide stable success rates and low latency, so it is impossible to guarantee SLA based on them. For industrial-grade parsing, a combination of residential, data center, and mobile proxies with automatic rotation is required, as well as channel redundancy. Only then can you achieve the 99% success rate levels specified in contracts.