Data collection at scale, engineered to keep working

The frontier is the heart of it: a queue of URLs with priority, next-fetch time and retry state. Priority is driven by business value and volatility — a competitor's price on a fast-moving SKU is refetched hourly; a company profile page monthly. This is what allows a system to scale to millions of pages without simply crawling everything constantly.

Workers pull from the frontier and fetch. For static content that is a plain HTTP request, which is an order of magnitude cheaper than a browser and should always be preferred where it works. For JavaScript-rendered pages we use headless Chromium through Playwright, in a pool sized to the target's tolerance rather than our capacity.

Parsing is deliberately strict. Every field has an expected type and a validation rule, and a page that yields a null where a value is expected raises a parse failure rather than storing a blank. This is the single most important design choice, because the failure mode that destroys a dataset is not the scraper crashing — it is the scraper quietly returning empty results for three weeks after a layout change.

Storage keeps the raw HTML alongside the parsed record. When a parser is later found to be wrong, or a new field becomes interesting, history can be reprocessed rather than recollected — which is both faster and far more polite to the source.

A collection system that hammers a target will be blocked, and it deserves to be. We configure per-domain concurrency and delay based on the target's size and observed response behaviour, back off automatically when latency rises or errors increase, honour robots directives and crawl-delay, and identify ourselves honestly in the user agent with a contact address where the client is comfortable with that.

Proxies are used to distribute load and to access geographically varied content — a marketplace shows different pricing and availability in different regions, and collecting that legitimately requires exit points in those regions. We manage residential, datacentre and mobile pools with health monitoring and automatic rotation away from degraded exits.

What we do not do is treat proxy rotation as a way to evade a target's explicit refusal. If a site has clearly signalled that it does not want automated access, the correct response is to stop and, where a commercial relationship is plausible, to ask about an API or a data licence. Several of our clients now receive data by agreement rather than collection, which is cheaper and more reliable for everyone.

Every collection system breaks, because websites change. The question is only how quickly you find out. We monitor extraction health continuously: parse success rate per source, field-level fill rate, distribution shift in values, page-size anomalies and response-code patterns. A source whose price field fill rate drops from 99% to 40% triggers an alert within the hour, long before anyone notices a gap in a report.

Repairs are usually small — a selector change, a new page variant, an added consent overlay. Our median time from alert to fix is under two hours during working hours, and our maintenance agreements commit to it. Across a large estate we typically see a given source require attention two to four times a year.

We also version parsers, so a repair can be applied retroactively to raw HTML captured during the broken window. In most cases a breakage produces no permanent gap in the dataset at all.

Competitive pricing and assortment monitoring across marketplaces and brand sites, feeding a pricing team or an automated repricing rule. This is the most common request and the one with the clearest return.

Tender and notice monitoring across government and PSU portals — a niche with enormous value for engineering and construction firms, where missing a notice means missing a quarter. We monitor dozens of portals and deliver structured, deduplicated notices with attachments the same morning.

Commodity and input price collection for manufacturers whose costing depends on volatile inputs, joined into the same warehouse as production data so margin impact is visible immediately.

Regulatory and filing monitoring — company filings, licence registers, compliance databases — for financial services and diligence use.

And browser automation of internal processes: downloading bank statements for reconciliation, filing returns on portals with no API, checking status across supplier systems, and any workflow where the browser is the only interface available.

We are not lawyers and we do not give legal advice, but we have had this conversation many times and can tell you where the questions usually land. Publicly accessible factual data collected without circumventing access controls, at a rate that does not burden the target, is the least contentious position. Personal data brings India's Digital Personal Data Protection Act into scope regardless of public availability. Content that is copyrightable — articles, images, substantial text — raises different questions from facts like prices and availability. And a site's terms of use may create contractual obligations, particularly where access required accepting them.

What we do operationally: document the basis for each source, keep collection within stated limits, avoid personal data unless the client has established a lawful basis, retain raw evidence of what was collected and when, and stop immediately on a request from a target. Where a source looks contentious we say so before building rather than after.

Collected data is only useful where your people already work. We deliver into your warehouse as modelled tables, into an API your applications call, as scheduled files to S3 or SFTP, into a dashboard, or as an alert on email, Slack or WhatsApp when a threshold is crossed — usually several of these at once.

Because our data engineering and scraping practices are the same team, collected data lands in the same lakehouse as your ERP and plant data, with the same quality gates and lineage. That is where the compounding value is: a competitor price series is interesting; a competitor price series joined to your own cost per tonne and margin by SKU is a pricing decision.