A scheduled crawl that quietly returns empty or malformed data is worse than one that fails loudly — the first looks fine on a dashboard for weeks. This note covers the retry and backoff logic we built to close that gap.
The problem
Source sites change markup without warning: a class name gets renamed, a page moves behind a redirect, a rate limiter starts responding with a soft-block page that still returns HTTP 200. None of these are outright errors, so naive scrapers keep running and keep writing bad data.
What changed
- Structural checks on every response, not just status codes — a page that 200s but doesn't match the expected shape is treated as a failure
- Exponential backoff per source, so one flaky site doesn't burn the whole crawl's retry budget
- Alerts fire on a drop in extracted-field completeness, not just on hard errors
The goal isn't zero failures — it's making failures visible instead of silent.