September 14
We'd spent weeks waiting on this day. Our residential proxy provider had gone dark with a billing failure back in July, and for two months a growing slice of our 1,375 sources were crawling degraded or not at all — fetches timing out, IPs getting blocked mid-session, the kind of slow bleed that doesn't show up as an outage so much as a shrinking number nobody notices until it's embarrassing. We renewed. New pool, 20,000 rotating residential session lines, probed every exit before letting anything near production. 100% of the probe passed. AE, DE, JP — all live, all clean.
So we ran a supervised crawl, the kind where someone actually watches the dashboard instead of trusting the cron. I expected the opposite of a disaster.
What we got was a database that had lost 84,401 rows in a single pass, with 102 sources down more than 25% from their prior snapshot. Worse than the two months of proxy starvation we'd just fixed.
The wrong theory, briefly
My first instinct — and I'll own this one plainly — was that the new pool was somehow worse than the probe suggested. Maybe the session IPs were getting rate-limited mid-crawl in a way a one-shot probe wouldn't catch. It's a reasonable theory. It's also wrong, and believing it for even half a day cost us time we didn't have, because the crawl cron stays paused the whole time you're debugging a theory like that — you don't want a bad pass overwriting a good snapshot while you're still guessing.
The tell that broke the theory was dubizzle. One of our highest-volume UAE sources, it had actually fetched 32,595 listings during that run — a perfectly healthy number, proxies clearly working — and then discarded almost all of them. Not failed to fetch. Fetched, and then thrown away.
September 16: finding the actual bug
That's when it stopped being a proxy story and became a row-replacement story. Our persistence layer snapshots each source as a full overwrite of <site>_cars_for_sale.csv every pass — current listings only, replace the whole table. That's correct behavior when a crawl finishes. It's catastrophic when a crawl times out partway through and the scraper writes whatever partial batch it managed to collect as if it were the complete snapshot. A source that normally holds 9,000 live listings gets overwritten with the 1,200 it scraped before the clock ran out, and the other 7,800 vanish — not because the cars sold, but because our own code decided an incomplete fetch was a complete one.
The proxies weren't lying to us. The retry logic was the problem, and it had been quietly bad for a long time, just masked by a healthier pool that used to finish fetches before the sweep mattered.
Here's what that sweep actually cost, once we did the arithmetic:
| Old retry sweep | Fixed sweep | |
|---|---|---|
| Retries per proxy | 3 | 1 |
| Delay per attempt | 1s | — |
| Effective ceiling across a 50-proxy pool | ~56 minutes per URL | 120-second total budget |
| Behavior on timeout | writes partial batch as full snapshot | source skipped, prior snapshot kept |
Fifty-six minutes to give up on a single URL. Multiply that across a crawl touching over a thousand sources and you get exactly the kind of silent, unevenly-distributed failure we saw — some sources fine, 102 of them gutted, nothing in the logs screaming "proxy outage" because, technically, there wasn't one.
September 25: the fix, and the part that matters to you
The patch that shipped nine days later did two unglamorous things. It capped the retry math so a stuck fetch fails fast instead of eating most of an hour, and — this is the one that actually prevents the data loss — a partial fetch no longer replaces the full row set. If a source only gets through 1,200 of its usual 9,000 listings, we keep the previous complete snapshot instead of overwriting it with a fraction. Same day, a second change landed that writes a visible CRAWL_PAUSED flag our monitoring checks before it pages anyone about a dip — so a deliberate pause reads as a pause, not an alert.
Why should a reader shopping for a used Hyundai care about any of this? Because the below-market flags we publish — 12,823 of them live on the site as of this week — are only honest if the cohort we're comparing a listing against is the real, complete regional market and not a timeout-truncated fragment of it. A deal badge built against a snapshot that lost three-quarters of its comparison set isn't a deal badge, it's noise wearing a green label. We'd rather pause a crawl for nine days than publish that.
The dataset today sits at 1,108,535 listings across 73 regions and 1,375 sources — bigger than it was before any of this started, proxies renewed and the row-replacement bug closed. But the number I keep coming back to isn't the total. It's 84,401 — the gap between a crawl that looked healthy at the proxy layer and one that actually was.
