Residential proxies are the caviar of scraping infra. They work when nothing else does, and they cost accordingly. The mistake I see over and over is teams routing every single request through residential IPs because a handful of pages needed it. You end up paying steak prices to fetch a robots.txt.
On one client I cut BrightData spend by 90% and ScrapingBee by 67% without touching success rates, and the whole change was structural. Stop treating all traffic as equally sensitive. Tier it, and only escalate when you have to.
The waterfall
The idea is a fall-through ladder. Cheap options first, expensive last, and you only move down a rung when the current one demonstrably fails.
- Tier 0, no proxy. Plenty of endpoints don’t care. Public JSON APIs, sitemaps, image CDNs, anything that isn’t fingerprinting you. If it works direct, that request costs you nothing. On my own stack naked direct requests clear roughly 40% of targets by themselves.
- Tier 0.5, still no proxy, but fix your TLS fingerprint. This is the rung everyone skips. A plain HTTP client has a dead-giveaway TLS/JA3 handshake that Cloudflare and similar defenses match before your request even reaches the app. Swap in a client that mimics a real Chrome fingerprint (cycletls and friends) and a slice of “blocked” sites start returning 200s. Still free, still no proxy. This one change bought me a double-digit percentage-point jump on Cloudflare-fronted pages that direct requests couldn’t touch. Do it before you spend a cent on IPs.
- Tier 1, datacenter / mobile proxies. Cheap, fast, plentiful. Good enough for the large majority of HTML and API traffic that just wants a non-suspicious IP with reasonable rate limiting. Rotating mobile IPs sit around here too and punch above datacenter for not much more.
- Tier 2, residential / rendered. The expensive tier, reserved for flows that actually fingerprint hard: checkout, some search endpoints, sites that clearly block datacenter ranges, anything needing real JS execution.
- Tier 3, managed anti-bot / unlocker service. The nuclear option. Slowest and priciest per request, handles CAPTCHAs, clears ~99%. Only when everything above bounces.
The point of the ladder is the distribution. If the top is free and clears 40%, a fingerprint fix pushes that higher for nothing, and the ~99% unlocker only ever sees the genuinely hard tail, your average cost per page collapses even though your worst-case tool is still expensive.
The two things that make this work and not just be a fancy retry loop: knowing when a tier failed, and remembering what worked.
Failure is not just a status code
A 403 is easy. The dangerous case is a 200 that’s actually a soft block. You get back a page, it has the right content-type, your parser even runs, and it’s a captcha interstitial or a stripped-down decoy. If you only check the HTTP status, you’ll happily record garbage and never escalate.
So validation is content-level, not transport-level:
def is_good(resp, expect):
if resp.status_code != 200:
return False
body = resp.text
if len(body) < expect.get("min_bytes", 500):
return False
if any(m in body for m in BLOCK_MARKERS): # captcha/challenge strings
return False
# the actual proof: did the data we came for show up?
return expect["must_contain"] in body
def fetch(url, expect):
for tier in TIERS: # cheapest first
resp = tier.get(url)
if is_good(resp, expect):
remember(url_pattern(url), tier) # cache the winner
return resp, tier
raise AllTiersFailed(url)
Enter fullscreen mode Exit fullscreen mode
must_contain is the honest check. A price marker, a known JSON key, a product-id pattern. If the thing you came for isn’t in the body, you didn’t get the page, whatever the status line claimed.
Remember what worked
The other half of the savings is remember. Once you learn that a given domain or URL shape needs Tier 2, cache that decision with a TTL and start future requests to that pattern at the right rung instead of walking the whole ladder every time. Walking the ladder on every request means you pay the datacenter attempt and the residential attempt for pages you already know need residential. Cache it and you pay for the tier that works, most of the time, and re-probe occasionally in case the site relaxed.
Flip side: expire those decisions. Sites change defenses. A domain that needed residential in March might be fine on datacenter by June, and a stale cache keeps you overpaying. A day or two TTL was the sweet spot for me.
Where the money actually went, once I measured it: the vast majority of requests were fine on Tier 0 or Tier 1, and a small slice of genuinely sensitive flows justified the residential spend. Before the waterfall, that small slice was setting the price for everything. That’s the whole trick. Not a cheaper proxy vendor, a cheaper distribution of traffic across the vendors you already have.
One caveat worth stating: this adds moving parts. A tier ladder, a validation layer, a decision cache with expiry. If you’re scraping a few thousand pages a month off one friendly site, skip all of it and use one proxy. The waterfall earns its complexity at volume, when a 90% cut is real money.
I do cloud and infra cost work under reducecost.cloud, where this kind of “same result, structured cheaper” thinking shows up constantly, and the scraping side of it feeds cartpie.com, an e-commerce product-data scraping platform with a free tier live now.
답글 남기기