I’ve been “the computer guy” my whole life. The one you call when the Wi-Fi is haunted. But I never wrote code until this year, when AI assistants closed the gap and I built a movie-discovery site (flickomatic.com).
Current user count: me and my wife. So imagine my delight when the weekly hosting bill hit $82. And climbing. Here’s what I found when I went digging. Four culprits, none of which I’d have guessed.
An impossible number
The biggest line item: “ISR Writes: 8.69M ($34.75).” ISR is the scheme where a page renders once and gets served from cache for a week. My site has about 2,000 pages. There’s no arithmetic where that becomes 8.69 million.
Except there is, one sentence deep in the pricing docs: writes are billed in 8 KB chunks. That “8.69M writes” was really 70 GB of payload shoveled into the cache. New question: shoveled by whom?
1. Bots were inventing infinite versions of my pages
My URLs look like /movie/603/the-matrix, but it turns out the title part was decorative. /movie/603/absolutely-anything rendered the same page, and each variant got cached as a brand-new entry at ~13 billed writes a pop. No human would ever notice. Crawlers, which hoard URLs in every variant they’ve ever seen, noticed constantly.
I now know the word “canonicalization”: wrong URL gets a permanent redirect to the one true URL. One cached page per movie. Novel concept.
2. I was archiving movie posters by accident
My link-preview cards fetched each movie poster with a “cached fetch,” which sounds responsible. What it actually does is save the entire JPEG into the billed cache. About 19 writes per poster, thousands of movies, read back approximately never. A very expensive museum nobody visits. One changed line killed a third of the problem.
3. My site’s biggest customer was my site
The traffic dashboard: homepage 58.7K requests/day, internal API 58.1K. Suspiciously identical. Top visitor identity: something called “node.” At, you guessed it, 58.1K.
“node” is my own server. The homepage loaded its movie list by making a full HTTP request to its own API, once per visit, cached under a key that included every filter combination. My site was DDoSing itself. Politely. On a schedule. And I was paying for both directions.
If your function invocations outnumber your page views, go find yourself in your own logs before blaming outsiders.
4. My domain has a past life
Heavy bot traffic kept hitting /blog/1, /blog/2, pages I never built. Bing’s dashboard explained it: my domain was first “discovered” in January 2010. I bought it this year. Someone ran a blog here fifteen years ago and crawlers are still knocking on that dead door. They now get HTTP 410, “Gone,” a status code I didn’t know existed and now love.
The bot zoo
“Bot traffic” turned out to be four different animals:
- Search and AI-assistant crawlers (Googlebot, Bingbot, the crawlers behind Claude and ChatGPT search): how anyone might ever FIND my site. Welcome guests. The goal was making their visits cheap, never blocking them.
- AI training scrapers: bulk-download everything, send nothing back. Asked to leave via robots.txt.
- SEO-tool crawlers: AhrefsBot alone was a THIRD of my traffic. It indexes your site to sell reports to marketers. Asked to leave; complied within hours. Most courteous freeloader on the internet.
- A scraper farm on Alibaba Cloud, 51K requests/day, claiming to be Chrome on Windows and also Chrome on a Mac. Pick a lane. Every request shared one identical TLS fingerprint (a “JA4”). Real devices differ at that layer; 51K identical handshakes is one program in costumes. The firewall can challenge by fingerprint: humans pass, scripts fail. Gone in 60 seconds. Most satisfying minute of the month.
Results, two days later
- Cache writes: $2.30/day down to $0.75, still falling
- Data transfer: 65 GB/day down to 8
- Compute: down 75%
- Every crawler I actually want: untouched
Back to hobby money. Site still wide open to the bots that matter.
What the computer guy learned
- Read what the billing UNITS mean. One sentence explained an impossible number.
- Every URL your site will answer is a tiny liability. I was offering the internet infinite URLs. Free for them, anyway.
- Your own server is a visitor too. It wears a name tag that says “node.”
- Domains have exes. Check what they left behind.
- Before changing anything, write down what number should move, and by how much, if your theory is right. I borrowed that habit from people far more experienced than me. It’s the difference between fixing something and poking it.
Full disclosure: I didn’t untangle this alone. I build with an AI coding assistant. I bring the dashboards and make the calls; it brings the patience to explain ISR billing to me twice. Twenty-five years of being everyone’s tech guy, and it turns out what I was missing wasn’t aptitude. It was a collaborator who never gets tired of my questions.
The site all this drama was protecting: https://flickomatic.com. Feedback welcome. I’m told that’s how you get better at this.
답글 남기기