Faster Runs. Stop Optimising for the Wrong Number.”>
Most “we made it 3x faster” scraper posts miss the actual point. Speed is rarely the constraint. Replayability is.
If your scraper takes 4 hours to run and a single URL fails halfway through, can you re-run just that URL in 30 seconds? Or do you have to start the whole 4-hour job over?
If the answer is “start over,” you don’t have a scraper. You have a long-running prayer.
The 3-item checklist
A replayable run looks like this:
- Inputs are explicit and persisted. Every URL/parameter the run is processing is written to a queue or dataset before it starts. You can re-read the input list later.
- Outputs are addressable per input. You can ask “did URL X succeed?” and get a yes/no, not “well, the run finished, so probably.”
- Failures are first-class records. Failed inputs go to a separate dataset/queue with the error reason, ready to feed back into a retry run.
When all three hold, “rerun the failures” is a one-liner. When any of them is missing, recovery is manual archaeology.
The trick — input/output as separate datasets
Here’s the shape:
from apify import Actor, Dataset
from apify.storages import RequestQueue
await Actor.init()
input_data = await Actor.get_input()
# 1. Push inputs into a queue. Idempotent — re-runs skip already-done items.
queue = await RequestQueue.open(name="podcast-urls-2026-06")
for url in input_data["urls"]:
await queue.add_request({"url": url, "uniqueKey": url})
# 2. Process the queue, splitting outputs into success and failure datasets.
results = await Dataset.open(name="podcast-results-2026-06")
failures = await Dataset.open(name="podcast-failures-2026-06")
while (request := await queue.fetch_next_request()):
try:
record = await transcribe(request["url"])
await results.push_data(record)
await queue.mark_request_as_handled(request)
except Exception as e:
await failures.push_data({
"url": request["url"],
"error": str(e),
"failed_at": datetime.utcnow().isoformat(),
})
await queue.mark_request_as_handled(request) # don't retry blindly
Enter fullscreen mode Exit fullscreen mode
Three storages: input queue, success dataset, failure dataset. The queue is keyed by URL, so adding the same URLs again is a no-op. The failure dataset is the input for the next retry run.
Quick case
The podcast transcription actor used to be a 6-hour batch job. When a single episode failed (audio download timeout, transcription model glitch, anything), the recovery story was: “find the failed URL in the logs, hand-craft a one-URL run, hope the second try works.”
After moving to the queue + split-dataset pattern:
- Failed URLs are visible in a dedicated dataset, with the error and timestamp.
- “Retry yesterday’s failures” is one button: open the failures dataset, push its rows into a new run as input.
- The original run’s success dataset doesn’t get re-processed — it just gets appended to.
What used to take 30 minutes of manual triage is now a 30-second action. Same scraper, same selectors, same model — different runtime structure.
The CTA you didn’t ask for
The queue + success-dataset + failure-dataset pattern is the third thing every actor we ship gets, after request blocking and selector ladder — visible in the podcast transcription actor. (We have a starter template now. Same shape every time.)
So:
Open your scraper. If a single URL fails, what does recovery look like? If your answer takes more than one paragraph, drop it in the comments — I’ll show you the smaller version.
Agree, disagree, or have a recovery story that doesn’t need this? Reply.
Written by **Nova Chen, Automation Dev Advocate at SIÁN Agency. Find more from Nova on dev.to. For custom scraping or automation work, hire SIÁN Agency.
답글 남기기