Anthropic Batch API: 112 of My 1,842 Jobs Never Came Back

A user emailed me a screenshot of an empty report. Score field blank. I opened the database and found the row sitting there with score = NULL, created four hours earlier, no error logged anywhere. My worker had run, my batch had finished, my code had written nothing and moved on cheerfully.

That was the third one that week. So I pulled 30 days of logs and counted: I had pushed 1,842 scoring jobs through the Anthropic Batch API, and 112 of them never produced a usable result. Not 112 crashes. 112 silent holes, because “the batch ended” and “my requests succeeded” are two completely different facts and I had written code that assumed they were the same one.

TL;DR

• Over 30 days I sent 1,842 requests in 96 batches through the Anthropic Batch API. 1,730 were usable (93.9%). 71 errored, 24 expired at the 24-hour wall, and 17 “succeeded” but came back truncated.
• batch.processing_status == "ended" does not mean success. Read request_counts and switch on each result line’s result.type (succeeded / errored / expired / canceled).
• Results come back in arbitrary order. Zipping them against your input list silently attaches the wrong output to the wrong record. This was my worst bug, and it never threw.
• A succeeded result with stop_reason: "max_tokens" is a truncated JSON body. My parser swallowed it and wrote a null.
• Latency is wildly skewed: median 9 minutes, p99 over 6 hours. Batch is for work nobody is staring at. My bill halved ($209 → $104), which is the only reason any of this was worth it.

What is the Anthropic Batch API actually good for?

It is asynchronous bulk inference at roughly half price. You hand Anthropic a list of up to 100,000 requests, each wrapped in a custom_id, and poll until the batch ends. The tradeoff is the entire point: you give up latency guarantees and you get a discount. There is a 24-hour ceiling on processing.

My workload fit on paper. I run a platform that does mock voice interviews and hands back a written report afterward, plus a separate scoring pass over the candidate’s portfolio. The portfolio pass is nightly, nobody is watching, and it is a fat prompt. Perfect batch job.

The interview report is not a batch job, and I put it in one anyway. That was the first mistake and I’ll get to it.

batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": session_uuid,          # not the user's email
            "params": {
                "model": "claude-sonnet-5",
                "max_tokens": 4096,
                "messages": [{"role": "user", "content": prompt}],
            },
        }
        for session_uuid, prompt in pending
    ]
)

Enter fullscreen mode Exit fullscreen mode

One small thing worth doing on day one: custom_id comes back to you in the results file, so make it an opaque internal ID. I used the session UUID. Do not put an email address there because it was convenient.

Why did 112 of 1,842 Anthropic Batch API jobs never come back?

Because I only checked one flag. Here is the real breakdown from the 30-day run:

result.type count succeeded 1,747 errored 71 expired 24 canceled 0

Of the 71 errored: 52 overloaded_error, 14 invalid_request_error (transcripts that blew past my own token budget), 5 generic api_error. Then, of the 1,747 that succeeded, 17 came back with stop_reason: "max_tokens" — valid API responses containing half a JSON object. My json.loads threw, my except block logged at DEBUG, and the row stayed null.

71 + 24 + 17 = 112. A 6.1% silent failure rate on a pipeline I thought was green.

Does a batch with processing_status: "ended" mean my requests succeeded?

No. ended means Anthropic is done working on the batch, including the parts it gave up on. The per-request outcome lives in two places: batch.request_counts, and the JSONL results stream.

My original loop, in full embarrassing honesty:

while batch.processing_status != "ended":
    time.sleep(30)
    batch = client.messages.batches.retrieve(batch.id)

results = list(client.messages.batches.results(batch.id))
for session, entry in zip(pending_sessions, results):   # <-- both bugs live here
    save_report(session.id, entry.result.message.content[0].text)

Enter fullscreen mode Exit fullscreen mode

Two failures in one line. entry.result.message does not exist on an errored or expired entry, so those raise an AttributeError I was catching too broadly. And zip against pending_sessions assumes the results arrive in submission order.

They do not.

Why did my results get attached to the wrong session?

Because batch results come back in arbitrary order and I matched by position instead of by custom_id. When every request succeeds, the order is often close enough to input order that nothing looks wrong. The moment one request errors out, the list shortens by one and every entry after it shifts up a slot. Candidate A gets Candidate B’s report.

This is the bug that actually scared me. No exception, no alert, no 500. Just quietly wrong output rendered in a nice template. I found it by reading a report that praised a Kubernetes project the person had never mentioned.

Full disclosure: the product is Preterview, which I built and run. It does voice interviews with three interviewer styles and returns a written report, and a report is the entire deliverable, so shipping one person’s feedback to another person’s account is about as bad as my failure modes get. Two sessions were affected before I caught it. I emailed both people, which is a conversation I recommend avoiding by writing the loop correctly the first time.

The fix is four lines and I should have written them on day one:

by_id = {}
for entry in client.messages.batches.results(batch.id):
    by_id[entry.custom_id] = entry          # never trust order

for session in pending_sessions:
    entry = by_id.get(session.id)
    if entry is None:
        mark_retry(session.id, "missing_from_results")
        continue

    match entry.result.type:
        case "succeeded":
            msg = entry.result.message
            if msg.stop_reason == "max_tokens":
                mark_retry(session.id, "truncated")
            else:
                save_report(session.id, msg.content[0].text)
        case "errored":
            mark_retry(session.id, entry.result.error.type)
        case "expired" | "canceled":
            mark_retry(session.id, entry.result.type)

Enter fullscreen mode Exit fullscreen mode

Every branch writes a row. There is no path through that block where a request disappears.

How long does an Anthropic Batch API job actually take?

Median 9 minutes across my 96 batches. p90 was 51 minutes. p99 was 6 hours 20 minutes. And 24 requests, spread across two unlucky batches, sat until the 24-hour ceiling and came back expired.

That distribution is the whole design constraint. The median tempts you into thinking batch is a slightly slower sync API. It is not. You have to build for the tail, because the tail is where your users live.

Which is why the post-interview report is back on the synchronous API, where it always belonged. Someone just finished talking for 25 minutes; they are not going to refresh for six hours. The nightly portfolio rescoring stayed on batch, and that is the workload the discount was designed for.

The reconciler is the actual deliverable

The piece of this I’d write first next time is not the submitter. It is the reconciler: a table of (custom_id, batch_id, status, attempts) and a cron job that sweeps it.

Rules I landed on after the postmortem:

1. Write the row before you submit. Every custom_id exists in the database as pending before the batch is created. If it never comes back, it is already visible as a gap.
2. Switch on result.type exhaustively. No default branch that silently passes.
3. Check stop_reason on success. max_tokens is a failure wearing a success costume.
4. Two batch attempts, then fall back to the sync API. Paying full price for 6% of jobs is cheaper than an empty report.
5. Alert on the gap, not the error. My monitor now compares rows submitted to rows resolved per batch. The one number that would have caught all 112.

After the reconciler shipped, the next 30 days ran 2,100-ish requests with zero unresolved rows. Still had errors. Still had a handful of expiries. But every one of them now ends up retried or dead-lettered instead of vanishing into a null column.

So what happens to the jobs that never come back from the Anthropic Batch API? Nothing happens to them. That is the problem. A batch reaching processing_status: "ended" only means Anthropic stopped working on it, and the results JSONL can contain errored, expired, or canceled entries alongside successes, in arbitrary order, plus succeeded entries truncated at max_tokens. If your consumer zips results against the input list and reads .result.message without checking .result.type, those requests are not reported as failures — they are dropped, or worse, shifted onto the wrong record. Match on custom_id, handle all four result types, verify stop_reason, and reconcile submitted-versus-resolved counts per batch. In my 30-day run that was the difference between 6.1% silent data loss and a clean retry queue, at half the API cost.

Written by the developer behind Preterview, an interview prep platform.