When you hand an agent work with external effects, two worries always come up.
First: when the process dies, what happens to the work in progress? An agent paused at an approval gate crashes — can it resume, or does everything restart from scratch?
Second: when an LLM retries, does it execute the same side effect twice? Retrying a send or a data mutation carries a real double-execution risk.
My previous article measured HITL approval, audit trails, and structured output across three frameworks. It recorded a failure where Strands double-fired a publish call with the identical draft. This article measures what comes next: 34 more runs, same task, same model, same recorder proxy.
Repo (all code, traces, and analysis scripts open):
https://github.com/sunnydachs/agent-framework-showdown
(Read the previous article here. here.)
The experiment design
Same task, same model, same recorder proxy: 3 frameworks x 3 cells x multiple runs.
[agent] → [approval gate] → [publish (destructive)]
↑ ↑
crash it here pauses waiting
Enter fullscreen mode Exit fullscreen mode
The three cells:
- Cell A — crash recovery: SIGKILL a process paused at the approval gate; can a NEW process resume from where it stopped?
-
Cell B — idempotency: call
publishunder 3 key strategies (position key / content hash / no key) and count duplicate executions on retry - Cell C — audit under retry: score whether an auditor reading the traces alone can detect the duplicate
The three idempotency key strategies:
position key: {workflow}:{step}:{tool}
content hash: sha256(the raw arguments)
no key: nothing
Enter fullscreen mode Exit fullscreen mode
Cell A: crash recovery — a durable checkpointer is the difference between resume and redo
SIGKILL the agent while it waits for approval, then resume in a new process:
Framework Persistence Resume time State survived LLM calls to resume LangGraph (durable checkpointer) checkpointer on disk 0.01-0.02s 3/3 zero LangGraph (no checkpointer) none – 0/3 – Strands none built-in 4.9s avg full re-run 5.3 avg CrewAI none for agents 4.2s avg full re-run 2.0 avgLangGraph’s durable checkpointer persists the graph state to disk even while the process is dead. The new process restores to that position in 0.01s — with zero LLM calls. The difference between resume and redo is only whether the state lives outside the process.
Without a checkpointer, an identical-looking “resume” is a full re-run: Strands runs its average 5.3 LLM calls again and pays the full token cost a second time.
What if the crash happens before the first checkpoint?
LangGraph has a known issue here (#8764): if the process dies before the first checkpoint is persisted, recovery may find no checkpoint and no record that the run was ever accepted. On the version I tested, resuming the empty thread succeeded without raising — so the behavior is version-dependent. Don’t rely on the error either way; keep an external acceptance ledger.
Cell B: idempotency — the content-hash key silently fails exactly when the model rewords
Average duplicate executions when an LLM retry re-calls publish:
This is the core result. The content-hash key silently fails the moment the model rewords the arguments on retry.
The reason is simple. An LLM retry does not replay the saved HTTP request. It reasons again from a context that now includes the timeout error, and emits a new tool call. The arguments get reworded, the order changes, fields appear. With sha256(args) as the key, the retry produces a different hash, sails past the dedup check, and executes the side effect a second time.
The position key ({workflow}:{step}:{tool}) identifies the intent — where the call sits in the workflow — not the bytes. The same position with the same operation yields the same key no matter how the arguments change.
One more measurement: every duplicated publish carried a different tool_call ID on the wire. Nothing at the protocol layer detects “this is the second time for this operation.” Detection lives at the recording level only.
Note: the position-key strategy rejects “same position, different arguments” calls as a caller bug (2.67 of the runs here). That is by design — it flags intent drift instead of letting the key be reused.
Cell C: audit under retry — the trace-only auditor
Can an auditor reading the traces alone recover these four facts:
- Decision rationale is readable: 100% in all cells
- The duplicate is detectable: 100% in all cells
- Dedup is provable (which attempt was deduped, and why): 100% in all cells
- The retry directive is visible: 100% in all cells
But this is only because the recording is at the wire level. The proxy keeps every attempt — first call, dedup, caller-bug rejection — as its own record, so the auditor can reconstruct everything.
Framework-level trace surfaces show none of the double-firing. What matters in audit design is where the evidence lives.
The most important finding: the failure shape changes with the key strategy
Across the 34 runs, each key strategy fails differently:
- Content-hash key: fails silently. The hash just changes — no error, dedup passed, and only the side effect runs twice
- Position key: fails loudly. “Same position, different arguments” is rejected as a caller bug, so intent drift surfaces as an error
- No key: prevents nothing. Every retry executes twice
For enterprise use, the silent failure is the scariest class. The intuition “a hash key makes retries safe” breaks the moment the caller becomes non-deterministic — that is the conclusion from these measurements.
Honest limitations
- 3 runs per cell is directional, not a statistical claim
- The approver is scripted; no real UI or notification flow
- The destructive action is simulated (though whether the gate held is read directly from the recorded traffic)
- Single model. A different model may reword arguments differently
Reproduce it
Everything is open. The README has the commands for all experiments (34 runs here + 72 from the earlier articles):
https://github.com/sunnydachs/agent-framework-showdown
This is a personal OSS project — no warranty. Use at your own risk, and issues are welcome.