Webhooks fail in ways that stay invisible until production traffic hits them: the same event arrives twice, a slow database makes the sender give up and retry, and events land out of order. The fix is not a bigger server — it’s treating every webhook as at-least-once delivery and making your handler idempotent, fast to acknowledge, and safe to replay. Do those three things and most 2 A.M. pages disappear.
I’ve built and debugged webhook receivers for payment providers, Git hosts, and internal event buses. The bugs are almost always the same handful, and they’re all preventable. Here’s the mental model and the code.
Why do webhooks get delivered more than once?
Because the network is unreliable and senders choose safety over precision. A webhook provider (Stripe, GitHub, Shopify, your own service) sends an HTTP POST and waits for a 2xx within a timeout — often just a few seconds. If your endpoint is slow, returns a 5xx, or the acknowledgment packet gets lost on the way back, the sender assumes failure and retries. Your handler already did the work, but the sender never heard “yes.”
This is at-least-once delivery, and it’s the correct design on the sender’s side. It means the burden of not double-processing lands on you, the receiver. Every serious provider documents this. Stripe, for example, is explicit that you may receive the same event more than once and that handlers must be idempotent.
The mistake I see most is a handler that does real work — charge a wallet, send an email, insert a row — before it returns 200. Any slowness turns one logical event into two side effects.
Takeaway: assume every webhook will be delivered at least twice, and design so the second delivery is a no-op.
How do you make a webhook handler idempotent?
Idempotency means processing the same event twice produces the same result as processing it once. The reliable way to get there is a dedup key plus a uniqueness constraint in your database — not an in-memory Set, which evaporates on restart and doesn’t work across multiple instances.
Most providers send a stable event ID (id on Stripe events, the X-GitHub-Delivery header on GitHub). Store it. Let the database reject the duplicate atomically:
CREATE TABLE processed_webhooks (
event_id TEXT PRIMARY KEY,
event_type TEXT NOT NULL,
received_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
Enter fullscreen mode Exit fullscreen mode
// Node + node-postgres. Returns true only the FIRST time an event_id is seen.
async function claimEvent(client, eventId, eventType) {
const res = await client.query(
`INSERT INTO processed_webhooks (event_id, event_type)
VALUES ($1, $2)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id`,
[eventId, eventType]
);
return res.rowCount === 1;
}
Enter fullscreen mode Exit fullscreen mode
The ON CONFLICT DO NOTHING ... RETURNING pattern does the check and the claim in one atomic statement, so two concurrent deliveries of the same event can’t both win the race. If claimEvent returns false, you’ve already handled this event — acknowledge with 200 and stop.
For the strongest guarantee, do the claim and the side effect in the same transaction, so a crash between “claimed” and “processed” doesn’t strand you:
async function handleEvent(pool, event) {
const client = await pool.connect();
try {
await client.query('BEGIN');
const isNew = await claimEvent(client, event.id, event.type);
if (!isNew) {
await client.query('ROLLBACK');
return { status: 'duplicate' };
}
await applySideEffect(client, event); // same transaction
await client.query('COMMIT');
return { status: 'processed' };
} catch (err) {
await client.query('ROLLBACK');
throw err; // let the caller return 500 so the sender retries
} finally {
client.release();
}
}
Enter fullscreen mode Exit fullscreen mode
If your side effect touches an external system you don’t control (sending an email, calling a third-party API), you can’t wrap it in the same SQL transaction. There, make the external call idempotent too — many APIs accept an Idempotency-Key header; pass your event ID as that key so the downstream service dedupes on its end.
Takeaway: a database uniqueness constraint is the only dedup that survives restarts, deploys, and horizontal scaling.
Should you process webhooks synchronously or queue them?
Queue them. The single highest-leverage change you can make is to acknowledge fast and process asynchronously. Verify the signature, persist the raw payload, return 200, and let a background worker do the real work.
The reason is the retry timeout. If your handler takes 8 seconds and the sender’s timeout is 5, you get a retry even on success — you did the work, but the sender never saw your 200 in time. Under load, this snowballs: slow handlers cause retries, retries add load, more load makes handlers slower.
Here’s the split:
app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), async (req, res) => {
// 1. Verify signature on the RAW body (see next section)
let event;
try {
event = verifyAndParse(req.body, req.headers['stripe-signature']);
} catch {
return res.status(400).send('bad signature');
}
// 2. Persist raw + enqueue, then acknowledge immediately
try {
await enqueue(event); // durable queue: DB table, SQS, Redis stream...
return res.status(200).send('ok');
} catch {
return res.status(500).send('retry me'); // couldn't even enqueue → let sender retry
}
});
Enter fullscreen mode Exit fullscreen mode
The worker then runs handleEvent from above, with retries and backoff you control. Your “durable queue” can be as simple as an events table with a status column if you’re not ready for SQS or a Redis stream — the point is that acknowledgment no longer waits on business logic.
Takeaway: acknowledge within the sender’s timeout; do the slow work behind a durable queue you own.
What order do webhooks arrive in, and can you trust it?
You cannot trust order. Retries and parallel delivery mean a subscription.updated can land before the subscription.created it logically follows. Design for it.
Two defenses. First, treat each event as a fact about state at its timestamp, and ignore stale ones. If an event carries an updated_at or a version/sequence number, compare against what you’ve stored and drop anything older:
// Only apply if this event is newer than what we've already recorded.
const applied = await client.query(
`UPDATE subscriptions
SET status = $2, source_updated_at = $3
WHERE id = $1 AND source_updated_at < $3`,
[sub.id, sub.status, sub.updated_at]
);
if (applied.rowCount === 0) { /* stale or duplicate — safely ignored */ }
Enter fullscreen mode Exit fullscreen mode
Second, when an event references an object you haven’t seen yet, re-fetch the current state from the provider’s API instead of reconstructing it from event history. The webhook tells you something changed; the API tells you the truth right now. This also rescues you from events you missed entirely during an outage.
Takeaway: webhooks are change notifications, not an ordered event log — reconcile against the source of truth when order matters.
How do you keep signature verification from silently breaking?
Signature verification is where “it works on my machine” bites hardest, because the failure is a 400 the sender sees, not an error you see. Two rules:
-
Verify against the raw request body, byte for byte. If your framework parses JSON before you compute the HMAC, re-serialization changes whitespace and key order, and the signature won’t match. In Express, mount
express.raw()on the webhook route only — not a globalexpress.json()that runs first. - Keep the signing secret in config, rotate it deliberately, and support two secrets during rotation so in-flight events signed with the old key still verify.
Never skip verification “temporarily.” An unverified webhook endpoint is an unauthenticated write to your database that anyone who learns the URL can call.
Quick reference: the failure modes and their fixes
Symptom Root cause Fix Same action happens twice At-least-once retries DB uniqueness constraint on event ID Retries even when handler succeeds Handler slower than sender timeout Ack fast, process in a background worker “Object not found” mid-handler Out-of-order delivery Re-fetch current state from provider API Older data overwrites newer Order not enforced Compare timestamps/versions before writing Intermittent 400s from sender Signature computed on parsed body HMAC the raw bytes; raw parser on that route only Events lost during an outage No catch-up path Reconcile via API; use provider’s event replayBottom line
If you build only one thing, build the idempotent claim: a processed_webhooks table with the event ID as primary key and ON CONFLICT DO NOTHING. That single constraint kills the most damaging class of bug — double side effects — no matter how many times an event is redelivered. Add fast acknowledgment with a background worker next, because slow handlers manufacture their own retries. Then handle ordering by reconciling against the provider’s API rather than trusting the sequence. Verify signatures on raw bytes, always. Do these four and your webhook receiver stops being the thing that wakes you up.
답글 남기기