실제 재시도에서 살아남는 Idempotent Decision Endpoint 설계

Retries are normal in a distributed system. A client can lose a response after the server commits a decision, a webhook can be delivered again, or a queue consumer can restart after doing part of its work. An idempotent decision API lets the client send the same request again without creating a second business outcome.

That guarantee is stronger than adding a cache. The service must connect a stable operation key to the canonical request, the recorded result, the policy version, and any external effects. It must also define what happens during concurrency, partial failure, expiration, and payload mismatch. The result is an API contract that remains predictable during the failures production systems actually experience.

HTTP methods and idempotence use cases

In computer science, idempotence means that applying an operation again has the same intended effect as applying it once. HTTP makes this distinction at the method level. RFC 9110 explains that PUT, DELETE, and the safe methods are idempotent, while POST is not inherently idempotent.

A business decision is usually submitted with POST because it evaluates facts and records a new outcome. The method does not become safe merely because its calculation is deterministic. Without idempotency controls, two accepted submissions can create duplicate resources, conflicting audit records, repeated workflow transitions, or side effects like sending emails twice.

The practical rule is simple: the same logical operation should reach one recorded final state, regardless of how many times transport failure causes it to be submitted. A deliberately new business operation must use a new key.

How to implement idempotent APIs for decision requests

Start by writing the behavior as a contract rather than as an implementation detail. For a protected API endpoint:

• the initial request carries a unique key and a complete request body;
• the service associates that key with a canonical fingerprint;
• concurrent attempts cannot both own processing;
• subsequent requests with the same fingerprint retrieve the recorded result;
• reuse with a different fingerprint returns an error; and
• the response identifies whether it is original, in progress, or replayed.

This is what an idempotent API guarantees. It does not promise that every response byte or timestamp is identical. It promises that the protected business operation and its committed effects happen once per accepted contract.

Choose an idempotency key and API endpoint fingerprint

The client should create a high-entropy identifier for each logical operation. A UUID v4 is common, although another cryptographically random string can work. Do not derive the idempotency key from a mutable field or a low-cardinality value such as a customer number. A predictable key increases collision and replay risk.

The server must store more than the key. Canonicalize the fields that define business identity, then hash that representation. Normalize object ordering, number and date formats, omitted defaults, and insignificant whitespace before computing the fingerprint. Include policy or rule-set identity when a repeated call must reproduce the decision made under the original version.

When the same key appears with a different hash, return 409 Conflict rather than serving an unrelated cached result. If the key does not exist, reserve it before evaluating the decision. If it exists and the fingerprints match, follow the stored state.

The emerging IETF Idempotency-Key field draft describes a reusable HTTP header pattern. It is useful design input, but teams should document their own accepted syntax, retention, mismatch behavior, and response semantics instead of implying that every platform implements the draft natively.

Reserve async processing atomically under concurrency

Two workers may receive the same API call before either has saved a result. A read-then-insert sequence is unsafe: both workers can observe absence and both proceed. Use a database uniqueness constraint, conditional write, or transactional compare-and-set so only one worker can create the reservation.

A minimal state model is PROCESSING, SUCCEEDED, and FAILED_RETRYABLE or FAILED_FINAL. Store the operation key, fingerprint, state, timestamps, decision identifier, response snapshot, policy version, and recovery metadata. The reservation and the transition to a completed result must be durable.

For short work, competing callers can wait briefly and then return the completed response. For an async decision, return 202 Accepted with a status URL. A subsequent call can poll that resource. If processing is still active, do not start another evaluation merely because the first response has not arrived.

Here is platform-neutral Python-style pseudocode:

def decide(request, key):
    fingerprint = canonical_hash(request.body)
    record = reserve_or_read(key, fingerprint)

    if record.fingerprint != fingerprint:
        return conflict("key already represents another request")
    if record.state == "SUCCEEDED":
        return replay(record.response)
    if not record.owned_by_this_worker:
        return accepted(record.status_url)

    result = evaluate_policy(request.body, record.policy_version)
    return commit_result_and_outbox(record, result)

Enter fullscreen mode Exit fullscreen mode

The transaction boundary matters more than the programming language. The server process must not expose a successful decision that it cannot later retrieve.

Keep Kafka and external effects inside a reliable boundary

A database record can be written exactly once while a notification is still sent twice. If evaluation triggers messages, ledger writes, or workflow commands, commit an outbox record in the same transaction as the decision. A separate publisher can deliver that event with retry and backoff.

Consumers still need deduplication because brokers commonly provide at-least-once delivery. Use the decision ID or event ID as their unique key. Kafka producer features or broker acknowledgements can reduce repetition, but they do not replace application-level ownership of the business effect.

This separation also clarifies recovery. If the connection fails after commit, the client retries, the service reads the completed record, and the outbox continues independently. The second request does not create duplicate transactions merely because delivery status was uncertain.

Define response and status-code behavior

Clients need a deterministic map from stored state to HTTP status code:

• 201 Created or 200 OK for the first completed result;
• 200 OK for a replay, with a field or response header that marks it as replayed;
• 202 Accepted while another worker owns active processing;
• 409 Conflict when a key is reused with a different payload; and
• a documented final error when processing failed and automatic continuation is unsafe.

A 404 Not Found can be appropriate when a status resource has expired, but it should not silently authorize the client to recreate the old operation. Document whether an expired key may be reused and how the caller should create new work.

Return provenance with the decision: decision ID, operation key, processing status, policy version, evaluated time, and replay indicator. Do not recalculate a duplicate under newer rules. If the caller wants a current answer, that is a new operation with a new contract.

Set retention, expiration, and security rules

Retention should cover credible client retries, delayed redelivery, incident recovery, and the business period in which a repeated action would be harmful. A short Redis cache may be useful for speed, but it is not a sufficient source of truth when a request can return after the cache entry expires.

Choose a TTL from domain risk rather than convenience. Payment-style use cases may require a different period from content recommendations. Stripe’s idempotent request documentation is a useful concrete reference, but another system should not copy its retention policy without evaluating its own risk and privacy obligations.

Treat operation keys as untrusted input. Limit length and character set, scope them to the authenticated tenant, prevent cross-user retrieval, and avoid logging sensitive request data. Rate-limit repeated mismatches. If a key must expire, retain enough tombstone or business evidence to prevent an old operation from being mistaken for new work where that risk matters.

Test failure paths and observe retries

Unit tests are not enough. Run concurrent requests against the real persistence constraint, kill a worker after reservation, fail after decision commit, delay the outbox publisher, and resend after a timeout. Test retries of the same request as well as key reuse with a changed payload. Exercise network outages, failover, clock boundaries, and expiration.

Measure original operations, replayed responses, active-processing responses, conflicts, stale reservations, recovery actions, and event-delivery lag. Trace the key and decision ID across services without exposing raw sensitive facts. A rise in mismatch conflicts can indicate a broken client integration or abuse; a rise in prolonged processing records can reveal a crash loop or unavailable dependency.

Production checklist

Before launch, verify that:

• the API contract defines key generation, scope, and expiry;
• canonicalization and fingerprint behavior are deterministic;
• a unique constraint prevents two owners;
• payload mismatch cannot return an unrelated result;
• the response records policy provenance and replay status;
• the decision and outbox entry commit atomically;
• consumers dedupe repeated delivery;
• security and tenant isolation cover operation keys;
• concurrent, partial-failure, and delayed-redelivery tests pass; and
• dashboards expose conflicts, stale work, replays, and recovery.

DecisionManager publishes engineering guidance for governed, event-driven decision services. The essential principle is durable and testable: one business operation, one recorded outcome, and a predictable response to every safe resend.