In this final article, we’re going to explore Production practices
22. Events Are Delivered, Not Guaranteed
A common misconception in event-driven architecture is that publishing an event guarantees it will be processed exactly once.
Production systems rarely provide that guarantee.
Events may be:
- delivered multiple times
- delayed
- delivered out of order
- temporarily unavailable
- replayed long after publication
Reliable systems are not built on ideal conditions. They assume these scenarios will occur.
As a result, designing resilient consumers is just as important as designing reliable producers.
- Duplicate Delivery Is Normal
Consider the following workflow.
OrderConfirmed
|
v
Inventory Service
|
Inventory Reserved
Enter fullscreen mode Exit fullscreen mode
The Inventory Service processes the event successfully. Before acknowledging completion, it crashes.
After recovery, the event is delivered again.
From the messaging system’s perspective, this is correct behavior. From the application’s perspective, the same business event has arrived twice.
Without safeguards, inventory may be reserved twice.
Duplicate delivery is not an edge case. It is expected behavior in distributed systems.
23. Idempotency Makes Event Processing Safe
Idempotency ensures that executing the same operation multiple times produces the same result.
For event consumers, this means processing the same event repeatedly should not create duplicate business outcomes.
- Using Event Identifiers
Each event should include an eventId.
Consumers track processed events:
Processed Events
----------------
E-1001
E-1002
E-1003
Enter fullscreen mode Exit fullscreen mode
When a new event arrives, say: eventId = E-1002
The consumer checks if it has already processed it.
If yes, ignore it
If no, process it
Flow:
Receive Event
|
v
Already Processed?
|
+----+----+
| |
Yes No
| |
Ignore Process Event
|
v
Record eventId
Enter fullscreen mode Exit fullscreen mode
This approach ensures correctness even with duplicate delivery.
- Idempotency Belongs to Business Logic
Infrastructure can reduce duplicates but cannot eliminate them.
Idempotency must be enforced at the application level.
Example: a payment service must never charge a customer twice for the same event.
This guarantee belongs to business logic, not messaging infrastructure.
24. Ordering Cannot Always Be Assumed
Developers often assume events arrive in the order they were published.
In reality, delivery order can change due to:
- network latency
- retries
- parallel processing
- independent consumers
Expected sequence:
OrderCreated
|
OrderConfirmed
|
OrderShipped
Enter fullscreen mode Exit fullscreen mode
A consumer might observe:
OrderConfirmed
|
OrderCreated
|
OrderShipped
Enter fullscreen mode Exit fullscreen mode
The producer behaved correctly. The delivery order changed.
Consumers should not rely on strict ordering unless explicitly guaranteed.
- Design Consumers to Tolerate Reordering
Consumers should validate state before processing.
Example:
Receive `OrderShipped`
|
Order Exists?
Enter fullscreen mode Exit fullscreen mode
If required state is missing:
- retry later
- delay processing
- move the event for later handling
This approach handles temporary inconsistencies without assuming perfect ordering.
25. Correlation Makes Distributed Systems Understandable
A single user action can trigger multiple events across services.
Consider a sample flow:
Create Order
|
OrderConfirmed
|
InventoryReserved
|
PaymentCompleted
|
ShipmentCreated
|
InvoiceGenerated
Enter fullscreen mode Exit fullscreen mode
Without correlation, these events appear unrelated. But, with a shared identifier like REQ-98451 the entire workflow becomes traceable.
Correlation improves observability without affecting business logic.
26. Events Become Part of Your Audit Trail
Events naturally capture business history.
Example lifecycle:
OrderCreated
|
OrderConfirmed
|
PaymentCompleted
|
ShipmentCreated
|
OrderDelivered
Enter fullscreen mode Exit fullscreen mode
This sequence provides a complete record of what happened. Because events represent completed facts, they should never be modified.
History must remain immutable.
27. Observability Should Include Events
Traditional systems usually track:
- request latency
- response time
- database queries
Event-driven systems require additional metrics:
- published events
- consumed events
- failed processing
- retry count
- duplicate events
- processing latency
- dead-letter events
Without these, failures may go unnoticed until business issues appear.
Observability must include event flows, not just services.
- Tracing Event Flows
Distributed tracing becomes more effective with correlation identifiers.
Example:
Customer Request
|
Order Service
|
Inventory Service
|
Payment Service
|
Shipping Service
Enter fullscreen mode Exit fullscreen mode
Tracing allows engineers to follow a complete business workflow instead of isolated logs.
This significantly improves debugging and incident analysis.
28. A Practical Checklist
Reliable event-driven systems consistently follow these practices.
Design
✓ Events represent business facts
✓ Events expose business concepts, not internal models
✓ Payloads contain meaningful business data
✓ Technical metadata is separated from business data
Evolution
✓ Schemas evolve carefully
✓ Breaking changes are avoided
✓ Optional fields are preferred over removal
✓ Compatibility is continuously validated
Reliability
✓ Consumers are idempotent
✓ Duplicate delivery is expected
✓ Ordering is not assumed
✓ Correlation identifiers are included
Operations
✓ Contracts are validated automatically
✓ Event processing is observable
✓ Historical events remain immutable
✓ Producers and consumers evolve independently
Final Thoughts
Event-driven architecture is often described as a messaging pattern. In practice, it is a contract-driven architecture.
Messaging transports events. Contracts enable independent evolution.
Reliable systems are not defined by tools or frameworks. They are defined by disciplined practices:
- designing events as business contracts
- evolving schemas safely
- validating contracts continuously
- building resilient consumers
These principles apply regardless of technology.
When applied consistently, events become more than messages. They become stable, long-lived contracts that support scalable and maintainable distributed systems.
Assisted AI to paraphrase.
답글 남기기