Imagine that your team has been running a distributed event sourced architecture for many years and then one day you realised that you’ve been doing event souring wrong the whole time.
This is where we are at now.
As you can see, change events are published from a microservice directly to a notification service while at the same time the application state is written to a database.
The problems
The event and the application state are not written within the same transaction. This means that if one fails and the other succeeds, the system as a whole will be in an inconsistent state. For example, if the event is published successfully but the application state failed to be written to the database, now you have an event that does not match the current state.
The microservice publishes events like AccountChanged or PersonalDetailsChanged which contain the full state rather than a delta. At a glance, these events don’t tell us much about what actually happened and you’re forced to look at the content and compare events before and after to determine what actually changed and why.
The solutions
Firstly, we implemented the outbox pattern and utilized DynamoDB transactions to at least guarantee consistency of state and events.
In the diagram there are three tables. One for storing the domain events, another for store the current state and another for sending out integration events which are what those AccountChanged and PersonalDetailsChanged events are. Why trigger an event from a DynamoDB stream from the application state you might ask? Well, the events include additional metadata which would also need to be included with the state. Since the state is read heavy and DynamoDB does not provide a way to filter out fields in order to reduce read cost a separate table with a TTL was used instead to contain the event record. A stream event from that table then sends publishes the message to SNS.
The other change was to have more business process orientated events written to the domain events table. Events such as AccountCreated and AccountClosed, EmailAddressReplaced, NameChanged, MovedAddress, to name a few. These also contain only the delta of what changed with a revision number.
The application state also includes a revision number which we use to control concurrent updates as well as allow consumers to reorder events using the resequencer pattern. Initially we thought we could try to guarantee message order by using FIFO queues, but the truth is that the more systems messages have to pass through the greater the chance they will have to get out of order. Take DynamoDB stream events for example. DynamoDB only guarantees the order of events per item and with domain events requiring both a partition key (eventId) and sort key (revisionNumber), each revision of the same event will get streamed through a different shard and processed in parallel. In addition to this, not only can out of order messages be catastrophic it’s also really hard to debug. Therefore, we decided that we’ll leave resequencing to the consumer and we simply provide the means.
References
AWS Outbox Pattern
https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html
Implementing Domain Driven Design
https://amzn.asia/d/077Wd0OX
AWS DynamoDB Streams
https://tomodahinata.com/en/blog/dynamodb-streams-event-driven-architecture-cdc-lambda-eventbridge-guide
Resequencer Pattern
https://amzn.asia/d/07c9UWXQ