Implementing the Transactional Outbox Pattern With EventBridge Pipes

Learn how to use EventBridge Pipes to implement transactional outbox pattern with zero or minimal additional code footprint.

Photo by Alina Grubnyak on Unsplash

An increasing number of systems have adopted an event-driven architecture. This means that as part of the system’s lifecycle, you will have to handle two operations: persisting your state and emitting the events associated with the state change.

This seemingly ordinary set of actions hides a potential problem in plain sight: those two are not atomic. This means at any given time you may face a problem when one of those two fails, leaving your system in an inconsistent state.

In this article I will revisit the pattern — previously covered here — providing an updated take on the implementation trying to leverage specific AWS functionality to handle this problem with minimal code involvement.

Transactional Outbox Redux

This pattern aims to solve the problem where due to your persistence and messaging medium being different the operation is not protected by any transactional controls.

Figure 1 illustrates the common case where you may be using a relational database to persist the current state of your system and publishing the associated event with a message broker.

Figure 1. Persisting state and event are done separately.

If one of these operations fails you could end up, for example, not notifying of a change that happened. In today’s reality where many event-driven systems adopt choreography, this would lead to an inconsistent state.

Figure 2. Failure to publish an event can leave your system state inconsistent.

If you adopt the transactional outbox pattern you fight this by making both operations go to the same medium, and then having a separate process that exclusively handles emitting the event