System design: Microservice.io - Chris Richardson | Transaction Log Tailing With Debezium — Part 1

 April 5, 2021

Here is the article on medium.com. 

Sometimes there can be more than one action we need to take in any business logic, especially when coding a business on the backend development side. There may be event-driven actions such as publishing a message to any message broker in these actions to be taken.

As an example, let’s assume that we receive the information that the customer returns are approved and processed in the database in a return process. We also need to leave a message in a message broker system to pay the money back. 

Failure to send the event is possible. What should we do to avoid such a situation? The Transactional Outbox pattern comes into play right at this point.

This model provides an effective solution for reliably publishing events. The purpose of this approach is to have an “Outbox” table in the database of the service. When receiving the registration request, not only addition is made to the ReturnOrder table, but also a record representing the event is added to the Outbox table.

An asynchronous process monitors the Outbox table for new entries and publishes any event on the Event Bus. The model improves reliability by simply dividing two processes on different services.

Polling Publisher

The Polling Publisher approach is a structure built to publish the records in the database’s Outbox table in order. A scheduled job that runs periodically processes the records in the Outbox table, then deletes the corresponding record. This structure is easy and costless to implement in low-scale systems. However, it also has some disadvantages.

• If you have a multi-instance running schedule job, it will be necessary to manage ‘Optimistic Lock’ to ensure that the recordings are published only once.
• Publishing events in order is difficult.
• As the number of calls to the database increases as the scale increases, it will cause a performance problem after a while.
• Not applicable in all NoSQL databases.

Using the outbox pattern with polling publisher approach

Transaction Log Tailing

This is another option to access distributed transactions. The idea is to queue the database transaction log and publish each change as an event. Each committed update is displayed as an entry in the transaction log of the database. A transaction log miner can read the transaction log entry and publish each change as a message to Message Broker. Converts each relevant log entry corresponding to the added data into a message. The messages are then published in a Message Broker. The benefit of this model is that there is no need for any changes at the application level; everything happens at the database level. It is a little difficult to avoid duplicate publishing. Therefore, in order to ensure data consistency, the application that will consume the related events must be idempotent.

The benefits of applying this pattern are;

• There is no two-phase commit.
• Transactions take place in order.
• The accuracy of the data is guaranteed.
• No extra calls are made to the database through an application.
• It can be applied in high scale systems.

Using the outbox pattern with transaction log tailing approach

More detailed information about the Transactional Outbox pattern is available on Chris Richardson’s microservices.io site.