Communicate in microservices with Saga Pattern, Event Sourcing using RabbitMQ

In this article, I want to summary all of parts (patterns and stacks) to implement Inter-communication:

  • First, someone can communicate by transmitting messages. So what is message and message pattern enterprise?
  • Second, the transmitting messages isn’t simple between services. It make sure good consistency of database. So Saga pattern is a good solution.
  • Third, Saga pattern have 2 popular ways to implement. But in our scenario, we will choose Events / Choreography, so we have Event sourcing pattern.

1/ Message Patterns

There’re a lot of message patterns, but internal-communication between services in this article is built from some below popular ones.

In scenario is finding how to communicate between micro-services, I will introduce some patterns must have and how to integrate technologies with it.

Messages

A Message is an atomic packet of data that can be transmitted on a channel. Thus to transmit data, a service must break the data into one or more packets, wrap each packet as a message, and then send the message on a channel. Likewise, a receiver service receives a message and must extract the data from the message to process it. The message system will try repeatedly to deliver the message (e.g., transmit it from the sender to the receiver) until it succeeds.

This is duty of all of services which need to send and receive messages to process its works.

Channels — Messaging services transmit data through a Message Channel, a virtual pipe that connects a sender to a receiver. A newly installed messaging system doesn’t contain any channels; you must determine how your applications need to communicate and then create the channels to facilitate it.

Routing

Message Router, an application component and filter in the pipes-and-filters architecture, which will determine how to navigate the channel topology and direct the message to the final receiver, or at least to the next router.

Publish-Subscribe Channel

A Publish-Subscribe Channel works like this: It has one input channel that splits into multiple output channels, one for each subscriber. When an event is published into the channel, the Publish-Subscribe Channel delivers a copy of the message to each of the output channels. Each output channel has only one subscriber, which is only allowed to consume a message once. In this way, each subscriber only gets the message once and consumed copies disappear from their channels.

2/ Saga Patterns

Ref: https://blog.couchbase.com/saga-pattern-implement-business-transactions-using-microservices-part/

Ref: https://blog.couchbase.com/saga-pattern-implement-business-transactions-using-microservices-part-2/

A saga is a sequence of local transactions where each transaction updates data within a single service. The first transaction is initiated by an external request corresponding to the system operation, and then each subsequent step is triggered by the completion of the previous one.

There are a couple of different ways to implement a saga transaction, but the two most popular are:

  • Events/ Choreography: When there is no central coordination, each service produces and listen to other service’s events and decides if an action should be taken or not.
  • Command/Orchestration: when a coordinator service is responsible for centralizing the saga’s decision making and sequencing business logic.

Events / Choreography

In the Events/Choreography approach, the first service executes a transaction and then publishes an event. This event is listened (subscribe) by one or more services which execute local transactions and publish (or not) new events.

  • Explaining:
  • Order Service -> (1): Create New Order published a event named ORDER_CREATED_EVENT.
  • (1) -> Payment Service-> (2): Payment Services subscribed event ORDER_CREATE_EVENT, so it received signal then process data, after that it publishes to event BILLED_ORDER_EVENT.
  • (2) -> Stock Service -> (3) -> Delivery Service -> (4) -> Order Service -> In the end.
  • Order Service saves a pending order and asks Order Saga Orchestrator (OSO) to start a create order transaction.
  • OSO sends an Execute Payment command to Payment Service, and it replies with a Payment Executed message
  • OSO sends a Prepare Order command to Stock Service, and it replies with an Order Prepared message
  • OSO sends a Deliver Order command to Delivery Service, and it replies with an Order Delivered message
  • With our scenario, we use Events / Choreography to apply
  • Next, we will approach to Event Sourcing

3. Event Resourcing.

A service typically need to atomically update the database and publish messages/events. For example, perhaps it uses the Saga pattern. In order to be reliable, each step of a saga must atomically update the database and publish messages/events. Alternatively, it might use the Domain event pattern, perhaps to implement CQRS.

What is CQRS? CQRS is Command Query Responsibility Segregation. It mean each command is responsible for each event which is subscribed.

Customers and Orders is an example of an application that is built using Event Sourcing and CQRS.

That’s all. We go to how to implement

4. How to implement intercommunication on services.

After finishing researching necessary patterns, we need:

  • Event Bus works as a message broker, named “Channel” in Publish-Subscribe Pattern to receive and deliver messages.
  • Workers / Processes run along with services to receive signals of service then publish messages to Event Bus as well as always subscribe to events of Event Bus then send data to others.
  • Each event corresponding with each function, follow CQRS.

What is RabbitMQ? https://www.rabbitmq.com

RabbitMQ is lightweight and easy to deploy. It supports multiple messaging protocols.

RabbitMQ can be deployed in distributed and federated configurations to meet high-scale, high-availability requirements.

RabbitMQ allow services pub/sub using exchange, routing_key and queue.

RabbitMQ define routing as the combination of exchange, queue and routing_key.

Ref: https://www.rabbitmq.com/tutorials/tutorial-four-python.html

X — is a exchange which has a topic type.

*.orange.*, * . * .rabbit, lazy.# — are routing keys which is patterns help services determine need to communicate.

So we use Rabbit for a Event Bus

What is Celery? http://www.celeryproject.org

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, supports scheduling, support well with RabbitMQ and Django ORM.

We use Celery works a Workers run a long with services.

Implement

This is a example workflow for function named “Register a new account” after implementing inter-communication services.

References:

Keep calm and get unlimited potential