Event Driven Microservices architecture | Serverless360
← Return To Home

Event-Driven Microservices with Azure Functions, Event Grid and Cosmos DB

This blog is an extraction of the session “Event-Driven Microservices with Azure Functions, Event Grid and Cosmos DB” presented by Martin Abbott, who is Azure MVP, Regional Director.

Agenda

  • Microservices
  • Azure Functions
  • Event Grid
  • Cosmos DB
  • Scenario

We need to first have a deep understanding of all the services and components that are used to create the End to End application.

Traditional Multi-tier Architecture

Every architecture will have multiple layers for Business Logic, UI, Data Access layer, etc. Martin explained the pros and cons of traditional multi-tier architecture.

Traditional Multi-tier Architecture

Pros

  • Separation of concerns if controlled
  • Low service orchestration complexity
  • Easy to test through inversion of control

Cons

  • Mini- Monoliths reinforce “heritage thinking”
  • Tight coupling between layers
  • Hard to change without large scale deployments

Microservice Architecture

How microservice architecture is different from the traditional one. In this case, we break the architecture down into essential objects.

Every architecture or a software pattern has its own set of pros and cons. Compromise is inevitable.

Microservice Architecture

Pros

  • Clear Separation of concerns
  • Single responsibility and deployment boundary so extremely agile
  • Flexibility to use the best skills for each service

Cons

  • Higher service orchestration complexity
  • Harder to maintain consistency, hence difficult to maintain state
  • Need to deal with every transaction differently

Now let’s talk about Event-Driven design to get a deeper understanding of Microservices.

Event-Driven Design

Getting into the heart of this session, Event-driven design is not a traditional publish and subscribe architecture. Here in the first picture below, we have a bunch of services may interact with one another, there might be Service Bus involved to fulfil the messaging needs. What difference an introduction of Event Service can be inferred from the picture on the right.

Event Driven Design

Patterns for Writing Microservices

While developing in a new language or framework it’s always good to get started with a template that is readily available so that you can play around and understand the architecture. You can get some patterns for Event-Driven architectures here

Database / Service!!

Most people tend to stick with this traditional approach of having a separate data store below your microservice. This pattern favors the code in the language of choice against the best data store for the problem. The separation of data allows the scale to be applied as required.

However, this increases the complexity of the query consumer. Cross-database queries are hard considering the technology stack used between different services. To solve this problem, you need to consider data aggregation tools.

Saga

Choreography-based Saga predicted on events between services, unlike the traditional messaging pattern. You can explore this pattern here. Services can act independently so very loosely coupled. However, it increases complexity due to no transactional control. Martin also mentioned about the CQRS architecture which is used for separating read/write commands.

Saga-pattern

Transactional Outbox

In this case, when we write to our data store, along with the data we write something which indicates that something has happened and some other needs to be triggered based on the data.

  • Useful when two-phase commits are not possible or transaction boundaries span services
  • Events can be written to a record or a forward-only manner
  • Message relay processes to distributes the events to interested parties

pattern overview

Azure Functions

The king of microservice architecture. It has a vast number of users and more people are starting to try out to create their microservice in Azure Function.

Below are the key insights on how Azure Function stands out.

Azure-Functions

Serverless360 offers out-of-box capabilities for Azure Functions. Monitor and auto-correct state, view key performance indices in a real-time dashboard, detect runtime errors, evaluate performance, and have an eye on consumption and do much more that can improve the overall operational efficiency. Get an overview of all the features offered by Serverless360 for Azure Functions here. Refer to the product documentation for more details.

Event Grid

Martin justified why he uses Event Grid on his architecture. Since we have Event Grid bindings for Azure Function it will be super easy to push and pull events to the Event Grid endpoint.

EventGrid

The key reasons to choose Event Grid are

  • Reactive programming
  • Push not Poll
  • Filters to allow subscription
  • sub-second end-to-end latency in the 99th percentile
  • 99% Availability
  • 10,000,000 events per second per region
  • 10,000,000 subscriptions per region
  • 50ms publisher latency 
  • Transparent regional failover
  • 24 hours retry with exponential back off for events not delivered

If the events are not delivered, they get dead lettered. Here is where Serverless360 can add value with its capability to view and process the dead lettered events in Event Grid Subscriptions. Serverless360 also offers monitoring solutions that can help detect the arrival of any dead letters in the Event Grid subscriptions. Get an overview of all features offered by Serverless360 for Event Grids here. Also, refer to the product documentation for more details.

Cosmos DB

The database of choice is Cosmos DB for most of the people who have used Azure. It is NoSQL and super easy to get started with. Azure also provides Cosmos DB bindings for Azure Functions.

Key reasons to choose Cosmos DB are

  • Multi-model NoSQL database
  • Global-scale through geo-distribution
  • 99.99% high availability
  • Different Consistency models due to multi-master replications
  • Built-in Synapse Analytics

Events in Cosmos DB

Events in Cosmos DB currently, has no support for Event Grid. Change feed acts as a forward-only log of Transaction Outbox. Azure Functions has a binding for change feed processing which can help in reacting to the new entry in the Cosmos DB. This is how event-driven processes with Cosmos DB becomes possible. There is an ordered list based on the committed changes within a partition. Deletes are not implemented, so need a different process to deal with these. Only the latest change is included, so the intermediate changes might be lost. Best is to trigger on every change, or we should make sure the last change is what the business expects.

pusing to Change Feed

Demo – Order Fulfilment

Finally, the big show, where we see all the services together and the complete end to end flow. Here is the list of services with their respective languages.

  • Cart – C# / MongoDB
  • Order – C# / SQL
  • Payment – JS
  • Fulfilment – JS
  • Backorder – Python

How it all works?

  • Cart writes data to MongoDB and it creates an event in the Event Grid
  • Azure Function with an Event Grid binding listening to the cart service which writes the data to Cosmos DB
  • As a result of the validated event which triggers the change feed, it pushes another event to Event Grid
  • Now the Payment JavaScript function is triggered, and this will again publish an event
  • Now the order is again updated in the Cosmos DB, which will trigger another change feed event and finally, event passes through the backorder processing

End to End

View the complete demonstration by Martin here.

To Production and Beyond

What needs to be considered while moving this system to production are below

  • Security
  • Application Insights for Azure Functions logging
  • API Management for authentication and abstraction
  • Exception handling and messaging to cope with downstream issues
  • Robust eventing and routing, deal with exponential back off timeout
  • Premium tier to avoid cold start and service timeouts
  • CI/CD is essential for agility

Also, consider Serverless360 to improve the operational efficiency while supporting Serverless application in Production

Wrap up

Below are the key takeaways from this session on Event-Driven Microservices with Azure Functions, Event Grid and Cosmos DB

  • Cosmos DB change feed trigger, input and output binding only work for SQL API
  • Cosmos DB change feed does not yet support deletes so these need to be handled differently
  • Event Grid binding only works for Event Grid events, NOT Cloud Events, so need to use HTTP Trigger
  • Calling between services is still okay, but consider an API gateway
  • The decision should be made on requirements, sometimes a monolith, or at least the impression of a monolith, is the answer
  • Adds complexity, need to consider consistency, event delivery and processing, service co-ordination and query aggregation
  • Adds agility and expands options so provides a great opportunity to future proof.

There are many ways to build microservices architecture with the bits in Microsoft Azure. In this session, Martin used Azure Functions to provide a lightweight implementation of services leveraging the capabilities of the Cosmos DB binding and Azure Event Grid to drive consistency across the data sources.