Event Driven Integration

Key Takeaways

Dan is an Enterprise Integration Consultant, Solution Architect, Application Developer, and Systems Analyst with over twenty years of experience. He is into integrating large enterprise systems within the private sector, Queensland Government, and Australian Federal Government. Microsoft certified in Azure, BizTalk Server, and .NET, Dan specializes in enabling Digital Transformation through Business Process Automation and Connected Systems, leveraging the Microsoft solution stack both in the cloud and on-premises.

Dan is also a Microsoft MVP (Azure), a Microsoft Certified Trainer (MCT), and a published Pluralsight author. He currently serves as a Senior Lead for the Platform Engineering division of Deloitte Australia, where he is considered a specialist in Microsoft integration.

The importance of event-driven integration in real-time

We live in a world where API runs everything! They are everywhere, and that is how applications talk to each other. One of the challenges is getting the data synchronized across everything, which is where integration helps. We expect everything to happen immediately, and we want the information when it is current. Let’s take, for example, that most of us have online bank accounts. When you rack up a significant charge on it, you might get a text message on your phone to warn you about that, just in case, it wasn’t you who did the charge.

It wouldn’t be as helpful if you got that text message the next day because of an overnight batch process. Data is only as valuable as it is current, and a lot of data loses its value very quickly. Event-based integration allows you to get the data as it happens in real time. It is essential in a lot of situations like banking.

The difference between discrete events and series-based events

A discrete event is an actionable event. For example, an event is created in a financial transaction, and if somebody is interested in it, you can notify that person. They can also choose to do something about that information. But other events only get their meaning from the context of the events around them. For example, if you’re monitoring the temperature in a room and taking temperature readings every minute. The fact that the temperature in this room right now might be 25 degrees, it’s probably not actionable.

But if in three minutes, it’s 28 degrees, and in five minutes, it’s 35 degrees, that suggests something that becomes very significant. It’s essential to understand the distinction between those events and which ones you’re dealing with when you design your application. Because naturally, the platforms that will be used to process those events must be different.

Azure services that are ideal for event sourcing in event-driven integration

There are two primary services in Azure for handling events. The first is the event grid, which would be your event notifications. In the first scenario of discrete events, you pass the events out as they happen, and the people who receive those events can react to them as they see fit for event sourcing. Sometimes, you need to persist those events and store them because you may need to replay them and use them, for example, to rebuild the state of something or to be able to do some analysis over them. The product for that would be Event Hubs in Azure because Event Hubs can record those events and replay them, which Event Grid does not. Understanding what architecture you need depends on your requirements and how you will use those events.

How resilient and reliable are these events services?

Resiliency and error management are things you must consider when you’re building any application. Event Grid as a service is highly resilient. It will try to deliver messages for up to 24 hours and automatically retry if it’s pushing out to a subscriber. If it can’t reach that subscriber, it will continually retry, and it will be very clever about how it does, and it uses an exponential backoff.

Because you can imagine if the receiver is being hammered and can’t receive the message because it’s under stress, then continually trying and getting it every five seconds will add to the problem.

Event Grid will try for five seconds, then 10 seconds, then 30 seconds, and then wait a couple of minutes and expand that out. It also randomizes those intervals slightly because if you have a lot of events coming through at once, if they all retry at the same interval, then you are just pushing the problem down the road. Randomization will cause it to fan out slightly so that you can give the receiver a chance to receive them.