Microsoft Azure Event Hubs
Microsoft Azure owns the finest business communication technologies like Service Bus, Event Hubs, Event Grids and Storage queues. This pillar page will focus on Event Hubs and impart the knowledge on its concepts, properties, metrics and business needs. It also highlights how Serverless360 extends its assured service to Manage and Monitor the Azure Event Hubs. It is a comprehensive guide to brief all the debriefed queries related to Event Hub and provides useful links to explore further. This page will be revamped frequently to stay up to date.
Explore out of box capabilities of Serverless360 for Azure Event HubsBook a Demo
We promise to never spam or give away your email.
Event Hub Overview
The fundamental of any business application is the communication between its processes. As a business application has different processes involved, it is necessary to choose the right form in which the communication is exchanged between them. The foremost action is to analyse whether the communication is via messages or events.
Going forward in this article we would focus on the Azure Event Hubs which defines the business workflow.
Messages Vs Events
A business application might opt for events for certain processes and move to message for other processes. To set the required communication, it is necessary to analyse the application’s architecture and its use cases.
Messages are raw data that toggles between the sender and receiver processes in the application. It contains the data itself, not just the reference. The sender process expects the receiver process to process the message content in a specific way.
Events are less complex than messages and frequently used for broadcast communications. It is a notification that indicates any occurrence or action that takes place in the process.
In general, the use of a message or an event can be determined by answering the question “Does the sender expect any particular processing to be done in the communication by the receiver?”>
If yes,then move to message or pick up an event if it is no.
Azure offers two categories based on the communication forms, message-based delivery and event-based communication. The message-based delivery includes,
- Azure Service Bus
- Azure Storage queues
Event-based communication technologies are,
- Event Grid
- Event Hub
Value of event-based technology in Business
Azure Event hubs and Event Grid are used in business orchestrations for event storage and handling. Let’s take a simple Vehicle availability management scenario for better understanding.
In the orchestration above, the event hub capture feature automatically writes batches of captured events into the Azure Storage Blob containers and enables timely batch-oriented processing of events.
The Event Hubs emits an event to the Event Grid when the capture file is created. These events are not strongly correlated and don’t require processing in batches. Hence, Event Grid is selected to provide a reliable event delivery at massive scale.
Event Grid delivers the event to the Azure Relay which securely exposes the service that runs in the corporate network to the public cloud. Therefore, the actual business logic to process the telematics data in the storage blob container for decision-making analytics resides in the on-premise service.
Azure Event Grid Vs Event Hub
As discussed before, an event is a lightweight communication method. To make the best use of it Azure brings in two event-based technologies. Let’s see the contrasted capabilities showcased by the Event Grid and Event Hub with real-time scenarios that fit in.
Azure Event Hub
Event Hub is a data ingestion service which streams a huge count of messages from any source to provide an immediate response to business challenges. The event Hub comes into play when handling of events along with data is required. Unlike Event Grids, Event Hubs perform certain additional tasks apart from just being an event broadcaster.
Azure Event Grid
Event Grid is an event-based technology which allows the publisher to inform the consumer regarding any status change. It is an event routing service. This service acts as a connector to tightly bind all the applications together and routes the event messages from any source to any destination.
Real time Scenario
Consider a complete e-commerce application where various processing scenarios like ordering, telemetry, shipping etc are compiled together. Every individual scenario will have a huge number of events and data streaming in it. Let us consider two scenarios which will make the best use of the Event Hub and Event Grid.
Starting with telemetry scenario, where you want to capture the telemetry data flowing out from the application. In such cases to ingest the data emitted along with events processed it is wise to opt for an Event Hub.
While dealing with shipping and transporting, it is adequate to know the status of the action being performed. In any e-commerce application, it is must to keep the consumer notified whether the product has been manufactured, despatched and shipped. Event Grid plays a vital role in these cases, where a notification on an event is enough.
Event Grid is designed for one event at a time delivery and Event Hub is crafted to deliver a large stream of events along with data.
Followed by the segregation in the form of communication it is vital to have a piece of knowledge on the terminologies and concepts related to it.
The namespace provides a dedicated scoping container, where you can frame multiple event hubs.
Any resource that sends data to an event hub is a publisher. Events are published by using HTTPs or AMQP protocol. In case of low volume, publishing HTTPs will be of use but for high volume publishing, AMQP will be of best use as it inhibits better performance, latency and throughput. Event publishers will utilize the Shared Access Signature (SAS) token for identification. It can hold a unique identity or avail a common SAS token depending on the business requirements. Events can be published as a single event or grouped event, but a single publication’s maximum size limit is 265kb. On exceeding the maximum size limit the event would end up in an exception.
Event Hubs Capture permits automatic capturing of the data published by the Event Hubs and conserve it either in Blob Storage account or in an Azure Data Lake Service account. Capture can be enabled right from the Azure portal by defining the size and time interval to perform the capture action. By using the event hub capture the desired storage account or container to store the captured data can be specified. Event Hub capture reduces the complexity of loading the data and allows to focus on data processing.
Event Hub splits up the streaming data into partitions in which the consumer or receiver will be restricted only to a specific subset or partition of the data. The storage and retrieval of the data are different using the partitions, unlike Service Bus queues and topics. In Service Bus queues and topics each consumer reads from the single queue lane. It is modelled based on “Competing Consumer” pattern where the single queue lane method will end up in scale limits. In contrast to the Service bus modelling, Event Hubs is based on the “Partitioned Consumer” pattern where multiple lanes are assigned which boosts up the scaling.
Another added advantage in Event Hub is, the data in the partition stays even after it is read by the consumer, whereas in Service Bus queues the message is removed once it is been read. The events in the partition cannot be removed manually it gets automatically deleted after the specified period expires.
Shared Access Signature in a namespace is considered as an access token to grant limited access to the namespace. The access is purely based on the authorization rules, which are configured either on namespace or an entity. The authorization policy supports the following activities,
- Send - provides access to send messages to the application
- Listen - permits the listen or receive capability
- Manage – gives the right to manage the entity
Any application that read the data from the event hub is a consumer. Multiple consumers can be allocated for the same event hub, they can read the same partition data at their tempo. Event hub consumers are connected through AMQP channels, this makes data availability easier for clients. This is significant for scalability and avoids unnecessary load on the application.
Requirements vary depending on the consumer, to cope up with this, event hub provides consumer groups. Consumer groups are the effortless view of the entire data in the event hub. Data in the event hub can be accessed only via consumer groups, partitions cannot be accessed directly. A default consumer group is created right at the stage of event hub creation.
The W’s of Event Hubs
The W’s section emphasizes on three pillar actions “Why, When and Where” Event Hubs comes into play.
- Event Hubs allows you to raise a data pipeline capable of processing a huge number of events per second with low latency.
- It can process data from parallel sources and connect them to different infrastructures and services.
- It supports repeated replay of stored data.
- To validate many publishers and to save the events in a Blob Storage or Data Lake.
- When you want to get timely insights on the business application.
- To obtain reliable messaging or flexibility for Big Data Applications.
- For seamless integration with data and analytics services to create big data pipeline.
- Anomaly Detection
- Application Logging
- Archiving data
- Telemetry processing
- Live Dashboarding
These 3 W’s will incorporate the need for an event hub and its necessity in any business application. It is a compilation of the characteristics /properties of an Azure Event Hub.
Event Hubs is developed across platforms with the support of various languages with a broad ecosystem available for the languages such as,
With the support of these languages, you can easily start processing your stream from event hubs. All supported client languages provide low-level integration. To have a regular update on the Latest SDK’s for event processing please check out (Microsoft docs link).
Event Hubs and Stream Analytics can provide an end-to-end solution for real-time analytics. Event Hubs can feed events into Azure in real-time, Stream Analytics jobs can process those events in real-time. For example, you can send web clicks, sensor readings, or online log events to Event Hubs. You can then create Stream Analytics jobs to use Event Hubs as the input data streams for real-time filtering, aggregating, and correlation.
The downfall is a common pitch faced by any cloud provider. Similarly, Azure regions or datacentres encounters a downfall if no availability zones are used. Due to the fall in datacentres, data processing gets affected and switching the processing into a completely different region or datacentre seems critical. To overcome the disrupt Geo-disaster recovery and Geo-replication comes into action, which are important features for any enterprise. Azure Event hubs support both geo-disaster recovery and geo-replication at the same namespace level.
The Geo-disaster recovery is feasible only for standard and dedicated SKUs. The Geo-disaster recovery is feasible only for standard and dedicated SKUs.
Outages and Disasters
Outages are short term unavailability of Event hubs and can affect some components or even the entire datacentre. For example, an outage can be a power failure or network issues in datacentre this does not bring in any loss of messages or events.
Disaster is a long-term loss of an event hub which might result in permanent loss of some messages or events. It is more like an earthquake or flood, the affected region or datacentre may or may not be unavailable or may be down for hours or days.
The disaster recovery feature is reliant on primary and secondary recovery namespaces and imposes metadata disaster recovery.
The name of the disaster recovery configuration that you set up. The connection is done through the alias and no connection string changes are required.
Primary Namespace is one among the namespaces that correspond to the alias. It is “active” and receives messages.
In contrast with primary namespace, a secondary namespace is “passive” and does not receive messages.
Metadata is the replication of entities and their settings associated with the namespace. Messages and events are not replicated. The metadata between primary and secondary namespaces is in sync, so both can easily accept messages without the use of any application code or connection string changes.
The process of initiating the secondary namespace.
Setup and failover flow
In setup, primary and secondary namespaces are created, and pairing occurs between them. The pairing results in an alias which replaces the use of connection string. Failover pairing can only accept new namespaces. Monitoring is required to detect if a failover is essential.
Certain steps are essential to initiate a failover,
- In the case of another outage, it is necessary to failover again. It is advisable to set up another passive namespace and update the pairing.
- Once the messages are recovered pull them from the primary namespace and use that namespace outside of the geo-recovery set up or delete the old primary namespace.
Security is one of the important parameters in any resource. Event hubs provide security at two levels,
Every request to a secure resource must be authorized to ensure that the client has required permissions to access the data. Azure event hubs provide few options for authorizing access through,
- Azure Active Directory
- Shared Access Signature
Authentication is the process of validating and verifying the identity of a user or process. The user authentication ensures that the individual is recognised by the Azure platform. User verification can be done by,
- Authenticating with Azure Active Directory
- Authenticating with Shared Access Signature
Security control is a characteristic of an Azure resource that inculcates the ability to prevent, observe and respond to security defects. Azure Event Hubs possess few security controls in various perspectives,
|Network||Service endpoint support
Network isolation and fire walling support
|Monitoring and |
|Azure monitoring support
Control and management plane logging and audit
Data plane logging and audit
|Data Protection||Microsoft-managed keys
Encryption in transit
API calls encryption
|Configuration management support|
Advanced Message Queuing Protocol 1.0 is a transfer protocol used to securely transfer messages between two bodies. It acts as the primary protocol for both service bus and event hubs, these resources also support HTTPS.
Manage Event Hub in Azure Portal
Event Hub is an ingestion service which deals with millions of events per second and managing such a massive count is a complex process. Azure comes up with effective event hub management capabilities like,
- Enable Capture
Let us consider a Power/Energy company, building an infrastructure where delivering the maximum throughput requested by the customer becomes critical. This can be rectified easily with the help of concepts like auto scaling, where the enough infrastructure can be ingested to the company and can be scaled down later.
Scaling of Event Hubs is done with the Throughput Units which is the throughput capacity of the event hubs. A single throughput unit includes,
- Ingress: Up to 1 MB per second or 1000 events per second
- Egress: Up to 2 MB per second
These values of throughput units are adequate when there is predictable usage in the infrastructure. When the situation arises to have more data transfer through the Event Hubs, customers usually increase the number of throughput units manually. This manual intervention can be neglected by using the Azure Event Hubs: Auto-Inflate functionality.
Azure event hubs: Auto Inflate is a cost-effective functionality offered by Microsoft that provides control based on the usage and demands without any changes in the existing setup. It enables users to automatically scale up throughput units to meet the demands.
The Auto-Inflate functionality can be enabled during the deployment of event hub and the Auto-Inflate Maximum Throughput Units can also be determined.
Capture can be enabled right during the creation of the event hub and the captured events are stored in the Azure Blob Storage or Azure Data Lake Store account. Initiating a capture is an easy process, it does not involve any administrative cost to run and scales paralelly with throughput units.
It is also possible to enable the capture from the “Capture” option available in the left blade under the Features section.
By enabling the capture option , a window appears where the storage details (Azure Storage Blob or Data Lake Store) are to be mentioned.
Monitor Event Hub in Azure Portal
Event Hubs can be monitored by providing access to certain event hub metrics in the Azure monitor. The overall status of the event hub can be assessed at the namespace level and in entity level. This is done by the valuable set of metrics data possessed by the Azure Monitor.
Azure services are monitored with the help of combined user interfaces provided by the Azure Monitor. For complete documentation on Azure Monitor, verify Azure Monitor overview – Azure Monitor | Microsoft Docs.
How to access the metrics?
The metrics can be accessed either through the Azure portal or using APIs and Log Analytics. Metrics are enabled by default and most recent data of 30 days can be accessed or to retain data for a longer duration, the metrics data can be archived to an Azure Storage account.
Metrics can be monitored straight from the Azure portal directly from the namespace level just by clicking on the metrics option under monitoring section available in the left blade.
To bring in the required metrics for monitoring, specify the desired metric namespace and select from the metrics filtered to the scope of that event hub.
The metrics are assigned based on various perspectives to attain a complete monitoring. The metrics are,
- Request Metrics
- Throughput Metrics
- Message Metrics
- Connection Metrics
- Event Hubs Capture Metrics
- Metrics dimension
The detailed description on the above mentioned metrics is briefed in, Metrics in Azure Monitor – Azure Event Hubs | Microsoft Docs.
Diagnostic Logs is configured to have an overall view of very task that is happening. It covers activities that takes place from the time of creation till deletion and even the updates and activities during the run time.
A diagnostic setting specifies a list of categories to perform logs and metrics to be collected from the resource and one or more destinations to stream them. Event Hub enables diagnostic logs for two categories
- Archive Logs: Logs that are associated to Event Hubs archives.
- Operational Logs: Logs that provides information on all the activities (creation, deletion and updating) that takes place in the event hub.
The step by step guide to enable diagnostics logs and an illustration on the diagnostics logs schema, Set up diagnostic logs – Azure Event Hub | Microsoft Docs.
Manage Event Hub in Severless360
Managing the activities in a service like Event Hub, which deals with millions of events requires a tool with outstanding management and monitoring capabilities. Serverless360 comes into picture to fulfil all the complex management needs required by the Azure services. To know more about Serverless360 Sign up for 15 day free trial..
The Vehicle Management scenario exhibits how powerful the Event Hubs are, but the real challenge is to manage it in Azure portal. There are certain challenges faced by Azure users to manage and monitor the event hubs in the Azure portal.
- Lack of deeper/integrated tooling
- No consolidated monitoring
- No dead-letter event processing in Event Grid
- No Auditing
To overcome all these challenges Serverless360 has come up with extensive set of capabilities.
In certain business application there might be a requirement like importing an Event Hub from Non-production Namespace to Production Namespace. There is no straight-forward solution in Azure portal to achieve this. Serverless360 solves this with the help of import entities option from the namespaces tab available in the Serverless360 portal. With this capability, the user can import their Event Hub entities from Non-production to Production Namespace in a single click.
It is also possible to manage the shared access policy for Azure Event Hubs, view the properties of the Azure Event Hubs and Event Grids and create consumer group for Event Hubs.
When there is a need to test our business orchestration, we need to send some events to Event Hubs so that event will send to the Endpoint through the subscription. But in the Azure portal, it is not possible to test the orchestration. To solve this challenge, Serverless360 brought in the capabilities called Activities with which user will be able to send events to both Event and Event Grid Topic. It is also possible to schedule these Activities too.
Read more on this feature here .
Monitoring Azure Event Hub in Serverless360
Azure provides the entity level monitoring on their metrics, but the actual need would be Consolidated monitoring at the application level.
For monitoring Azure Event Hubs in multiple perspectives, Serverless360 has three types of monitors: Status Monitor, Threshold monitor and Data monitor.
Choose Serverless360 status monitor to get application health reports at a specified time in a day representing the state of Azure Event Hubs against the desired values of its state. It is also possible to monitor the partition of the Azure Event Hubs.
Read more on this feature here .
Monitor your Azure Event Hubs when their state violates desired values for a specified period, say few seconds/minutes. For Event Hubs, it is possible to monitor the status and size of the partitions in Threshold monitor.
Read more on this feature here .
There would be needed to monitor the performance of the Event Hubs, Count of events in the Capture of the Event Hubs. To monitor the entities on their extensive set of metrics, Serverless360 has brought in the capability called data monitor which provides the calendric view of historical alerts.
Read more on Data Monitor feature here .
Serverless360 provides Comprehensive way to manage and monitor your Azure Serverless Components. Read more on this Azure Event Hubs feature here .
Event Stream Processing
GLAS Smart Thermostat by Johnson Controls, uses Azure Serverless to connect their room thermostat with customer’s handheld devices. The GLAS Smart Thermostat monitors to report the indoor and outdoor air quality so that you can manage the air you breathe. Initially, they were using the VM’s to connect and process the data between the thermostat and the customer’s devices. Considering their vertical growth in the expanding consumer market they made this paradigm shift to Serverless.
The team tested this scenario with a simulation of 20,000 IoT devices besides the real devices
In this architecture, you can see four different activities, all controlled by different Azure Serverless components like Azure Event Hubs, Functions, Event Grid, Cosmos DB, and API Management. All the communication between the thermostat, Azure and the customer device is sent through the IoT Hub. As a first step when the customer installs a thermostat, JCI GLAS device sends a message to the IoT Hub to register the device. This event triggers the Functions app to save the data in Cosmos DB. The function app also raises an event into the Event Grid to trigger events like the registration success message to the customer mobile devices.
The second step is to pair both the JCI GLAS device and the customer mobile device. Again, the IoT hub triggers a pairing service to the Event Grid through the pairing Event Hub and Functions app.
Finally, the user sets the temperature range in which the device should trigger a message, this setting is forwarded into Azure through the IoT Hub. The Event Hub not only sends the settings into the Cosmos DB through Functions but also triggers an event into the Event Grid and integrate with SignalR services to update the user mobile device.
The next service is to send a message from Azure into the GLAS device, this message is normally about the local weather update from the internet. In this case, the weather update is raised as an event into the Event Grid and triggers the Azure Functions to check the Cache to check if the settings are up to date and sends the message to the GLAS device through the IoT Hub.
Here you can see that all the messages are going through 1 IoT Hub to 3 Event Hubs based on the message type. There are cases where the message from IoT, directly triggering the Azure Functions and the message is segregated through the if/else coding. But given the routing mechanism and throttling in Event Hubs, it makes sense to use Event Hubs.
Explore more on the interesting frequently asked questions on Azure Event Hubs at, Frequently asked questions – Azure Event Hubs | Microsoft Docs .
Documentation and Webinars
This Webinar covers in detail the capabilities of Serverless360 to facilitate managing and monitoring Azure Event Hubs. All concepts are dealt with real-world business use cases for better understanding.
This content will be continuously revamped to stay up to date.
Do you want to reprocess dead-lettered messages in Azure Service Bus? Explore Serverless360!Try for free
Free 15-day trial – No credit card – Easy setup