Monitoring Microsoft Flow Using Serverless360 BAM

Introduction

Business Activity Monitoring in Serverless360 allows a user to track the business process that is executed in Microsoft Flow to solve a particular business need. This blog will help you understand how Serverless360 can be used to track and monitor a business flow defined in Microsoft Flow.

Business Process Management

Business process management (BPM) is getting more popular as businesses are moving towards a faster, simpler, and more automated environment. BPM is a discipline that aims to optimize, enhance, accelerate, and automate business processes.

This can be done through automating workflows using workflow automation software or business process management software (BPMS). Microsoft Flow is one such tool, It is now called “Microsoft Power Automate”.

Microsoft Flow

Microsoft Flow is a task or process automation tool that allows us to integrate different services and applications. Though most of the applications that can be used with Microsoft Flows are cloud-based, Microsoft Flows can also be used with the on-premises environment. The different types of Microsoft Flows are,

Automated Flows

A flow that can be triggered automatically by an event, like an email arriving or a file changing.

Instant flows

These are triggered manually by a button press.

Scheduled flows

A flow that runs at a set time, either once or as a recurring action.

Business process flows

Ensure that everyone in your company follows the same process by defining a set of steps for people to follow.

UI Flows

Record clicks and keyboard strokes from your desktop and web applications and automate the playback of manual steps on legacy software.

Terms in Microsoft Flow

Triggers

Triggers are the events that kick-start the actions in a Microsoft Flow. The trigger can be a Message arriving in Service Bus Queue or Topic Subscription or a button press or event sent to an Event Grid Topic or it can be a timer trigger.

Actions

Actions can be defined as the stages or separate steps created in the Microsoft Flow. The actions can be either the actions from connectors published by various providers or default connectors provided by Microsoft Flow. An example of a Flow action can be sending a message in Microsoft teams. Apart from the connectors available in Power automate, we can create and use our custom connectors.

Order Validation Scenario

Let us consider an order processing scenario, where the orders placed to an e-commerce site must be validated and only the valid orders must be sent for further processing. The orders are sent to Microsoft Flow through Service Bus Queues with the order details in the message body. The Microsoft Flow validates whether the order has all the required details and pushes the valid orders to another Service Bus Queue for further processing.

The completely designed Microsoft Flow to implement this scenario will be as follows

Tracking the Orders in Microsoft Flow

All the orders that are validated by the Microsoft Flow can be tracked by viewing the run history of the Microsoft Flow.

On selecting a specific run in run history, it is possible to view the status of each action in that run and the input and output messages of each state.

The run histories displayed are more from an architecture perspective rather than a business perspective. It will be very difficult for a support user to diagnose any failures in the business flow as the tracked data is more technical and it is not easy to view the business-specific data.

These difficulties can be solved by using Serverless360 BAM.

Serverless360 BAM

In any kind of integration solution, there is a need for the organization to build an end-to-end traceability/monitoring solution that can be used by the business users. Often this is an afterthought and the operational support team struggles to run the solution that has been implemented. Sometimes the delivery team will work on a solution but often it is custom-made. It will be difficult to use across projects and teams spend about 20-30% of their time addressing this challenge like logging components, and web dashboards. Most of the time the solution will be premature, since building an enterprise-grade tracking/monitoring solution requires enormous effort.

That’s exactly the challenge we are tackling in Serverless360 with the BAM feature.

Business Activity Monitoring

It is a functional end to end business activity tracking and monitoring product for Cloud Scenarios. With Serverless360 BAM, you can get full visibility of your end-to-end business process flow across your Azure resources.

Terms in Serverless360 BAM

1. Business Process – A collection of 1 or more related business transactions
2. Transaction – An operation of the business which represents a set of related business activities or actions
3. Stage – An activity or action within the transaction. A transaction can contain 1 or more stages.

The below diagram explains the business terms used in BAM.

How to Integrate Serverless360 BAM with Business Flow

Serverless360 BAM provides the following components which can be plugged into your business flow to track the transactions.

• Custom connectors for Logic Apps and Microsoft Flow
• .NET Library for Azure Functions and OnPrem APIs

Creating a Business Process in Serverless360

The prerequisite for creating a Business Process is to have a Serverless360 account. You can create an account in Serverless360 here . Once the account is created, you can create a Business Process by following the steps below.

1. Associate an Azure SQL Database connection string and Azure Storage Account connection string in the BAM section of Serverless360. The SQL Database and Storage account will be used to store all your tracked data and messages.
2. Create a Business Process by providing a friendly name and the description.
3. Create a Business Transaction within the created business process. In our case, it should be order validation.
4. Once the Business Transaction is created, the business flow can be designed by adding the required stages in the transaction. Each stage should represent the activities in the business flow.

Tracking Properties

The business data flowing in Microsoft Flow can also be tracked at each stage. For example, we can track the address specified in the order, order number, etc. This can be done by configuring the property name and the path of the property in the flowing message.

The properties can either the part of the message body or the message headers passed to the BAM connector actions or .NET library.

The transaction configuration for the order validation scenario must look like the below illustration.

Integrating Serverless360 BAM with Your Microsoft Flow

To integrate Serverless360 BAM with your Microsoft Flow, follow the below steps.

1. Download the Swagger API file from the Business Process section. You can find this by clicking the BAM storage details button.
2. Create a custom connector in your Microsoft Flow account and upload the downloaded swagger file.
3. You can get the Host connector at the Access keys menu in the Settings section. Copy the function app URL from the BAM connection Details tab.
4. Save the connector.
5. Now, this connector can be plugged into your Microsoft Flow.
6. You can get the key for API connection from the same BAM connection details tab in Access Keys section.
7. The Start Transaction Action must be placed at the beginning of the Microsoft Flow that has to be tracked.
8. This can be followed by the checkpoint actions that are provided in the transaction instance-id returned by the start transaction action.
9. The last checkpoint of the transaction must be set with true for the “Is Transaction Complete” flag.
10. If you want to view the message passed to any of the checkpoints, you can set the “Archive Message” flag to true.
11. The status of any stage is defined as either of the following: InProgress, success, or failure.
12. Check Point with correlation action can be used whenever there is a need to correlate transactions. The correlation can be done using the tracked properties.
13. It is also possible to log exceptions at any stage by just passing the exception code and exception message to the connector actions.

The fully configured Microsoft Flow with Serverless360 BAM should look like the below illustration.

A start transaction checkpoint of Serverless360 BAM custom connector used in a Microsoft Flow must look like the below illustration.

Once the Microsoft Flow is configured with Serverless360 BAM connectors, all the tracked transactions can be viewed at the Serverless360 BAM Tracking section.

On selecting a particular transaction instance, it is possible to view the status of each stage and the properties tracked in each stage.

User-Friendly Querying

Tracked Transactions can be queried for specific results based on the transaction status and tracked properties in a much user-friendly manner.

The queries also support LIKE operator and we can also group multiple search query clauses using AND – OR Operators which helps us retrieve more focussed records.

For example, if want to retrieve only the orders from Chennai, I can use the Query City = “Chennai” which returns only the required records.

Business Activity Monitoring Dashboards

We can also set up dashboards to visualize the trend of our business process. We can create multiple widgets with different queries. This helps in having a quick picture of the health of Microsoft Flow. It also helps in understanding our business by viewing the transactions grouped by tracked properties. For example, from the below image, I can see that I am getting orders from all the cities in a uniform manner. This kind of business dashboards can be created in Serverless360 BAM.

Monitoring

We can also set up Serverless360 Monitors to monitor the transactions occurring in a flow. There are two types of Monitors available for BAM. They are as follows

1. Query Monitors
2. Exception Monitors

Query Monitors

Query Monitors can be used to monitor the transactions by periodically querying the transactions with the configured query and validating the results against the configured threshold error and warning values. Once the configured error or warning thresholds are not met the alert will be triggered. The monitoring results will be stored for 1 month for future references.

Exception Monitoring

Users can also get instant alerts in the configured notification channels whenever there is an exception is logged in the stages of a business transaction.

Serverless360 provides integration with the following notification channels.

1. SMTP
2. Slack
3. ServiceNow
4. Microsoft Teams
5. Custom Webhooks
6. Operation Management Suite (OMS)
7. Pager Duty

Reprocessing

It is also possible to reprocess the failed or any other required transactions. Serverless360 provides 4 types of reprocessing endpoints. They are as follows.

1. Http Endpoints
2. Service Bus Queue
3. Service Bus Topic
4. Event Grid Topic

The message body archived at the selected stage will be sent to the configured reprocess endpoints. This can be used to reprocess the failed instances by investigating the reason for failure at a later point in time.

Advantages of Using Serverless360 BAM for Microsoft Flow

1. Tracked data can be viewed in a business point of view rather than the technical point of view
2. It is possible to track the properties at different stages in Microsoft Flow
3. It is possible to filter or search the transaction instances based on the configured properties in addition to the status of the runs or transaction instances
4. It is possible to get alerted whenever there is an exception logged in the stages of the Microsoft Flow
5. It is possible to set up a custom user access policy so that users having only required privileges can access the message content tracked in the Microsoft Flow
6. We can set up a customized dashboard to monitor the transactions happening in a Microsoft Flow
7. It is easier to find the exact stage where the transaction instance has failed and fix it with minimal cost
8. It is also possible to track the stages that constitute the same business flow but is present in a different Microsoft Flow or a completely different infrastructure using the Checkpoint with Correlation action of BAM

Summary

Business Activity Monitoring in Serverless360 can be used to track the flow of data and the actions happening in Microsoft Flow. It can be used to find and fix the issues happening in a Microsoft Flow in a much easier and an efficient manner. It also reduces the efforts involved in monitoring the transactions happening in Microsoft Flow.

You can also take a look at our pillar page dedicated to Distributed tracing to get better insights on the topic.