Tags :  

End to End Tracking for Azure Integration Solutions

Last updated on: August 5, 2021

In any integration solution, when many Azure services come together to communicate with each other, there is a need for the organizations to build an end-to-end traceability/monitoring solution that can be used by the business users and IT support.

With Serverless360 BAM, customers get full visibility of their end-to-end business process flow across their Azure resources.

Perform end-to-end message tracking and capturing necessary data on business process flow across Azure resources and hybrid integrations.

Serverless360 BAM

Consider an organisation that has a well-defined Employee Onboarding process. In this case, the organisation should build a hybrid application consuming Azure resources like APIM, Azure Functions, Custom APIs, and Logic Apps. This is inevitable as the organisation is planning to move their application partly to cloud and retain a legacy processing on-premises.

Employee Onboarding process

As the application involves Logic Apps and multiple other Azure resource types, it is impossible to achieve the end-to-end correlation using the default correlation id feature in Logic Apps.

We are now in need of a technology-agnostic solution that can seamlessly correlate stages in a business workflow irrespective of which component executes it.

 It is also necessary to provide the Business Users with the right toolset to locate “What happened with onboarding a specific employee?”

Serverless360 Business Activity Monitoring (BAM) provides a simplified business-focused view of a complex underlying system to fulfill the business transaction.

mid cta

Why Serverless360 BAM? Why not a custom solution?

In any complex hybrid integration solution, there is a need for the organisation to build an end-to-end traceability/monitoring solution that business users can use. This tracking solution is often an afterthought, and the operational support team struggles to run the implemented solution. Sometimes the delivery team will work on a solution, but often it is bespoke and challenging to use across projects, and teams spend about 20-30% of their time addressing these challenges, building custom solutions like logging components and web dashboards. The solution will be premature since creating an enterprise-grade tracking/monitoring solution requires enormous effort.

That’s precisely the challenge we are addressing through Serverless360 BAM.

Get started with Serverless360 BAM

The first step in starting with BAM is configuring the storage details by associating your Azure SQL Database and an Azure Storage Account. The SQL database to capture the tracked transaction details and the messages flowing through the business application stages would reside in the Storage Account as a blob. Successful association of storage details will trigger the deployment of specific tables in the associated Azure SQL database. 

Let us consider the employee onboarding scenario itself as an example to configure further.

mid cta

Defining the Business Process

The business process in Serverless360 represents workflow in your business. For example, in the Employee Onboarding system, new employee onboarding can be a business process. This process will involve ‘transactions’ which in turn will have multiple stages. BAM allows you to define the business process to depict your workflow best. Each ‘stage’ in the transaction will get executed in one of the underlying business components. Irrespective of which component performs this stage, a Logic App or Azure Function or a custom API, Serverless360 BAM facilitates instrumenting them.

At this point, focus on defining the Business Process by mapping the stages against the required checkpoints in the business workflow. Below is the depiction of the business process in Serverless360 BAM for our Employee Onboarding scenario.

We recommend configuring the stages in the transaction to capture essential properties from the messages flowing through this component. These captured properties will enable searching for a specific transaction. For example, when the EmployeeId is a property, it becomes feasible to fetch the onboarding status of that employee. The scope of the defined property can be either limited to that specific stage or shared across stages. The former scope will consider it as a stage property, and the latter as a global property.

Instrumenting the Business Application

The next step is to instrument your business process to enable tracking. In this Employee onboard scenario, the first stage occurs in a Logic App; the subsequent stages occur in Azure Functions, APIM and custom APIs. Serverless360 BAM facilitates this instrumentation. 

BAM also correlates the stages even when they spread across hybrid integration services. Instrumenting these connectors is straightforward through checkpointing the occurrences of events in these stages.

There are 3 BAM connectors:

  • Start Transaction to initiate the transaction. This connector will generate a unique transaction instance-id that will be carried forward to the subsequent stages.
  • Checkpoint marks a stage. 
  • Checkpoint with Correlation is exclusively for scenarios like our Employee Onboarding case where there is a need to correlate the stages across components using the transaction instance-id that identifies the transaction.

To get BAM up and running, the user needs to perform relatively much less instrumentation into the business application compared to custom built, logging, or tracking solutions, BAM takes care of all security and scalability aspects as well.

Achieving the business values

End-to-end tracking

Once the instrumentation is complete, Serverless360 BAM can track the message flowing through various components in the business process. In this way, your business users and support operators get access to a convenient business workflow that they are familiar with, which abstracts the underlying complexities of the business components at work and provides them with self-service opportunities.

End-to-end monitoring solution

The visual representation of message flows helps to communicate information in a self-explanatory manner.

If the user wants to find out why a particular transaction failed, they can go into the stage details and figure out the reason for the failure.

Search

It is also possible to search for information using simple queries. For example, you can fetch all the employees that belong to a VLSI unit, entering this simple query ‘BusinessUnit = “VLSI”‘ in the search field will get the job done. You can also conveniently save the queries for future access.

End-to-end monitoring solution

Reprocess

The ‘Action Required’ tab in the tracking section presents the user with a focused view of the failed transaction. If the user wants to reprocess this transaction successfully, they should modify the necessary value and resubmit.

Search

Any reprocessed transaction will have a tag to notify the user.

Reprocess

Monitoring

Monitors allow us to keep an eye on the business processes. The first is the exception monitor, which can detect any exceptions occurring within a specified duration. For example, the user would get an alert if any exceptions were detected within the last 15 minutes. BAM can send alerts on several notification channels supported by Serverless360.

End-to-end monitoring solution

The query monitor helps you track the overall performance of the business process based on a query. For example, I can choose to get a warning alert when any failure detected in the last week and a high priority error alert when the failure count exceeds five.

query monitor

Dashboard

Utilising all the tracking information, Serverless360 BAM can present the user with a meaningful dashboard that can help with strategic business decisions. Looking at this widget, the user can easily understand field transactions and take necessary action here. There is a spike in the number of field transactions, and the user can drill down further to find out the cause and rectify the issue.

BAM Dashboard

Serverless360 BAM in action

Having understood BAM in Serverless360 let us visualise how a business user or a support user can use this feature to achieve operational excellence.

  • Support user gets a failure notification on an email from Serverless360 BAM.
  • Serverless360 BAM
  • The user visits the portal to find the failed transaction in the Action Required Tab in the Tracking section of BAM.
  • Viewing the transaction details helps the user figure out the reason behind the failure.
  • End-to-end monitoring solution
  • The user modifies the necessary values and reprocesses the transaction to restore the business.
  • Serverless360 BAM

The BAM feature in Serverless360 is a cohesive tool that can help you resolve issues faster with a unified view of business transactions.

Where does my business data get saved?

With the satisfaction of enabling, you with the necessity of BAM, it is our responsibility to comfort you with the data security principles we have in place. We understand the sensitivity of the business information we deal with. Hence, be rest assured your data resides in your Azure subscription only.

Key Takeaway

Advancements in Azure services have drastically improved how we build business applications, but this brings along its own set of challenges, especially the lack of visibility provided to us on what happens under these complex systems. Businesses often come across situations where time is spent chasing missing messages or failed transactions.  Serverless360 BAM can add value to your business in 5 key areas.

  1. End to end traceability of messages flowing through the various components in your business process. 
  2. Search your transactions based on critical values familiar to you and your business, like an order ID or an invoice ID. 
  3. If a failed transaction is detected, the business user can quickly reprocess the transaction to a successful state without relying on skilled Azure personnel, thereby reducing the overall cost of ownership. 
  4. Alerting systems can notify any failures in your business process using the Tracked information.
  5. A comprehensive dashboard helps you make strategic business decisions.
mid cta
Author: Ezhilarasi Chezhiyan

Ezhilarasi Chezhiyan, Technical Lead for the product, Serverless360, at Kovai Systems India Pvt Ltd. She has over 8 years of experience in Software development with knowledge spectrum spread across various domains. Problem-solving is her passion and she believes in the quote by Albert Einstein- "Intellectual growth should commence at birth and cease only at death"