A complete guide on Azure Integration Services

Azure Integration Services (AIS) provides tools to connect apps, data, and services across cloud and on-premises environments, enabling streamlined workflows, data synchronization, process automation, and seamless communication in business operations.

If you are reading this article, you will likely have been a user of Microsoft integration technologies such as BizTalk or consider moving to Azure. In this article, we want to help you understand what this journey might look like.

History of Microsoft Integration

Typically, three major areas defined the Microsoft Integration landscape before the emergence of Azure Integration Services.

BizTalk Server

BizTalk Server has been the flagship integration platform from Microsoft for many years. BizTalk was focused on durable messaging, workflow orchestration, and connecting to line of business systems.

BizTalk required a significant learning curve and skillset to implement, but many customers had delivered great solutions for nearly two decades.

SQL Server Integration Services

SSIS was a powerful data integration and ETL (Export, Transform, Load) tool that connects to databases like SQL server, Flat files (CSV, Excel, XML), and more. It offers a visual design environment to create data integration workflows. This platform also required a specific skill set of professionals to implement.

Custom Integration development

The biggest competitor to Microsoft’s integration platforms, such as BizTalk and SSIS, was customers who would write their integration solutions from scratch using .net code or Powershell and scripting languages.

Things have changed in the Integration landscape

Since the launch of Microsoft Azure, Microsoft has been developing cloud-based integration technologies that remove some of the constraints and overhead of previous server-based integration technologies.

As the adoption of cloud technologies has grown over the last few years, many customers have adopted Microsoft cloud integration technologies, which are called Azure Integration Services.

As the Azure Integration Services have matured, many BizTalk and SSIS customers have begun migrating their Server-Based workloads to Cloud-Based technologies. This is also true for the custom-developed solutions, which can now leverage platform-level services on Azure. There are many drivers for this adoption, such as:

• Cloud offers increased flexibility and quicker time to market for solutions, which has helped initiatives such as Digital Transformation for many customers.
• There are sometimes cost efficiencies on the move to the cloud.
• It is easier to manage the skill set needs of your integration platform.

What are the different types of Integration Services in Azure?

The primary technologies within Azure Integration Services for most customers are:

1. Azure Logic Apps
2. Azure Service Bus
3. Azure API Management
4. Azure Event Grid
5. Azure Functions
6. Azure Data Factory

Workflows – Logic Apps

Logic Apps help create workflows and subsequently orchestrate business processes like order processing in e-commerce to connect hundreds of dependent services in the cloud and on-premises

Messaging – Service Bus

Service Bus is a highly reliable message bus for processing important transaction messages at a high scale with durability and publish/subscribe capabilities.

APIs – API Management

API Management – is a complete service to publish and maintain the APIs that developers can use to provide a great user experience for APIs they expose to consumers, or developers can proxy APIs they consume from other parties to provide a centralized API hub.

Events – Event Grid

Event Grid – is a publish-subscribe service that allows a publisher to send events to a topic, which can then push events to consumers at a considerable scale and via different channels.

Compute – Azure Functions

Azure Functions – is a hosting platform for code execution when you need simple logic that needs to be encapsulated in a function that can be easily consumed by other services such as Logic Apps, Event Grid, and Data Factory.

ETL Process – Data Factory

Data Factory – is mainly used to integrate and connect data sources to construct ETL and ELT processes and accelerate data movements between systems.

Unified IPaaS solution from Microsoft Azure

The power of the Microsoft Azure Integration services is that they can be combined like Lego Bricks with the rest of the services on Azure to build the Enterprise Integration Platform you need for your organization’s requirements.

The image above illustrates how various Azure Integration services, either on their own or in combination, can fulfill different integration scenarios. For example, Logic Apps are the way to go if you want to implement Enterprise Application Integration (EAI). A combination of Integration Account, Logic Apps, and Functions may be necessary when handling data transformation scenarios.

If you want to replace SSIS integration, Azure provides Data Factory and Synapse pipelines, which are ideal for ETL (Extract, Transform, Load) processing.

For custom integration development, Azure Functions can serve as a lightweight code execution platform and API management capabilities to host and manage APIs securely. You may even use a Logic App to orchestrate several functions together.

Migrating from BizTalk Server to Azure Integration Service

The current BizTalk Server users are considering migrating to Azure Integration Services due to various driving factors such as enabling modern iPaaS (SaaS-led Integration), cost reduction, lack of skilled people working with BizTalk, support lifecycles, and more.

API or SaaS-led Integration

BizTalk Server was built when APIs were less dominant than today. The ability to manage REST APIs with BizTalk Server could be challenging as we switch from a messaging approach to HTTP methods like (GET and POST) and response status codes.

In Logic Apps and APIM, you have built-in features like out-of-the-box connectors with hundreds of APIs to connect with different systems with easy authentication methods like OAuth designed to support API and SaaS-led integrations efficiently.

Skills

For several years now, there has been a gradual migration of people who used to work with BizTalk all the time towards using Azure. These Integration specialists have experience across different systems and technologies, which makes them a good fit for architecture roles and development.

Also, due to the smaller learning curve, even non-integration professionals can participate in building part of the integrations. For example, we could see .Net developers learn Logic App skills and become productive quickly.

Cost Reduction and Scalability

Some organizations consider migrating to AIS for a perceived view of reducing cost in the long run or see immediate savings of up to 20% to 30%. One of the critical areas to get better ROI is when there are load bursts.

With BizTalk, there were good load leveling patterns, which are present in Azure too. Still, if you want to temporarily add additional BizTalk servers to your BizTalk group to increase your processing power, you need additional hardware and licenses. You couldn’t just rent an additional 10 licenses for a few hours.

With Azure Integration Services, you have several resources like Logic Apps Standard, Functions Elastic Premium, and API Management, which are easy to scale.

Support Lifecycle

Since the support for BizTalk Server 2020 concludes by November 2029, organizations are contemplating continuing BizTalk’s support. Given Microsoft’s focus on Azure Integration Services, which is receiving significant investment, these organizations are engaging in discussions about formulating migration plans—hybrid or cloud-native—despite their workloads functioning effectively on BizTalk Server.

Suppose you want to read completely about the value of migrating from BizTalk to Azure Integrations. Download this whitepaper.

The table below shows how the BizTalk capabilities map to the Azure iPaaS services.

A real-world example of Azure Integration Services

Let’s look at the Azure Integration Services example using the following use case scenario:

Use Case: E-commerce order processing

Scenario: Consider an e-commerce scenario where a Northwind company wants to optimize its order processing workflow by integrating various systems involved in the process. The company’s primary goal is to reduce the manual intervention of the employees and provide a seamless experience with enhanced operational efficiency.

1. A customer places an order on the e-commerce website,
2. The website publishes the order to a message queue for processing.
3. The order is picked up from the queue by a Logic App for processing.
4. The Logic App uses APIs exposed by API Management to check for product availability in the inventory system.
5. The Logic App processes a payment via the Payment Gateway.
6. The Logic App updates the shipping system.
7. The Logic App then publishes a message to a queue for any other systems that need to know about the successfully processed order.
8. A Function will process a message from the queue to update the Web Applications database to update the order status and any future shipping events.

The solution that is implemented will look like the below diagram.

Microsoft Azure Integration Services used

Considering the technologies used in this architecture, we would consider the following items.

Azure Logic Apps: Logic Apps are used to automate workflows to trigger actions at different stages of the order process, such as order placement, payment confirmation, and shipping status updates. For instance, when a customer places an order, Logic Apps might automatically check the inventory system, payment processing system, and order fulfillment processes.

Azure Service Bus: Utilize Service Bus to provide durable messaging between the e-commerce platform, inventory management system, payment gateway, and shipping provider. This ensures that order updates are reliably communicated to all relevant systems.

Azure API Management: Expose APIs for third-party payment gateways and shipping providers to integrate seamlessly into the order processing flow. This allows for a smooth interaction with external services while maintaining security and governance.

Azure Functions: Azure Functions provides near real-time compute service that updates the order value and status in the database.

Why end-to-end business process tracking is essential in Azure Integration Services?

In any integration scenario, if a message fails to arrive from point A to Point B, it becomes essential to isolate the failure point and investigate the root cause.

In the above real-world use case, consider a message that failed at the API Management point as the third-party system with which it interacts was down at the given time. Imagine a business user (Logistics Manager) responsible for seamless supply chain management operations who wants to know why a particular order has not yet been processed. They might need to send a support ticket to the IT department and wait for the technical team to return.

Instead, what if every stakeholder has the power to access and track the message or order flow relevant to their specific departments? Just think about the potential time savings for the Integration team when it comes to addressing these business inquiries. Data and Business Process democratization is an approach that promotes transparency by securely sharing relevant data for quick decision-making.

How can we achieve business process tracking with Azure Integration Services?

Azure Integration Services includes Log Analytics and Application Insights, which can accept telemetry of logs from components like Logic Apps and Azure Functions. This can allow you to create views of distributed tracing, which will help your developers troubleshoot what is happening across your architecture. This is powerful during deep dive troubleshooting performed by a developer.

Many organizations need something digestible by an IT Support Operator or Business Super User, as we saw in the above scenario, so they can do self-service tracking and diagnostics without needing a developer. In a nutshell, App Insights could not be a better option for business process tracking.

What’s the difference between business process tracking and Logging?

The key thing to understand is that Logging and Business Activity Monitoring are different.

Logging involves capturing telemetry events within your solution, providing details about when a code is executed, and offering contextual metadata.

On the other hand, Business Activity Monitoring focuses on establishing expectations for business transactions and linking telemetry data to those transactions. This approach enables you to shift from technical telemetry to business telemetry, allowing you to monitor the progress and status of your transactions more effectively.

Some operational challenges and limitations need to be carefully assessed;

1. Application insight is built with developers in mind, which is not ideal for business users.
2. The Azure Portal is too complex for a non-expert user, and security concerns can be associated with inexperience and too much access.
3. App Insights often uses sampling and is a lossy telemetry pipeline not intended for transactional consistency.
4. Complete flow tracking is impossible if you have a hybrid environment, both BizTalk and Azure.
5. Business users won’t be able to get desired insights from the logged data as it requires complex KQL query knowledge.

Read More

• Application Insights for Logic App data tracking
• Log Analytics for Logic Apps data tracking

What is the ideal solution for BAM or business process tracking in Azure?

Serverless360 BAM is purpose-built to achieve business activity monitoring and act as a self-service portal for business users. You don’t necessarily need to provide the business users access to the Azure portal.

Some of the key features of BAM

1. Define business transaction expectations and track the data flow
2. Supports hybrid scenarios, both Azure and BizTalk
3. Monitor any exceptions happening in the integration flow
4. Track exactly where your message has failed
5. Ensure that the message reaches the destination on time
6. Make it easy and user-friendly for non-developers and integration experts

Related Reading

• Servereless360 BAM for business process tracking

Conclusion

Azure Integration Services is a powerful and versatile platform for streamlining business processes, orchestrating data flows, integrating with external systems and partners, or building scalable APIs. As businesses seek ways to drive innovation, improve customer experiences, and stay ahead in the digital race, Azure Integration Services remains a trusted technology. Its scalability, security, and flexibility make it a reliable choice for enterprises of all sizes and industries.

We have also seen how Serverless360 helps organizations that embrace the Azure Integration stack to increase the maturity of their Integration platform to provide a better customer experience, which is the core objective of digital transformation.

Serverless360 goes beyond the Azure Integration monitoring and business process tracking by providing Cost Analyzer, which scans through your entire Azure environment and provides insights to optimize significant costs, and Azure documenter module to help you auto-generate seven different pieces of technical documentation like Architecture summary with diagrams, security summary, resource usage summary to be a complete package in successfully operating in the Azure ecosystem.