This blog is the continuation of the blog “Get the best out of Azure Data Factory – Part 1“. It explains how Serverless360 addresses the pain points and enhances the usage of Azure Data Factory. Serverless360 is a single platform solution that enables Operations and Support Teams to manage and monitor Azure Serverless services more efficiently.
Let us have a quick recap of part 1 before looking into new features that Serverless360 provides for Azure Data Factory.
- Monitor the Data Factory, Pipelines, and Integration Runtimes based on metrics.
- Trigger a pipeline execution, stop the execution of the pipeline.
- Search for a pipeline run based on its execution status, start time, and end time.
- Visualize the trigger runs and Integration Runtimes related to a Data Factory.
- Visualize Integration Runtimes and the Nodes, Linked Services, and Pipelines connected to it.
Listed above are the capabilities of the Azure Data Factory provided by Serverless360 discussed in part 1. Now let us discuss a few other features offered by Serverless360 for Data Factory.
Visualizing Data Factory Pipeline runs
Consider a scenario where there are multiple failed runs in a pipeline at a specific time interval, and there is a need to investigate the cause of the failures. In such cases, Serverless360 can retrieve the failed runs based on run status in the specified time window.
Once the failed runs in a specific time interval are retrieved, the next step is to find the activities which caused the failure. Serverless360 allows the user to view all the activities belonging to the pipeline run by clicking on the name of the run. These activities can also be retrieved based on their execution status.
Once we have found the activity which caused the failure of the pipeline run, the next step is to investigate the input and output bindings of the activity which caused the failure. Serverless360 provides an option to visualize the input and output bindings of the activities by clicking on its name.
The next step would be to run the pipeline with the correct values. Serverless360 extends its capability to trigger a Data Factory Pipeline by dynamically passing values to the already configured pipeline parameters without having to switch to the Azure portal. Each time, the pipeline can be triggered with a distinct set of parameter values. When a parameterized pipeline run is re-run, the parameters of the original run are carried forward to its execution.
This scenario explains how Serverless360 can efficiently analyze the failures caused in the Data Factory Pipeline and rectify the same with the least effort.
Monitoring Integration Runtime
Consider a scenario in the Integration Runtime where the number of available nodes in the cluster goes down for some reason, or there is a sudden spike in the CPU utilization of the Integration Runtime, indicating some failure during non-business hours. Now here arises a need for a user intervention to set things correctly.
To mitigate such scenarios, Serverless360 allows the users to configure monitoring on such critical properties and notifies the users when any violation is observed on the set threshold values. Additionally, periodic notifications can also be delivered even when there are no violations to know the status of the critical metrics.
Granular user access policy
Any number of users can be added to the Serverless360 account to manage and monitor the resources associated with it. However, not all users can be granted the authorization to perform all the operations. Serverless360 allows fine-grained access control definition to the users with personalized functional capabilities to restrict unauthorized access.
Consider a scenario where a user should be given access only to perform monitoring on the Data Factory, or a user should be restricted from performing certain operations. All these requirements can be achieved by enforcing the user access policy provided by Serverless360.
Roadmap for Azure Data Factory in Serverless360
Serverless360 constantly adds new features to its portfolio. The roadmap for Azure Data Factory includes the below list of items
- Support to Integration Runtime Node that includes metric monitoring and dashboards capabilities
- Property monitoring on Data Factory and Integration Runtime
- Stopping a pipeline trigger
Conclusion
This blog briefs on some pain points faced by the users of Data Factory and how Serverless360’s Azure Data Factory Monitoring capabilities can address those scenarios. Stay connected to learn more about Serverless360’s features in managing the Azure Serverless services.
