Azure Integration Services Ask Me Anything – Dec 2018

The Microsoft Azure Integration Services team hosted a special Ask Me Anything (AMA) session on Twitter, Thursday, December 06, 2018 from 8 AM to 10 AM PT. Questions to @AzureSupport with #IntegrationAMA were answered during the session.

We took an effort to blog the session that will help you to refer the questions and answers from one place rather than scrolling in your twitter feed.

1. When are you going to support nested folder structures in Blob Storage?
   Today you can use an Azure Event Grid trigger with an event grid subscription and a filter for the subfolder.
2. I really have enjoyed working with Logic Apps but have the following questions. 1.Any plans to improve CI/CD for Logic Apps, maybe build GitHub directly into the product. Any plans to improve JSON integration?  Support for GUID would be nice, Split JSON Arrays?
   Hi! Sorry we took a while to get to your question. Yes, we are looking at places to improve the CI/CD experience, the first place this is likely to appear is in our Visual Studio code add-in. For your second question, yes, we are looking at adding greater JSON support (e.g. reusable schema) – can you explain more about what you mean wrt GUID support and JSON array split on?
3. Could you please give a better view about the new and updated features in the Azure official roadmap?
   Hi, all Azure service roadmap details are now shared through the Azure updates site here. An update is coming soon – bear with us.
4. Please add the “read only” support for the FTP/SFTP/FTPS connector in order to be integrated with read-only FTP servers.
   Hi, this is an interesting scenario, can you provide some more details on how these servers are configured as ‘read only’ servers. All these file connectors provide capabilities to read from the servers, so please help me understand more about your scenario and requirements.
5. Are you going to change the way the Integration Account is billed?
   Nothing concrete to share yet. We have received feedback for consumption-based integration account and we are considering it – but no specifics yet. however, that won’t change the way we bill, it would add another billing mechanism.
6. When can we finally see Azure API Management on-premise API gateways?
   We have definitely heard this feedback from our customers and that’s something we keep on our minds.
7. I write to a vendors SFTP server, before the SFTP can validate that the file has dropped, the SFTP server moves the file. Any chance having the ability for the logic app SFTP action not to check to see if the file was written.
   Hi, my apologies that you are running into the issues. We are aware of this issue and would like to work with you to get more specifics about your scenarios. Can you please send me a direct message and we’ll take if further. Thanks!
8. Are there ETAs for the Integration Service Environment Public Preview and GA?
   The ISE is currently in private preview and we are planning to take it to public preview early in the new year.
9. What other connectors are in the roadmap to be supported on the Integration Service Environment? Are Enterprise Connectors going to be attached to the VNET? What about File connector?
   Initially we are planning on including all of the on-prem connectors (the ones that go through the on-prem data gateway) in the ISE as well as connectors that connect to Azure services that support connecting to VNETs (e.g. Cosmos DB, Storage, etc).
10. Is there an ETA for KeyVault Support and Selective Logging on Logic Apps?
    We are currently working on a KeyVault connector and early in the new year we will enable property obfuscation.
11. What’s in the roadmap to make Logic App workflows more testable?
    We are currently working on mock/static outputs which should be released early in the new year.
12. Are there plans to support advanced scheduling for polling triggers? E.g. Service Bus trigger polling every 5 mins only from 9 AM to 5 PM?
    We intentionally kept the scheduling feature simple. Logic Apps has a trigger that is configurable and there are a few others. Turning that into a complex scheduler feature would cost a lot of compute cycles that would cost lots of performance. Oops. I misunderstood the question, it appears. I understood that as to refer to Service Bus scheduled messages. Are you asking about Azure Functions or Logic Apps? You can accomplish this today in Logic Apps with an advanced recurrence trigger and having the first action read messages from service bus.
13. When I have a solution with multiple (chained) Logic Apps it would be nice to see quickly which Logic App run fails rather than to check each one separately. Any chances to have a sort of Logic Apps runs group view?
    Hi, this is feedback we hear a lot from customers – I appreciate you raising it. We are looking at ways we could implement this – no specifics to share right now, but watch this space! 
14. Are there any plans for Service Bus to support deferring messages using a Timespan, e.g. defer by 2 hours. The current Defer method, which requires a Sequence number, is not so simple to implement. Using ScheduledEnqueueTimeUtc can only be used for new messages and requires consuming the message and creating a new copy of it which is not ideal.
    There are no plans, but that is a good suggestion that I’ll make a note of. I understand that the scheduled message alternative is a little clunky.
15. Please can you guys not reuse service names across different products?
    Hi there! Thanks for the feedback! Can you elaborate by posting on this forum?
16. Did not see pricing plan for API management consumption plan?
    The consumption tier pricing is published here. The first million calls are free.
17. How has the logic apps adoption so far? what is the monthly volume of logic apps transactions currently?
    we have many customers using Logic Apps, and process a huge number each day, but don’t share specifics publicly – sorry. Many thanks for your question!
    While we don’t have at the moment large enterprise named costumer we mention for public adoption example, Microsoft B2B and A2A integration itself runs on Logic Apps. That’s a pretty big scale in itself and only a fraction of what Logic Apps run. Note furthermore that Gartner magic quadrant market leader analysis includes adoption in its criteria (and AIS ended up in the leaders).
18. When could we expect to have Logic apps on-prem? Could you share what your vision is for this
    We’re actively looking at how we could allow customers to deploy Logic Apps on-premises, at the moment we don’t have any specifics to share. Very soon, we will be previewing our Integration Service Environment offering, allowing customers to deploy a dedicated Logic Apps service with virtual network support to connect back to on-prem.
19. Are there plans for the Logic apps integration account to have a serverless consumption tier?
    We do want to offer consumption tier for Integration Account and are working on it. Can you share what are your scenarios and motivators for consumption tier. Is only XML sufficient? and by XML i mean maps/schemas
20. Are there plans for Service Bus to provide an explorer on the Azure portal?
    If you mean going as far as inspecting messages from the portal … that’s a bit of a tough call because messages are transient. It’s a bit different from editing a database. There’s also a rather practical issue in the portal control plane not having insight into the data plane. Serverless360 has a really nice set of tools that complement the portal that you might want to take a look at and that you don’t need to wait for.
21. We are looking to migrate our BizTalk apps to Azure. Would you say Azure Integration Stack matured enough to build robust full blown integrations as #BizTalk? Also what about the performance in Hybrid integration scenarios? Any whitepaper published?
    Yes, absolutely, we have many customers who have built and are running Enterprise-scale integration solutions on Azure today. Hybrid scenarios are very common, and with technologies like VNet, customers can connect to their on-prem assets in a performant way. Take a look at our whitepaper on Azure Integration Services as a starting point.
22. Why do I need to chose Azure Integration Services rather than products like MuleSoft, DellBoomi, etc… Whats the advantage of migrating #BizTalk apps to Azure instead of other offerings?
    There are many benefits to our platform. I’ll do my best to list some of them below
    Get started easily, and pay only for what you use by taking advantage of Serverless. Logic Apps (and just announced Azure API Management) allow you to try out the service, at low cost, then move to production and scale without having to worry about infrastructure.
    2. The power of Azure. By adopting Azure Integration, you can easily use the power of other Azure services as part of your solutions. For example, easily include machine learning into your solution using connectors to Azure Cognitive Services.
    3. For BizTalk customers, you can easily bring Maps and schemas from BizTalk to Azure Integration with our Integration Account. Refer to the official documentation here.
    4. When you want to go dedicated with your service, we give you the flexibility to do so.
    5. You don’t have to migrate all at once! Start building new solutions in the cloud, and connect them to your existing BizTalk solutions on premises.
23. Are we still depending on Visual Studio to build maps. Do you have a Azure mapper available in Azure portal? 2) Can we migrate the existing BizTalk maps files with Scripting functoid and other Advanced functoids to Logic Apps?
    Hi, currently our mapper is available via Enterprise Integration Tool SDK for Visual Studio. We do have plans to provide web based mapper. You should be able to migrate BizTalk maps to Logic Apps – we do support functoids. Please DM if you are seeing any issues.
24. Are there any plans to support startTime for polling triggers, like Service Bus? If we could change the startTime of a running workflow, we could potentially implement the circuit breaker pattern more easily.
    You can leverage Service Bus events to Event Grid to trigger the logic apps. A push system that manages polling without actually polling! Refer to the official documentation here.
25. Is it possible to create an Azure Event Grid Subscription for a particular Service Bus queue or topic, instead of filtering the events received by an Event Grid Subscription, from the Service Bus Namespace? What is the right approach?
    Yes, you can. If you look at the subject structure of the SB events, you’ll see that you get the full path and Event Grid lets you match by prefix. Refer to the official documentation here.
26. Why should I go for Event Grid Topics, when I am able to create an Event Grid Subscription even without any Event Grid Topic? What is the real advantage or use case for an Event Grid Topic?
    Great question! Event Grid Topics allow you to send your own custom defined events to Event Grid and route/filter them in a PubSub manner. Subscribing directly to Azure resources gives you access to Azure defined events.
    You need a topic when you publish. You don’t need a topic when you want to subscribe to events that others, including built-in services, publish.
27. Before Azure Functions 2.0, state of the function can get while receiving list of functions, after 2.0, state is not available while receiving function through exposed API. How can we get to know the changes in API before it breaks our functionality.
    We put release notes on the host GitHub, but upcoming changes (including breaks we made during 2.0 preview) can be found here. There’s also. 1.0/2.0 comparison doc, but unfortunately this particular change doesn’t seem to be captured there.
28. Why hybrid connections inside a #Relay namespace are not being retrieved when using NamespaceManager (WindowsAzure.ServiceBus)?
    Because that client doesn’t support HC, only WCF Relays. The universal management surface area remains Azure Resource Manager.
29. What is the egress cost for API management if we have a service hosted it on premise?
    The additional costs, on top of API Management cost, depend on your setup and architecture. We suggest that you follow up with the respective Azure Networking services.
30. Can API management support if the mediated service takes around 15 minutes to complete?
    It is not supported as well as not recommended. We suggest that you return a URL in the first response, and then try polling that URL for the proper response payload.
31. Can I call using c# the Azure Events Grid to trigger events manually?
    Yes, you can! Here’s more on how to publish events using our c# SDK. Refer the official documentation here.
32. What will the best approach for the API management to get all the failures for a service in the last 3 hours?
    The recommended way is to integrate your API Management instance with Application Insights. You can also use alternative ways, such as logging to Event Hub on failures.
33. When using custom webhook as an event handler in Azure Event Grid subscription, that custom endpoint must be validated. How can that be achieved?
    There are two ways you could go about this:
    1.Validation Code handshake – have your webhook echo back the validation code from the “subscription validation event” sent by Event Grid to the webhook at the time of subscription creation.
    2. Validation URL handshake (manual handshake) – If you don’t control the source code at the endpoint, do a GET request on the validation url sent as a part of the “subscription validation event”. Find more details about this here.
34. Logic Apps retry fires on 408, 429 and 5XX. Can this be made configurable, as not all API’s and connectors return correct HTTP error codes.
    That’s a great feature request. You could add it here. In the meantime, you can accomplish this by using an Until loop which catches the specific error codes and retries the request based on your criteria.
35. NamespaceManager (WindowsAzure.ServiceBus) does not have a method to perform operations on hybrid connections. Are there any plans or ETA for those?
    Because that client doesn’t support HC, only WCF Relays. The universal management surface area remains Azure Resource Manager.
36. Any target date for consumption based billing of the integration account?
    No concrete date yet but we are working on it. Can you share what are your scenarios and motivators are for consumption tier. Is only XML sufficient?
37. In this blog, it’s stated that Azure Integration Services will be able to run on-premises. When can we expect this?
    Our Integration Product Roadmap
    It’s something we’re actively looking at, but we can’t share any specifics right now. Customers needing on-premises connectivity can use our data gateway, or our new Integration Service Environment offering for VNET support.
38. Can we get good insights into an up-to-date roadmap for Logic Apps? Trello has been stopped and Azure Roadmap / Logic Apps release notes are almost empty.  Visibility on this is extremely important!  Impossible to know what is new and what will come?
    Roadmap for all Azure services is now through the Azure Updates site here. An update is coming soon, bear with us.
39. More than a year ago testing/mocking capabilities of Logic Apps were announced. When can we expect this?
    The mock outputs feature is currently in development and should be released early in the new year.
40. Only 2 Logic Apps Live sessions this year, where we used to have this monthly. How will we get updates on new features and work in progress?
    We are planning on having a Logic Apps live very soon. Watch the @logicappsio handle for the announcement.
41. What are the next Azure services that will emit Event Grid events? Interested in SQL DB change feed, Logic Apps / Data Factory runtime events, etc.
    Thanks from Logic Apps team, we will consider the idea of emitting events from Logic Apps engine.
42. When can we expect real DR support for Logic Apps (e.g. connector state replication), including documentation on this?
    We are planning on doing trigger state replication in the next semester. 
43. Why Azure Event Grid trigger is recommended instead of blob trigger to get notified when there are entries in blob storage from undelivered events from event grid subscription?
    Using the BlobCreated trigger allows you to create event driven architectures that cut down on latency and improve efficiency. Without the blob trigger you have to poll for changes which is never as efficient.
44. When performing Logic Apps cost calculations, in many cases a huge percentage goes to polling, if we don’t consider integration account. This is quite difficult to explain to customers.   Will it remain this way?
    Where possible, we suggest using something like Azure Event Grid to create an event based model to avoid polling.
45. What’s solution for Logic Apps connecting to on-prem, where ISE is too expensive (we assume) and where OPDGW is not considered enterprise-ready?
    Besides ISE and OPDGW you can also use APIM configured across the VNET that can be called by your logic app.
46. What is the best way to get help on API management for any technical issues during our poc to setup open id or auth2 security protocols.
    We have comprehensive documentation here. You can also contact @AzureSupport for help, as well as post questions on StackOverflow

47. Will the message size limitation of 100MB remain? Enterprise integration often needs higher message sizes.  Only support in some connectors is not sufficient, we must be able to transform and process these messages.
    We do plan to increase the message size limitation. Please do share your scenarios and inputs on what limits would be sufficient. 
48. Azure API management how long will it take to restore API management backup to another data center for disaster recovery management.
    Usually, creation of the API Management instance and scaling it to your desired capacity is the longest step in this process – you can expedite it by making sure your API Management instance is pre-provisioned. Also, make sure to check your backup/restore flow before DR.
49. Can we really bid farewell to BizTalk and migrate 100% to Azure Integration Services? Sad situation though.
    Yes, absolutely you can build solutions on Azure Integration Services. BizTalk still exists and is fully supported, see our blog post for the summary
50. Why is the pricing of hybrid connection higher than the WCF relays?
    Hybrid connections are aimed at a different use-case scenario than the WCF relay. The WCF relay was built (and priced) with an eye on RPC scenarios and hybrid connections is designed and priced for tunneling scenarios. Some customers save $1000s by avoiding complex VPN setups.
51. When will we be able to disable actions? e.g. Disable an email alert action for testing a change. Or disable the database write action for the same purpose.
    We are working on a feature called mock/static outputs that will enable you to provide a static result that doesn’t call the actual endpoint.
52. When will we be able to mock actions and runs?
    Mock/static outputs is currently in development and should be released early in the new year.
53. Are there any plans to make the HTTP trigger webhook URLs shorter and not include the query string? I’ve run into a few cases where I’ve had to use a Function proxy to get around a 3rd party’s restrictions.
    API Management allows you to expose HTTP trigger with a shorter URL without querystring. For low volume you now have the new starter tier of APIM also.
54. Is there a documented/supported way to cleanly relay messages from an @RabbitMQ (AMQP0.91) exchange to #ServiceBus (e.g. mapping routing keys to autoforwarded second-level topics). This is a huge question for us wrt Azure migration!
    We are planning a “Service Bus for RabbitMQ developers” guide early next year.
55. Do we have plans to make developer tooling more attractive through visual studio or code. Highly depended on internet and very slow?
    For which Azure service? There are already both Visual Studio and Visual Code tools for Logic Apps. Thanks for the feedback – we have updates coming pretty soon for our @code add-in, you can also contribute to it if you wish as it is open source 🙂
56. Can we support gremlin query with CosmosDB connector?
    Love it, please add or upvote this idea here

57. Any plans to add geo-distribution to cheaper SKUs of Azure API Management?
    We do not have immediate plans to add multi-region support to cheaper SKUs of API Management.
58. Do we have better story about using Microsoft graph with logic apps! Some documentation and MSI support for graph-based resources would be great !!
    What part of Microsoft graph are you trying to use? Do you need a Microsoft graph connector? Could you expand what your scenario is?
59. Is there a way to set up the Service Bus relay such that it would low-latency/high-throughput stream on-prem MSSQL table row changes over to SB? That would be a HUGE enabler for us.
    App Services Hybrid Connections give you a tunnel of that sort and that runs over the Relay. Refer the official documentation here.

    If you are up for an adventure, there’s a new bridge project that’s in the works which lets you create that same kind of bridge from and to anywhere. Refer here for insights.

60. When connectors other than the HTTP connector will support Managed Identities (MSI)?
    We agree this is the logical next step. We’re working on expanding MSI support to connectors for Logic Apps.
61. How accurate (in %) is speech 2 text API (“API”) 2. Can API distinguish live vs recorded voice? 3. Can API distinguish human vs artificial voice? 4. USPs vs Nuance Speech to text API?
    Hi is this a question about Cognitive Services? @AzureSupport can you direct?
62. your recommendation is to have a perpetually running and cost-incurring APIM resource + everything else in its RG in a separate region?
    There’s no single answer to this question, it depends on your business requirements and the desired reliability. For maximum reliability, we usually recommend a Premium instance in at least two regions.
63. Do we have any plan for better tracking and run visualisation (like Power BI) through portal?
    Please use the Logic Analytics for Logic Apps for tracking, alerts, etc. at scale. You can view it as a blade in the Azure management portal. we already provide UI based tracking for logic app runs via Log Analytics. Have you tried it? If you have any specific, ask please do let us know.
64. Event Grid needs Service Bus premium to send events to Logic Apps. There are any plans to support lower service bus tiers?
    Yes, we do plan to support it in the near future!
65. Do you have any plans to build tooling for creating Liquid transformations? Also, we need more documentation on what are the features available for Liquid.
    Thanks for your feedback on documentation. We’ll work on it. We are also looking into improving our mapper experience both for XSLT and Liquid by providing an improved and web based mapper in Azure portal. If you have any specific scenarios/requirements do share with us.
66. Logic Apps can we have visualized map. Both XSLT based or liquid template does not have visualisation. We tend to move more towards code to do message translation.
    We do offer visualized map experience for XSLT in our current mapper, unless you are looking for something else – can you share more details on this ask. We do want to offer an improved and web based mapper to our customers – so keep the inputs coming.
67. The docs and code example on monetization via Stripe are woefully out of date. Any chance of updates? Upcoming developer portal has ease of payment integration in its scope?
    Thanks for letting us know, we will take a look at that. Monetization is not a focus area of the developer portal work.
68. Can we have webhook trigger on SharePoint library and getting modified document name?
    Please add or upvote this idea here
69. When Logic Apps in the web will be connected directly to a repository? There are any plans to evolve around source code control?
    Sounds like using Azure DevOps Services with a release pipeline doing ARM deployment to Azure. This is possible today.
70. API Managment: any plans to release the gateway to be deployed on premise?
71. Service Bus: is it possible to have metrics by topic subscriptions? (And not only by topic)?
72. Are there plans to bring features like transactions, dead-lettering and scheduling to Event Hubs, or will these forever belong to Service Bus alone?
73. I’m getting a little confused for an ideal @AzureFunctions based API, is Functions Proxy still going to be in play between the Gateway and Functions? Is there a good overview/drawing of the current SOA arch? Thanks!
74. When integration of Apim with Azure DevOps? What are the best methods to CI/CD Apim apis, policies, etc?
75. Any plans to add Logic Apps debugging (something like in the Azure Data Factory)?
76. Are there any plans to support the use of Logic Apps parameters within the URL to call Functions, Nested Logic Apps and APIM?
    This limitation is very painful when creating Release Pipelines. More info here
    
77. I have a couple question, why is API Management so awesome? Why everyone should use it?
78. Is there a specific library for Azure Functions like Microsoft.Azure.Management .ServiceBus.Fluent library for Service Bus?
79. API management – to avoid single point of failure should I need to have one API management per data center so need to buy minimum 2 ( expensive premium )
80. API management- Looking at the pricing we pay egress charges +  application gateway charges for all outbound data transfers
81. All our services currently are hosted on premise is it a good approach to use Azure API gateway for our scenario
82. Are there any plans to allow vnet and multi zone support for non-premium tiers (expensive for our volume)?
83. Azure API management can you onboard azure ad users and non-ad users to consume the API’s
84. Having a direct URL to a specific Logic Apps run detail monitoring view, would be so powerful in notifications. When will this be available?
85. Best way to get monthly API management API metrics reports number of calls and failures per service
86. In what scenario, logic apps metric ‘RunFailurePercentage’ go beyond 100 percentage? how the metric value is calculated for the metric which has Unit as Percent?
87. Azure Integration Services was promised to be more than a marketing name. What have been the efforts in this area, to make it a more unified offering?  The reference architecture is quite limited

Note: The answer thread in Twitter conversation may have been continued even after this blog is published.