Message Resubmission challenges with ‘RequiresDuplicateDetection’ enabled Service Bus Entities

|  Posted: May 21, 2018  |  Categories: Azure Serverless360 Service Bus

We have come across many business scenarios involving message resubmission to an Azure Service Bus entity with ‘Requires Duplicate Detection’ property enabled, and the requirement is to prevent message loss during resubmission. Serverless360 features are designed after a careful investigation into real word integration solution challenges and feedback from our existing customers. With a lot of attention to detail, we have built this capability to avoid message loss. Especially, when you try to resubmit messages to Azure Service Bus entities with ‘Requires Duplicate Detection’ property enabled. To understand this article, it is a pre-requisite to have knowledge of the ‘Requires Duplicate Detection’ property and ‘Duplicate Detection Time History Window’ property on an Azure Service Bus entity.

Requires Duplicate Detection

When set to true, the ‘Requires Duplicate Detection’ property will ensure that all enqueued messages with a duplicate value of the Message-Id property within the ‘Duplicate Detection History Time Window’ will be ignored.

Duplicate Detection Time History Window

If duplicate detection is enabled for a queue the ‘Duplicate Detection History Time Window’ value will specify how long the Message-Id values for the received messages will be retained to check for duplicate messages. The property is a Timespan with a default value of 10 minutes and a maximum allowed value of 7 days.

Now, consider a business scenario, where an integration solution orchestration has an application upstream, that pushes messages into a queue with the following properties:

  • Requires Duplicate Detection: True
  • Duplicate Detection History Time Window: 10 minutes
  • Time To Live: 1min
    NOTE: Once the entity is created with the ‘Requires Duplicate Detection’ property enabled, it can never be modified.

The application downstream is expected to process the active messages from the queue defined above.

Serverless360 can facilitate Queue property definition at the time of creation itself which is not possible in Azure Portal.create-queue

In the image below you can see that Serverless360 is displaying the properties of the created queue. By turning on ‘Enable Dead-Lettering on Message Expiration’ and sets ‘Default message Time to Live’ to 1 minute. With this configuration, as the application upstream enqueue messages to the created queue, after 1 minute the messages will automatically be dead lettered due to ‘TTL Expired’ exception.

Edit-Queues-1

Serverless360 has the capability to send messages to queue for testing purpose, to mock the application upstream. Send 10 messages to the queue and after 1 minute, the active messages will be dead-lettered.

dead-letter

The challenge with this scenario

In the above-explained scenario, the message in the queue get dead -lettered before the downstream application could process them. Even if the ‘Default message Time to Live’ is redefined to a higher value, the requirement is to resubmit the already dead lettered messages. As the ‘Duplicate Detection History Time Window‘ is 10 mins, while resubmitting the dead lettered messages, the message will be lost being detected as the duplicate message.

Solution with Serverless360

Serverless360 provides a solution to overcome the above challenge and achieve the above requirement. By enabling the ‘Requires Duplicate Detection’ property while resubmitting message, the system would provide the following options.

  • Regenerate the selected single or multiple messages id (new GUID will be created and assigned to every message id)
  • Edit the message id to meet your business requirement, like invoice/202#/####.message-resubmit

As the new message ids are regenerated, these messages will not be lost, when resubmitted to the queue with ‘Requires Duplicate Detection’ property enabled. Also, Serverless360 helps you to automate these tasks by creating Activities. For example, while creating a ‘Dead- Letter Activity’ on a ‘Requires Duplicate Detection’ property enabled ServiceBus Messaging Entity. Enabling ‘Regenerate Message Ids’ option will take care of resubmitting messages with the newly generated GUID as message Id.activities

Read more documentation on Serverless360 capabilities – Documentation link

Serverless360 is a one platform tool to operate, manage and monitor Azure Serverless components. It provides efficient tooling that is not and likely to be not available in Azure Portal. Try Serverless360 free for 30 days! 

Free-Trial