Azure Service Bus dead-letter Queue
Azure Service Bus queues and topic subscriptions provide a secondary sub-queue, called a dead-letter queue. The purpose of the dead-letter queue is to hold messages that cannot be delivered to any receiver, or messages that could not be processed. So, any message that resides in the dead-letter queue is called a dead-lettered message.
Why monitor Service Bus dead-letter Queues?
Below are a few limitations that users face in real-time while using Azure Service Bus,
- There is no default way to notify a sender or receiver when an active Service Bus message is moved to the dead letter queue.
- You might end up storing a bulk number of Dead-letter Queue messages if not closely monitored.
- The Dead-letter messages won’t be manually removed or processed again until we do so.
This is Service Bus Dead-letter Queue Monitoring and Alerting proves to be significant!
Dead-Letter Queue Reasons
Before investigating the best way to monitor and reprocess Service Bus messages efficiently, let us quickly hop in and understand the reasons behind why an active message get dead-lettered.
Most of the time, we could notice that the message fails to reach the receivers due to the following reasons;
- Dependent service not available
- Faulty message
- Process code issue
Dependent service not available
This could be one of the foremost and time after time reasons where the services that reply on message delivery may go down for a short period. For instance, Redis or SQL connection issues may often happen.
Faulty Message
If in case the message doesn’t have a mandatory parameter or some value is incorrect, then the message will end up in the dead-letter queue after the max delivery count is attained.
The failed delivery can also be caused by a few other reasons such as network failures, a deleted queue, a full queue, authentication failure, or a failure to deliver on time.
Here we can drill down the reasons into two ways:
- System-level dead-lettering
- Application-level dead-lettering
Reasons for System-level dead-lettering
-
- Header Size Exceeded – Maximum header size: 64 KB
- Error on processing subscription rule
- Exceeding time to live value
- Exceeding maxDeliveryCount -When DeliveryCount exceeds MaxDeliveryCount, the message is moved to the DLQ, specifying the ‘MaxDeliveryCountExceeded’ reason code. This behaviour cannot be disabled, but you can set MaxDeliveryCount to a very large number. You could specify the Maximum Delivery Count between 1 and 2147483647.
Do you want us to help set up and showcase how to monitor and handle dead-letter messages in a smarter way? Talk to us
- When Session id property is set to true (the default is false)
Reasons for Application-level dead-lettering
- Messages that cannot be properly processed due to any sort of system issue
- Messages that hold malformed payloads
- Messages that fail authentication when some message-level security scheme is used
Process code issue
This is a very rare case, given a good number of resources in the community to fetch flawless code. It is the developer’s responsibility to keep all scenarios in mind and handle all exceptions.
The best practice for the first and third scenarios is to run a flawless code that will reprocess the dead-lettered messages; you can find the sample code below;
internal class Program
{
private static string connectionString = ConfigurationSettings.AppSettings["GroupAssetConnection"];
private static string topicName = ConfigurationSettings.AppSettings["GroupAssetTopic"];
private static string subscriptionName = ConfigurationSettings.AppSettings["GroupAssetSubscription"];
private static string databaseEndPoint = ConfigurationSettings.AppSettings["DatabaseEndPoint"];
private static string databaseKey = ConfigurationSettings.AppSettings["DatabaseKey"];
private static string deadLetterQueuePath = "/$DeadLetterQueue";
private static void Main(string[] args)
{
try
{
ReadDLQMessages(groupAssetSyncService, log);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
throw;
}
finally
{
documentClient.Dispose();
}
Console.WriteLine("All message read successfully from Deadletter queue");
Console.ReadLine();
}
public static void ReadDLQMessages(IGroupAssetSyncService groupSyncService, ILog log)
{
int counter = 1;
SubscriptionClient subscriptionClient = SubscriptionClient.CreateFromConnectionString(connectionString, topicName, subscriptionName + deadLetterQueuePath);
while (true)
{
BrokeredMessage bmessgage = subscriptionClient.Receive(TimeSpan.FromMilliseconds(500));
if (bmessgage != null)
{
string message = new StreamReader(bmessgage.GetBody(), Encoding.UTF8).ReadToEnd();
syncService.UpdateDataAsync(message).GetAwaiter().GetResult();
Console.WriteLine($"{counter} message Received");
counter++;
bmessgage.Complete();
}
else
{
break;
}
}
subscriptionClient.Close();
}
}
Approaches for monitoring the Dead-letter Queues
Native-Azure Monitoring tools
Azure Monitor offers monitoring and alerting features for Service Bus queues and dead-letter messages. It allows to set alert rules and action groups to receive notifications when your number of dead-letter messages rises to a certain threshold.
Still, there are certain areas where Azure Monitor falls short
- No consolidated monitoring on state, properties/metrics
- Monitoring Queues/Topic Subscriptions dead-lettered count in a single report is not possible
- It doesn’t allow viewing the subscription-level metrics for dead-lettered messages
- No features for repairing & resubmitting/auto-reprocessing dead-lettered messages
- Lack of automation for performing corrective actions on critical messaging issues
Is building custom solutions considered the best choice?
Few organizations prefer building monitoring solutions because they can tailor them precisely to their needs. This allows teams to develop exclusive features for overcoming the above challenges in the native-monitoring tools.
Drawback: It is time-consuming and requires a lot of manual effort to manage the developed solution.
Monitoring Service Bus dead-lettered messages made easy
Be it Azure Service Bus Queue or Topic Subscriptions, Serverless360 makes it handy to monitor the dead-letter messages in your testing, pre-production, and production environments.
It completely eliminates the complexity or limitation of Azure Monitor, which might not fulfill your production environment’s requirements. For instance,
- Monitor multiple Queues and Topics Subscriptions dead-lettered count in a single report
- Monitor a Queue metrics against another metric, for instance, compare active messages vs dead-lettered messages
- Consolidated monitoring of both the property values and status of Service Bus
I will take you through Serverless360 and showcase how quickly you can configure the alert rule to monitor dead-lettered messages.
Once you signed up for a free trial, associate the Azure Service Bus in Business Application (Serverless360 feature) – this module lets you access, monitor, and manage various Azure resources.
To configure rules, head to the desired resource type in the Manage resources section inside the Business Application. Select the respective queue or topic for which the rules should be configured, and choose the dead-letter count property with warning and error thresholds. That’s it.
Do you want us to help set up and showcase how to monitor and handle dead-letter messages in a smarter way? Talk to us
<Now, you will be notified based on configured value on your preferred choice of notification channel.>
Azure Service Bus dead letter queue resubmit
Serverless360 makes your life easier when it comes to repairing and resubmitting the dead-letter messages of queue or topic subscription providing out of the box features like
- Advanced search option to find the dead-lettered message
- Intelligent automation to bulk resubmit the dead-lettered messages
- Variety of message receiving modes like Peek-lock and Deferred mode
- Back up and purge messages
- Download particular set of messages
Now, look at the step-by-step process to resubmit the dead-lettered messages in Serverless360
-
-
- Click the queue whose dead letter messages must be processed, this will lead to a message retrieval dashboard.
- Select dead letter tab and click to get a specified count of messages
-
- Select those messages to be resubmitted and click resubmit
-
- Select the destination Queue and proceed to resubmit
-
- The messages can also be repaired and resubmitted
- The user can also view the properties, content and dead letter reason of a message
See for yourself how you can efficently handle the dead-lettered messages today! Get Free Trial
Monitoring dead-lettered messages on each Topic Subscriptions
Since the messages are not dead-lettered on the Topic itself but rather on a specific subscription, Azure Monitors’ inability to enable Subscription-level monitoring can be a significant drawback.
But in Serverless360, you get to monitor the Topic Subscriptions on various properties, including the Dead Letter Message Count, etc., and get alerted on threshold violations
Perform automotive operations on dead-lettered messages
Eliminate manual intervention for mundane/repetitive tasks when it comes to handing dead-lettered messages.
- Auto reprocess messages that get dead lettered for known reasons
- Auto-delete the dead-lettered messages once resubmitted
- Resubmit the bulk dead-lettered messages with intelligent automation
- Get notified when an automated task on your dead-letter queue is complete
-
-
Conclusion
I hope this blog gave a better understanding of Azure Service Bus Dead Letter Queue, the reason behind a message getting dead lettered. We also discussed on how Serverless360 can help to process Dead letter messages much smarter way.
Also, check how to Resubmit and Delete Dead-letter Messages of Queues and Topic Subscriptions blog for more insights.
