Introduction
There are two types of queuing mechanisms supported by Azure
- Azure Storage Queues
- Azure Service Bus Queues
What Is the Benefit of Using Queue in Azure?
Messaging queue acts as an effective communication mechanism between many integration services and distributed applications. Queues are very simple to use and provides better robustness with low cost.
Storage Queues
Storage Queues are part of the Azure Storage infrastructure, feature a simple REST-based GET/PUT/PEEK interface, providing reliable, persistent messaging within and between services.
Service Bus Queues
Service Bus Queues are part of a broader Azure messaging infrastructure that supports queuing as well as publish/subscribe mechanism, and more advanced integration patterns. To know more about Service Bus Queues/Topics/Subscriptions, have a look into the overview of Service Bus.
Azure Storage Queues were invented before the advent of Azure Service Bus Queues. The Azure Storage Queues are built on top of the Azure Storage Services. The Azure Service Bus Queues are built on top of the broader Azure messaging services, designed to solve the problems in application integrations and to integrate applications that vary in environments, communication protocols, domains, and networks.
In this blog, we will try to understand the key differences and features of Azure Storage Queue vs Service Bus Queue.
What Is the Difference Between Service Bus Queues and Storage Queues?
Azure Service Bus queue offers ordering with FIFO mechanism, but Storage queue does not support ordering. In Service Bus queue the maximum queue size is very low when compared with the Storage queue. The maximum queue size of the Storage account is 500TB and for the Service Bus queue, it is 1 GB to 80 GB. Even though the queue size for the Service Bus queue is low, it can handle huge messages up to the size 256 KB in the standard tier and 1MB in the Premium tier. On the other hand, the Storage queue can only accept messages with a maximum message size limited to 64Kb. There are a lot of other differences like the purge queue option supported in the Service Bus queue but not in the Storage queue. Now, let’s discuss when to choose which service.
Choose Between Azure Storage Queue vs Service Bus Queue
Message Size
Azure Storage Queues can be used when there is a need to store messages larger than the size of 80 GB. This is because the maximum size of a Service Bus Queue is 80 GB. Once this limit is reached, the queue will start to discard the incoming messages throwing an exception.
Polling
Service Bus Queues can be used when the application must receive messages without polling the queue.
Server-side Logs
Azure Storage Queues provide logging of all the occurred transactions in the Storage Queue which can be used for analytics or audit purpose.
Duplicate Detection
Azure Service Bus Queues provide a property called requires Duplicate Detection. When this property is enabled, all the duplicate messages within the provided duplicate detection time frame window will be discarded. The new message will be considered as a duplicate when there is another message already in the queue with the same message id.
Session Handling
If the application requires to process the messages within a session separately, Service Bus Queues can be used.
FIFO Handling
Service Bus Queue maintains the FIFO principle which cannot be guaranteed in the Storage Queues. This is due to the visibility time out feature of the messages in a Storage Queue. This causes the visibility time out of the expired message to be placed after a message, which was originally enqueued after it.
Failure Handling
If the application requires load balancing, failure tolerance, and increased scalability then Azure Storage Queues are the best choice.
Dead-letter
Dead-letter messages are unwanted or error messages due to an error that is stored separately in the Service Bus Queue or Topic Subscription. If the downstream application goes down, it is possible that the messages in the Service Bus Queue would get dead-lettered exceeding the Time to Live. This makes sure that none of the active listeners will try to process unwanted messages. It also provides us a way to view the unprocessed messages or unwanted messages.
In certain cases, the messages in the Service Bus Queue will be automatically dead-lettered. Some of such reasons are:
- Messages sent to session enabled queue without session id will be dead-lettered
- Messages that are received beyond the provided max delivery count of the queue will be dead-lettered
- Messages with header size more than 64KB will be dead-lettered.
Apart from the above-mentioned scenarios, the business application can also dead-letter messages by providing a custom dead-letter reason. For example, in an e-commerce scenario, while processing an order the message can be dead-lettered due to custom reasons like “Item Out of Stock”, “Invalid details” and so on.
Transaction Support
Local transactions in the context of a single queue are supported in Service Bus Queues
Flexible Access Control System
Both the Storage Queues and Service Bus Queues provide flexible and reliable delegated access control mechanisms. Users can provide access to the Namespace and Storage Account level or at the entity level.
Scalability
When it comes to scalability, Storage Queues are more preferred as the Storage Queue can store up to 200 TB of messages. It is also possible to create an unlimited number of Storage Queues in a Storage Account.
Reduced Messaging Operation Cost
Messaging operation cost can be reduced using the Receive and Delete mode of Service Bus Queues. When the messages in the Service Bus Queue are received in Receive and Delete mode, they are removed from the queue.
Auto Forwarding of Messages
The auto-forward capability provided by Service Bus Queues, allows us to configure another destination queue to which all the messages received by the source queue will be forwarded. This mechanism can be used to achieve security.
Naming Conventions
Allowed characters in Storage Queues names are lowercase alphabets, numbers, and hyphens with length ranging from 3 to 63. Name of a Service Bus Queue can be up to 260 characters long containing letters, numbers, periods, hyphens and underscores.
Maximum Message Size
The maximum message size in Storage Queues is 64 KB. If the messages are base64 encoded, then the maximum message size is 48 KB. Large message sizes are supported by combining queues with blobs, through which messages up to 200 GB can be enqueued as single data. The maximum message size of Service Bus messages is 256 KB in Standard Tier and 1 MB in Premium Tier. The maximum size of the message header is 64 KB
Maximum Number of Queues in Service Bus
A maximum of 10,000 queues can be created within a single Service Bus Namespace. If there is a need to create more than 10,000 queues within a single Service Bus Namespace, Azure support team can be contacted.
Poison Messages in Storage Queues
Whenever the messages in a Storage Queue are retrieved more than the specified Dequeue Count, the messages are then moved to the configured dead-letter queue.
For a more detailed comparison of Storage Queues and Service Bus Queues, we highly recommend referring to this content. Now let us see how both the messaging services can be better managed and monitored using Turbo360.
What Is Queue Storage Used For?
Azure Queue Storage is a service for storing large numbers of messages. You access messages from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue message can be up to 64 KB in size. A queue may contain millions of messages, up to the total capacity limit of a storage account.
Features Available for Service Bus Queues in Turbo360
View Messages
Consider when there are errors or Dead-letter messages in a Service Bus Queue. The need is to view the error details and the content of the messages. With Turbo360, users can get to view the content, properties and error details of a message.
Resubmit Messages
When there are Dead-letter messages with the business data, it needs to be resubmitted to process them further. Using Turbo360, messages from one Service Bus Queue or Topic Subscription can be sent to another Queue or Topic enabled with backup to the Storage blobs.
It is also possible to modify the content of the message and resubmit it to another Service Bus Queue or topic through repair and resubmit feature.
Delete Messages
The messages in the Service Bus Queue or Topic Subscription can be deleted using Turbo360. This feature can help controlling Azure spending in retaining unwanted Dead-lettered messages.
Automated Tasks to Process Service Bus Messages
Consider a scenario, where the client application picking up the messages from a queue is down for few hours, all the messages received by the queue with the minimal value set Time to Live will be moved to the dead-letter path of the Queue or Topic Subscription. Now when the client application resumes its state, there may a need to process the dead-lettered messages. In this case, you can schedule an Automated Task to resubmit the dead-lettered messages to the same queue. With the Resubmission Automated Task, it is possible to either resubmit the copy of the message or move the original message or purge of the messages.
Monitoring Service Bus Queues and Topic Subscription
Using the Threshold and Status Monitors of Turbo360, the following properties can be monitored.
- Status
- Transfer DeadLetter Message Count
- Size in Bytes
- Message Count
- Scheduled Message Count
- Active Message Count
- DeadLetter Message Count
- Transfer Message Count
Threshold monitor in Turbo360 will be useful for developers to get alert reports whenever there is a violation. Whereas with the Status monitor, users can get the health report of the Azure Services in the defined hours.
Using the Data monitors, some of the following metrics can be monitors which help in monitoring the performance and the reliability of the Queue or Topic
- Server Errors (Count)
- User Errors (Count)
- Size (Bytes)
- Incoming Requests (Count)
- Count of messages (Count)
- Count of active messages (Count)
- Count of dead-lettered messages (Count)
- Count of scheduled messages (Count)
Governance & Audit
With Governance & Audit in Turbo360, the user finds activities done by users on Service Bus and Storage entities. It is possible to clearly investigate Who did the activity, what activity has been done and at what entity, the activity has been done. It is also possible to track what content or properties that were changed in the message.
Features Available for Storage Queues
View Messages
With Turbo360, users can get to view the message content in the Storage Queues by retrieving the messages.
It is also possible to view the properties of the messages in the Storage Queue like dequeue count, Insertion time and more.
Message Processing in Storage Queues
Consider there is a need to send the messages in a queue to another queue or to purge the old messages in a queue. With Messages processing in Turbo360, the user can update, resubmit the messages, repair & resubmit the message and Delete the message.
Automated Tasks to Purge Storage Queue Messages
If the number of messages in the Storage Queue exceeds the limit that can be deleted manually, an automated task can be created to purge the Storage Queue periodically when the need is to deal with a huge volume of messages.
Conclusion
The above topics should help us in identifying the key differences between the Azure Storage Queue and Service Bus Queue. The decision of choosing between the Azure Storage Queues and Service Bus Queues can be depending on the business problem that needs to be solved.
