Overview
A claims processor received claim submissions from partner systems and passed them through Azure Service Bus queues to .NET workers on AKS. After a new partner feed went live, the submission queue backed up within hours, consumers sat at full CPU, and the platform team saw throttling errors on the namespace. A week later, producers to a different queue began failing with quota errors although that queue was almost empty.
Challenge
The workers abandoned a message on any exception, including validation errors that would never succeed, and the queue's maximum delivery count had been raised to 1000 to stop messages dead-lettering during outages. A malformed claim was therefore delivered hundreds of times back to back, and a batch of them kept every worker busy. The second problem was quiet: messages that expired on a related queue were dead-lettered, nobody read the dead-letter queue, and Service Bus has no automatic clean-up of it, so it grew until it used the entity's size quota and sends began to fail.
Environment
A Premium-tier Azure Service Bus namespace with queues and topic subscriptions, .NET workers using the Service Bus processor on AKS, partner feeds arriving over HTTPS and published by an ingestion service.
Approach
AceMQ went through delivery counts, dead-letter reasons and entity sizes with the customer's engineers. Peeking the dead-letter queue and grouping by reason showed the split between messages that expired and messages that were never going to succeed, and a replay in staging confirmed the retry loop before any setting was changed.
Solution
- Split worker exceptions into transient, which abandon the message with a backoff, and permanent, which dead-letter it at once with a reason and description
- Returned the maximum delivery count to a low value so a message that keeps failing reaches the dead-letter queue within minutes
- Set retry options with exponential backoff on the client and a lock renewal window longer than the slowest handler
- Drained and classified the old dead-letter backlog, then resubmitted the valid messages from Service Bus Explorer
- Alerted on dead-letter message count and on entity size as a percentage of its limit, for every queue and subscription
- Added a validation step at ingestion so a malformed feed is rejected back to the partner instead of entering the queue
Outcome
The next malformed feed produced a short burst of dead-lettered messages with clear reasons, while valid claims kept flowing. The quota errors stopped once the old backlog was cleared, and an alert now fires well before a dead-letter queue can affect sends.
Technologies
Azure Service BusAzure
Kubernetes.NET
RabbitMQ
PostgreSQL
Docker
OpenTelemetry
Java
Apache Spark
