A forgotten subscription can throttle every producer on the namespace

Overview

Pulsar retains messages until every subscription has acknowledged them. A subscription that stopped consuming — a decommissioned service, a crashed consumer, a test client nobody removed — holds the backlog open until the namespace quota is reached, at which point producers are throttled or rejected. AceMQ diagnoses this quickly and fixes the policy underneath.

Challenge

The reported symptom is that producers have started failing or slowed dramatically, with no change on the producer side. The cause is a backlog quota policy reaching its limit with a producer_request_hold or producer_exception action. Finding which subscription is responsible requires per-subscription backlog inspection across the namespace, and the retention and TTL policies interact in ways that are easy to get wrong when someone tries to fix it under pressure.

Environment

Apache Pulsar clusters on Kubernetes or bare metal with multi-tenant namespaces, mixed subscription types, and geo-replication in some deployments.

Approach

AceMQ engineers inspect per-subscription backlog and acknowledgment position across the namespace to find the subscription holding retention open, and confirm whether it is a live consumer that has fallen behind or an abandoned one. Relief is applied appropriately — skipping or expiring the abandoned backlog, or addressing consumer throughput for a live one — and the policy set is then corrected so it fails visibly next time.

Solution

• 15-minute emergency SLA with named senior engineers, 24/7, no tier-1 triage
• Per-subscription backlog and acknowledgment position inspection to identify the subscription blocking retention
• Backlog quota policy and action review across producer_request_hold, producer_exception, and consumer_backlog_eviction
• Retention and message TTL policy alignment so abandoned subscriptions expire rather than accumulate indefinitely
• Consumer throughput diagnosis for live subscriptions, covering receiver queue size, ack timeout, and negative-ack redelivery
• Subscription lifecycle and alerting so abandoned subscriptions and rising backlog are detected before quota enforcement

Outcome

Technologies