A forgotten subscription can throttle every producer on the namespace
Producer throttling clears within the incident, and the subsequent policy work means abandoned subscriptions expire on their own and rising backlog raises an alert well before it reaches the quota.
Overview
Pulsar retains messages until every subscription has acknowledged them. A subscription that stopped consuming — a decommissioned service, a crashed consumer, a test client nobody removed — holds the backlog open until the namespace quota is reached, at which point producers are throttled or rejected. AceMQ diagnoses this quickly and fixes the policy underneath.
Challenge
The reported symptom is that producers have started failing or slowed dramatically, with no change on the producer side. The cause is a backlog quota policy reaching its limit with a producer_request_hold or producer_exception action. Finding which subscription is responsible requires per-subscription backlog inspection across the namespace, and the retention and TTL policies interact in ways that are easy to get wrong when someone tries to fix it under pressure.
Environment
Apache Pulsar clusters on Kubernetes or bare metal with multi-tenant namespaces, mixed subscription types, and geo-replication in some deployments.
Approach
AceMQ engineers inspect per-subscription backlog and acknowledgment position across the namespace to find the subscription holding retention open, and confirm whether it is a live consumer that has fallen behind or an abandoned one. Relief is applied appropriately — skipping or expiring the abandoned backlog, or addressing consumer throughput for a live one — and the policy set is then corrected so it fails visibly next time.
Solution
- 115-minute emergency SLA with named senior engineers, 24/7, no tier-1 triage
- 2Per-subscription backlog and acknowledgment position inspection to identify the subscription blocking retention
- 3Backlog quota policy and action review across producer_request_hold, producer_exception, and consumer_backlog_eviction
- 4Retention and message TTL policy alignment so abandoned subscriptions expire rather than accumulate indefinitely
- 5Consumer throughput diagnosis for live subscriptions, covering receiver queue size, ack timeout, and negative-ack redelivery
- 6Subscription lifecycle and alerting so abandoned subscriptions and rising backlog are detected before quota enforcement
Outcome
Producer throttling clears within the incident, and the subsequent policy work means abandoned subscriptions expire on their own and rising backlog raises an alert well before it reaches the quota.
Technologies
Related Use Cases
Apache Pulsar BookKeeper IO Remediation
Resolving cluster-wide write latency caused by bookie journal and ledger device contention, and restoring write quorum headroom.
Apache Pulsar Multi-Tenancy Design
Designing tenant, namespace, and policy structure so independent teams can share a Pulsar cluster without interfering with each other.
Need Expert Apache Pulsar Support?
AceMQ's senior Apache Pulsar engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.