Overview
A logistics platform published shipment tracking events to Google Cloud Pub/Sub and turned on message ordering so that status updates for a shipment arrived in sequence. Within days, a subscription that had kept up easily was minutes behind, and some shipments showed no updates at all.
Challenge
The ordering key was the carrier ID, and the platform worked with only a handful of carriers. Pub/Sub delivers messages with the same ordering key one after another, so the parallel subscription had become a handful of serial streams. Separately, when a publish with an ordering key fails, the client library stops publishing for that key until the application resumes it, and the publishers never did. A transient error therefore silenced a carrier's events until the pod restarted.
Environment
Pub/Sub topics with message ordering enabled, Python publishers and subscribers on Kubernetes, shipment events from carrier integrations.
Approach
AceMQ looked at key distribution before touching any setting. A count of messages per ordering key showed the concentration immediately, and the publisher logs showed publish failures with no resume call afterwards. The changes were then tested against recorded event traffic in staging.
Solution
- Changed the ordering key from carrier ID to shipment ID, which keeps updates for one shipment in order and spreads load across many keys
- Added resume handling in the publishers, so a failed publish for a key is retried and the key is resumed instead of staying paused
- Handled publish errors by key, and logged the key and error so a stalled key is visible
- Matched subscriber flow control to the new concurrency so no single pod holds a large block of one key's messages
- Added a metric for oldest unacked message age and for messages published per ordering key
- Wrote down which flows genuinely need ordering and which can use an unordered subscription with a sequence number in the payload
Outcome
The subscription went back to keeping up with publishing, and a failed publish no longer silences a key. The team now treats ordering as a per-flow decision, with the key chosen for its cardinality.
Technologies
Google Cloud Pub/Sub
Kubernetes
Python
IBM MQ
RabbitMQ
Java
PostgreSQL
Terraform
OpenTelemetry
