Back to all use cases
Manufacturing / AutomotiveSupportAny

One quiet Kafka partition can freeze every event-time window in the job

CV
Connected Vehicle Telematics Provider

Overview

Flink's watermark is the minimum across all input channels, so a single partition that stops producing records pins the watermark and every downstream event-time window stops firing. The job appears healthy — no exceptions, no backpressure, records still flowing — while output silently stops. AceMQ diagnoses this class of problem and the related watermark faults.

Challenge

Teams lose hours to this because every obvious health signal is green. The variants are similar and confusing: an over-partitioned Kafka topic where low-traffic partitions go idle overnight, an out-of-orderness bound set too generously so windows fire long after the data arrived, a per-partition watermark strategy applied after a rebalance so ordering guarantees are lost, and late records dropped without a side output so nobody knows data is missing.

Environment

Apache Flink jobs consuming from Kafka or Kinesis with event-time semantics, windowed aggregations, and interval joins.

Approach

AceMQ engineers compare per-subtask current watermark metrics against source partition activity, which makes an idle-source stall obvious within minutes. From there the watermark strategy is reviewed end to end — where it is assigned, whether idleness is configured, and whether the out-of-orderness bound matches measured event lateness rather than a guess.

Solution

  • 15-minute emergency SLA with named senior engineers, 24/7, no tier-1 triage
  • Per-subtask watermark metric comparison against source partition activity to isolate the stalling input
  • withIdleness configuration on the watermark strategy, sized against real quiet periods for low-traffic partitions
  • Out-of-orderness bound derived from measured event-time lateness rather than a default value
  • Watermark assignment moved to the source connector so per-partition ordering survives rebalance and repartitioning
  • Late-record side outputs and lateness metrics added so silent data loss becomes visible

Outcome

Stalled event-time windows resume, and the underlying watermark strategy stops depending on every partition staying busy. Adding late-record side outputs usually surfaces additional data loss the team had not known about.

Technologies

Apache FlinkApache KafkaKubernetes

Ready to Get Started?

Whether you need architecture advisory, 24/7 support, or full managed services, AceMQ has the expertise to help.

Contact Us