Overview
A typical engagement starts with a data platform team that runs several Flink jobs on Kubernetes, fed by Kafka. The jobs work, but upgrades are put off because nobody is sure the state will restore, and checkpoint alarms land on engineers who do not run Flink every day. Under Tier 3 managed services, AceMQ operates Flink inside the customer's own clusters.
Challenge
The Flink Kubernetes Operator automates deployment and upgrades, but it does not choose the upgrade mode for each job, judge checkpoint health, or upgrade itself: the operator's custom resource definitions are replaced by hand, and a new release can reject a Flink version it no longer supports. Jobs without explicit operator uids cannot safely change shape, and checkpoints that take longer every week go unnoticed until they time out.
Environment
Production and staging Kubernetes clusters in the customer's cloud account, the Flink Kubernetes Operator installed with Helm, several Application-mode jobs on a mix of Flink minor versions, RocksDB state on persistent storage, checkpoints in object storage, and Kafka as the main source and sink.
Approach
AceMQ starts with an assessment: Flink and operator versions against the operator's compatibility page, uids on every stateful operator, upgrade mode per job, checkpoint duration and size trends, and the restore history. It remediates the active risks, agrees access and a change process in the customer's accounts, rehearses a savepoint restore on a copy, then takes the pager. The work is delivered by AceMQ with partner Flink specialists.
Solution
- Inventory every job's Flink version, upgrade mode, state backend and uid coverage, and compare the versions with the operator's supported, deprecated and rejected lists
- Choose an upgrade mode per job: savepoint for planned changes, last-state as the recovery mode, stateless only where the job rebuilds from the source
- Plan the operator's own upgrade as a sequence: version checks, helm template and kubectl diff, CRD replacement by hand, then helm upgrade
- Put checkpoint duration, size, failure rate and backpressure by operator on dashboards with alerts that fire on the trend, not the timeout
- Rehearse a stop-with-savepoint and restore on a copy of one production job and time it against the customer's recovery objective
- Take over monitoring and on-call, with the 15-minute Sev 1 response and a written root-cause analysis after each P1
Outcome
The customer receives an inventory of jobs and versions, an upgrade mode and runbook for each job, a documented operator upgrade sequence, checkpoint and state dashboards, a rehearsed restore and a handover pack. Upgrades, patching and the pager move to AceMQ, and the data platform team returns to building pipelines. Clusters, data and the cloud bill stay with the customer.
Technologies
Apache Flink
Kubernetes
KafkaRocksDB
PostgreSQL
dbt
Prometheus
Docker
Debezium
