Emergency Incident Response
Checkpoints failing cluster-wide, a job in a restart loop, or windows that stopped firing. With response targets from 15 minutes, by plan, a senior engineer joins a live bridge.
Stream processing support
Last reviewed by Tyler Eastridge
AceMQ supports Apache Flink 24/7 across Americas, EMEA and APAC on Kubernetes, YARN, standalone clusters and Amazon Managed Service for Apache Flink. Most calls start with one of three problems: checkpoints that time out under backpressure, state that grows until a TaskManager is killed, or a savepoint that will not restore after a deploy.
24/7 Apache Flink support with response targets from 15 minutes, by plan. Quote replies within 1 hour.
Planning a build, upgrade or migration instead? See AceMQ Apache Flink consulting and services.
Customers




Partners


Incident Triage
Seven Flink failures with the checkpoint metric, watermark value or configuration key we examine before restarting a job.
| Symptom you're seeing | What we check first |
|---|---|
| Checkpoints expire or fail, and the job keeps restarting | We read lastCheckpointDuration, lastCheckpointSize and numberOfFailedCheckpoints from the REST API or your metrics reporter, then split the time into start delay, alignment and asynchronous upload. A long start delay means barriers are stuck behind backpressure (look at backPressuredTimeMsPerSecond against busyTimeMsPerSecond per task); a long upload means slow state storage. Typical fixes are buffer debloating (taskmanager.network.memory.buffer-debloat.enabled), unaligned checkpoints, a sane execution.checkpointing.timeout and execution.checkpointing.min-pause, and execution.checkpointing.tolerable-failed-checkpoints so one slow checkpoint does not fail the job. |
| TaskManager killed by YARN or Kubernetes while the heap looks fine | RocksDB allocates native memory, so the container limit is hit outside the JVM heap. We compare taskmanager.memory.managed.fraction and state.backend.rocksdb.memory.managed with the container limit and the state size per slot, and check whether incremental checkpoints are on. |
| Event-time windows stop firing | We check currentInputWatermark per subtask. One idle Kafka partition holds the minimum back for the whole job. The fix is WatermarkStrategy.withIdleness(...) on the source, or watermark alignment where sources run at different speeds. We also count numLateRecordsDropped to see whether lateness settings are silently discarding data. |
| Kafka lag looks wrong | The Flink Kafka source commits offsets back to Kafka only when a checkpoint completes, so a stalled checkpoint makes lag grow in Kafka tools even when records are flowing. We compare Kafka's consumer group lag with Flink's currentEmitEventTimeLag and records-lag-max before changing anything. New partitions go unread until partition.discovery.interval.ms is set. |
| Exactly-once to Kafka stops committing | An exactly-once Kafka sink opens a transaction per checkpoint. If transaction.timeout.ms is shorter than the checkpoint interval plus restart time, or higher than the broker's transaction.max.timeout.ms, the transaction aborts and the job fails on restore. Consumers downstream must use isolation.level=read_committed or they will still see duplicates and aborted data. |
| A savepoint will not restore after a release | We compare operator UIDs between the savepoint and the new job graph, then check state schema changes and removed operators (--allowNonRestoredState is a decision, not a default). On the Flink Kubernetes Operator we check job.upgradeMode: savepoint, last-state and stateless have different failure modes. |
| JobManager failover loses the job | We check high-availability.type, the HA storage directory and the leader election records, and the restart strategy (restart-strategy.type) so a flapping dependency does not burn through restarts in a minute. |
Each P1 report records the checkpoint, state or connector setting at fault and the change that keeps the job from repeating it.
Deployment coverage
| Deployment | What AceMQ covers | What the platform vendor covers | Excluded |
|---|---|---|---|
| Flink on Kubernetes (Flink Kubernetes Operator or native mode) | Job and cluster incidents, checkpoint and state tuning, HA configuration, operator upgrade modes, upgrade planning | The Kubernetes distribution vendor supports the platform | Cluster control-plane faults; operators nobody maintains |
| Flink on YARN, including EMR | Same job-level scope; container sizing against YARN limits; classpath and dependency conflicts | The Hadoop or EMR vendor supports the cluster and YARN | Hadoop and YARN faults outside the Flink session |
| Standalone and session clusters | Incident response, HA, resource profiles, TaskManager sizing | No vendor; the open-source project has no support contract | Hardware and OS faults outside the agreed scope |
| Amazon Managed Service for Apache Flink | The application layer: parallelism, checkpoint and snapshot behaviour, connector and sink settings, state compatibility on upgrade | AWS runs the infrastructure, scaling and automatic state backup; you manage application code, IAM roles, VPC access, parallelism and snapshot retention (AWS Managed Service for Apache Flink FAQ, read 17 Nov 2026) | AWS platform faults; limits set by the service |
| Flink inside another vendor's streaming platform | The Flink job logic and how it behaves against your Kafka or object store | That vendor supports what it ships | Product defects stay with the vendor; we work the escalation with you |
Support options
Apache Flink is an Apache Software Foundation project with no vendor support contract. Enterprise support comes from a platform vendor, if you run Flink on a managed service, or from an independent provider for the Flink you operate yourself.
What's Included
Checkpointing, state backends, savepoints, Kafka connectors, job manager high availability and the Table API are all supported in one contract.
Checkpoints failing cluster-wide, a job in a restart loop, or windows that stopped firing. With response targets from 15 minutes, by plan, a senior engineer joins a live bridge.
Every P1 closes with a written RCA: what failed, why, the fix applied, and the configuration change that prevents recurrence.
RocksDB configuration, state TTL policy, and checkpoint interval tuning matched against your actual state size and throughput.
Source and partition design review to catch idle-partition watermark stalls before they silently freeze every downstream window.
Connector-by-connector review of whether your sinks actually support two-phase commit, so 'exactly-once' in the config matches exactly-once in production.
Flink version upgrades with savepoint compatibility testing, and rescaling strategy design for jobs that need to grow without a lossy restart.
Why AceMQ
The same senior engineers stay on your account. They know your state size, your checkpoint interval, and your watermark strategy — so a P1 starts with the checkpoint history, not you re-explaining your job graph.
You reach a senior Flink engineer directly by phone, email, or Slack. No help desk, no escalation approval process standing between you and a fix.
We support Flink alongside Spark, which means we think in the same terms for streaming and batch compute rather than treating them as unrelated disciplines.
Flink problems are frequently disk I/O, network, or Kafka-side problems wearing a checkpoint timeout. We diagnose across the whole path, including the state backend's underlying storage.
Round-the-clock coverage for customers in 26+ countries. Your 3am restart loop is someone's mid-afternoon — no overnight skeleton crew.
Quarterly health checks on state growth trends and checkpoint duration so a slow creep toward failure gets caught before it becomes an incident.
Flink on Kubernetes with the Flink Kubernetes Operator, YARN, standalone clusters, EMR and Amazon Managed Service for Apache Flink, with RocksDB state.
Version coverage
| Flink line | Latest release listed | Date | Note |
|---|---|---|---|
| 2.3 | 2.3.0 | 25 June 2026 | Latest stable release |
| 2.2 | 2.2.1 | 15 May 2026 | |
| 2.1 | 2.1.3 | 14 June 2026 | |
| 2.0 | 2.0.2 | 11 May 2026 | Removed the DataSet API and older APIs |
| 1.20 | 1.20.5 | 3 June 2026 | Labelled LTS on the Flink downloads page |
Source: Flink downloads page, read 17 November 2026. We also support older 1.x lines in production; the upgrade path is planned from your current version with a savepoint rehearsal.
Escalation Path
The engineer who answers reads checkpoint durations and per-subtask watermarks directly, so the call opens with the barrier, not a tour of your job.
Plan decides the response clock: a job that cannot checkpoint in production comes before an operator design question.
Pricing
Flink support typically costs $24,000–72,000 a year. The number of production jobs, state size, whether you deploy with the Kubernetes Operator or on YARN, and the response plan determine where a contract lands.
Prices are in US dollars (USD). Response targets begin at 15 minutes by plan, 24/7, and we answer quote requests within 1 hour. Contracts covering other technologies earn a multi-technology discount.
Support In Practice
Typical Apache Flink support, remediation and consulting engagements. Each card links to the full write-up: the streaming jobs, what the engagement covers and what it delivers.
Global Payments Processor
Resolving checkpoint timeouts under backpressure where RocksDB state has grown past what the configured checkpoint interval can absorb.
Connected Vehicle Telematics Provider
Diagnosing event-time windows that stop firing because a single idle source partition holds the watermark back across the whole job.
Smart Grid Technology Provider
Evaluating whether a proposed streaming workload belongs on Flink, and what the exactly-once, state, and operational requirements will really cost.
Data Platform Team
A typical Tier 3 engagement: AceMQ runs Apache Flink jobs on the customer's own Kubernetes clusters with the Flink Kubernetes Operator, owning checkpoint health, savepoint-based upgrades and the pager.
Regional Insurer
What a platform review covers when several teams want streaming and nobody has decided which broker, processor and operating model the company will run.
Online Fraud Detection Platform
Designing state backend, key partitioning, and rescaling strategy for large-state Flink jobs that must restart without hours of downtime.
Support options
AceMQ offers Apache Flink support in four tiers, each handing more of the day-to-day work to AceMQ: 24/7 support and incident management for jobs your team runs, proactive monitoring that catches checkpoint, state and CVE problems first, managed Flink on your own infrastructure, and fully managed Flink on AceMQ infrastructure. Kubernetes, YARN and standalone deployments are covered in all four.
Your engineers own the jobs, we take escalations
Best for: Teams writing Flink applications who want an engineer for checkpoint, watermark and state problems they cannot untangle.
Reviews of checkpoints, state and watermarks
Best for: Teams that see state growth and checkpoint duration creep only once a job has started restarting.
We operate Flink on your clusters
Best for: Organizations that keep Flink on Kubernetes, YARN or their own hosts and want specialists handling upgrades and recovery.
AceMQ runs your Flink platform as a service
Best for: Teams that want streaming jobs running reliably without staffing Flink cluster operations.
One partner for Apache Flink
The same team supports Kafka and the streaming neighbors around Flink, and publishes comparisons of stream processing options.
The Apache Flink project itself does not sell support. Managed services such as Amazon Managed Service for Apache Flink cover the platform, and AceMQ covers your jobs, state and connectors on any deployment.
Usually backpressure delays the checkpoint barrier, or state upload is slow. We split the checkpoint into start delay, alignment and upload time, then tune buffer debloating, unaligned checkpoints and the timeout settings against what we find.
The Flink Kafka source commits offsets on checkpoint completion, not per record. If checkpoints stall, committed offsets freeze and lag appears. Flink's own event-time lag metrics show the true position.
Consumers must read with isolation.level=read_committed, and the producer transaction timeout must exceed the checkpoint interval plus restart time while staying under the broker limit.
An idle input partition holds back the watermark. Add idleness handling to the watermark strategy or use watermark alignment, then watch per-subtask watermarks.
Usually yes. We take a savepoint, restore it into the target version in a test environment, check operator UIDs and state schemas, and run the real cutover with a rollback savepoint. Where state cannot migrate, we plan a dual run.
Yes, at the application layer. AWS runs the infrastructure and backs up state automatically; we work on parallelism, snapshot behaviour, connectors and what your code does with state.
We trend checkpoint duration and size, find backpressure by operator, measure state per key group, and review watermark progress, restart strategy and high-availability settings, then return ranked fixes for each job.
Yes. We read the execution plan, compare state size per operator, and look at join and aggregation behavior, then adjust mini-batch settings or rewrite queries so state stays bounded.
Yes. A checkpoint investigation, a watermark and idleness review, an upgrade rehearsal with savepoints or a job health check can each be scoped as separate work. Teams with recurring incidents normally move to an annual agreement with 24/7 response and named engineers.
Plans run from $24,000 to $72,000 a year (USD). We consider the number of production jobs, typical state size, the deployment model, the Kafka or other sources feeding them, and your response plan. Send a short description and we answer with a quote within 1 hour.
Let's talk
Send us the job graph and the last checkpoint history. We will tell you in one business hour whether the problem is state, backpressure or a sink.
Production incident right now?Email an Apache Flink Architect