Closing the telemetry gaps that live below the vendor's support boundary

Overview

A logistics provider had metrics and traces disappearing intermittently for short-lived workloads. The vendor's support confirmed nothing was wrong on the platform side, which left the customer's own agent and infrastructure layers as the place to look. AceMQ works those layers.

Challenge

Short-lived batch pods were terminating before the agent flushed its buffer, so their final metrics and any traces from the last seconds of execution never arrived. Separately, a network policy change had started dropping a portion of egress traffic during peak windows, and the agent's retry behavior masked it as intermittent rather than systematic. The two problems looked like one flaky symptom, which is why it had gone unresolved for months.

Environment

Datadog agents deployed as a DaemonSet on Kubernetes with high pod churn, egress through a shared network path with policy enforcement.

Approach

AceMQ does not operate the vendor's platform — we work the layers the customer owns and the vendor's support will not investigate. We instrumented the agent's own telemetry, correlated gap windows against pod lifecycle events and egress metrics, and separated the two overlapping causes before fixing either.

Solution

• Agent internal telemetry enabled and collected so flush failures and dropped payloads became directly visible
• Gap windows correlated against pod termination events, isolating short-lived workloads as a distinct failure class
• Pod termination grace period and agent flush behavior adjusted so batch workloads report their final data before exit
• Egress path instrumented to expose the peak-window drops that agent retries had been masking
• Network policy corrected and agent retry and backoff settings tuned so genuine drops surface as errors instead of silence
• Alerting added on agent-reported drop counts so future telemetry loss is detected rather than discovered later

Outcome

Technologies