Closing the telemetry gaps that live below the vendor's support boundary
Datadog
KubernetesLinuxOpenTelemetryTelemetry gaps for short-lived workloads stopped entirely, and peak-window loss went to effectively zero once the egress policy was corrected. The customer can now distinguish a real service outage fr…
Overview
A logistics provider had metrics and traces disappearing intermittently for short-lived workloads. The vendor's support confirmed nothing was wrong on the platform side, which left the customer's own agent and infrastructure layers as the place to look. AceMQ works those layers.
Challenge
Short-lived batch pods were terminating before the agent flushed its buffer, so their final metrics and any traces from the last seconds of execution never arrived. Separately, a network policy change had started dropping a portion of egress traffic during peak windows, and the agent's retry behavior masked it as intermittent rather than systematic. The two problems looked like one flaky symptom, which is why it had gone unresolved for months.
Environment
Datadog agents deployed as a DaemonSet on Kubernetes with high pod churn, egress through a shared network path with policy enforcement.
Approach
AceMQ does not operate the vendor's platform — we work the layers the customer owns and the vendor's support will not investigate. We instrumented the agent's own telemetry, correlated gap windows against pod lifecycle events and egress metrics, and separated the two overlapping causes before fixing either.
Solution
- 1Agent internal telemetry enabled and collected so flush failures and dropped payloads became directly visible
- 2Gap windows correlated against pod termination events, isolating short-lived workloads as a distinct failure class
- 3Pod termination grace period and agent flush behavior adjusted so batch workloads report their final data before exit
- 4Egress path instrumented to expose the peak-window drops that agent retries had been masking
- 5Network policy corrected and agent retry and backoff settings tuned so genuine drops surface as errors instead of silence
- 6Alerting added on agent-reported drop counts so future telemetry loss is detected rather than discovered later
Outcome
Telemetry gaps for short-lived workloads stopped entirely, and peak-window loss went to effectively zero once the egress policy was corrected. The customer can now distinguish a real service outage from a collection failure, which was previously guesswork.
Technologies
DatadogKubernetesLinuxOpenTelemetryRelated Use Cases
Datadog Monitor and Alert Noise Support
Ongoing support for a Datadog monitor estate producing more alerts than the on-call rotation could meaningfully act on.
Datadog Custom Metric and Ingest Cost Governance
Consulting engagement to govern custom metric cardinality, log ingest, and trace volume so observability spend tracks value instead of accident.
Facing a Datadog Production Issue?
AceMQ's senior Datadog engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.