An ingest architecture where a downstream stall is a delay, not a data loss event
Elasticsearch
KubernetesElasticsearch maintenance no longer affects log producers — the buffer absorbs the window and drains afterward. Adding a new service to the pipeline went from a change to shared configuration to an in…
Overview
A commerce platform's ELK ingest was fragile: any Elasticsearch slowdown propagated back to the applications producing logs. AceMQ designed a buffered architecture that decouples producers from the search cluster.
Challenge
Beats shipped directly into a Logstash tier that wrote synchronously to Elasticsearch, so any indexing slowdown became back-pressure all the way to the log producers. During cluster maintenance the only options were to accept data loss or block applications. On top of that, every team had its own field naming, so correlating a request across services meant knowing three different names for the same identifier, and the parsing logic to handle them all lived in one increasingly unmaintainable Logstash configuration.
Environment
ELK on Kubernetes ingesting from dozens of services, with growth in both log volume and contributing teams.
Approach
We introduced a durable buffer between production and indexing so the two can fail independently, moved simple parsing to Elasticsearch ingest pipelines where it belongs, and standardized on a common schema so per-service parsing logic largely disappears. The design was staged so teams migrate incrementally rather than through a single cutover.
Solution
- 1Durable message queue introduced between shippers and the indexing tier so cluster maintenance causes delay, not loss
- 2Simple parsing and enrichment moved to Elasticsearch ingest pipelines, leaving Logstash only where its capabilities are needed
- 3Common schema adopted for shared fields so correlation across services stops requiring per-team field knowledge
- 4Per-service pipeline definitions split into owned, independently deployable units instead of one monolithic configuration
- 5Dead letter handling defined so unparseable events are captured for review rather than dropped or blocking
- 6Staged migration plan letting teams move service by service with dual-path operation during transition
Outcome
Elasticsearch maintenance no longer affects log producers — the buffer absorbs the window and drains afterward. Adding a new service to the pipeline went from a change to shared configuration to an independent, owned deployment.
Technologies
ElasticsearchKubernetesRelated Use Cases
ELK Stack Log Volume and Retention Assessment
Assessment of log volume, field-level utility, and retention across an ELK estate where storage growth had outpaced any plan for it.
ELK Stack Logstash Back-Pressure Remediation
Remediation of an ELK pipeline where Logstash back-pressure stalled Beats agents and left log gaps across the fleet.
Need ELK Stack Architecture Guidance?
AceMQ's senior ELK Stack engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.