An ingest architecture where a downstream stall is a delay, not a data loss event

Overview

A commerce platform's ELK ingest was fragile: any Elasticsearch slowdown propagated back to the applications producing logs. AceMQ designed a buffered architecture that decouples producers from the search cluster.

Challenge

Beats shipped directly into a Logstash tier that wrote synchronously to Elasticsearch, so any indexing slowdown became back-pressure all the way to the log producers. During cluster maintenance the only options were to accept data loss or block applications. On top of that, every team had its own field naming, so correlating a request across services meant knowing three different names for the same identifier, and the parsing logic to handle them all lived in one increasingly unmaintainable Logstash configuration.

Environment

ELK on Kubernetes ingesting from dozens of services, with growth in both log volume and contributing teams.

Approach

We introduced a durable buffer between production and indexing so the two can fail independently, moved simple parsing to Elasticsearch ingest pipelines where it belongs, and standardized on a common schema so per-service parsing logic largely disappears. The design was staged so teams migrate incrementally rather than through a single cutover.

Solution

• Durable message queue introduced between shippers and the indexing tier so cluster maintenance causes delay, not loss
• Simple parsing and enrichment moved to Elasticsearch ingest pipelines, leaving Logstash only where its capabilities are needed
• Common schema adopted for shared fields so correlation across services stops requiring per-team field knowledge
• Per-service pipeline definitions split into owned, independently deployable units instead of one monolithic configuration
• Dead letter handling defined so unparseable events are captured for review rather than dropped or blocking
• Staged migration plan letting teams move service by service with dual-path operation during transition

Outcome

Technologies