Tracing back-pressure from missing log lines to the grok pattern causing it

Overview

An energy utility had intermittent log gaps across their fleet with no obvious cause. The gaps traced back through Logstash to a small number of filter patterns consuming most of the available CPU.

Challenge

Several grok patterns had been written with leading wildcards and nested optional groups, which caused catastrophic backtracking on log lines that did not match the expected shape. A single malformed line could occupy a worker thread for a very long time. With workers tied up, the persistent queue filled, Logstash applied back-pressure to the Beats inputs, and the agents stopped shipping. When the queue eventually drained, agents resumed from their registry — but files that had rotated in the meantime were gone.

Environment

On-premises ELK stack with Filebeat agents across a large server fleet, Logstash tier performing parsing and enrichment.

Approach

We profiled the filter pipeline to find where time was actually spent rather than optimizing by inspection. Once the pathological patterns were identified, we rewrote them with anchored expressions and dissect where the format was fixed, then made the pipeline resilient so a single bad line degrades one event instead of stalling the fleet.

Solution

• Pipeline profiled per filter to identify the specific patterns consuming disproportionate CPU time
• Catastrophic-backtracking grok patterns rewritten with anchoring, and dissect substituted where the log format is fixed
• Timeout and tag-on-failure behavior configured so an unparseable line is tagged and passed through, never blocking a worker
• Persistent queue sizing and worker counts tuned against measured throughput rather than defaults
• Beats registry and file rotation settings adjusted so a back-pressure stall does not lose rotated files
• Queue depth and filter duration monitoring added so developing back-pressure is visible before gaps appear

Outcome

Technologies