Overview
A typical engagement starts with a Zabbix server that monitors hosts, network devices and services for an operations team, and has begun to fall behind. Dashboards show old values, the queue of delayed items grows, and the database is busy all day. The team wants to know which of several possible causes is the real one.
Challenge
A server that falls behind can be limited by the database, the history cache, the pollers or the housekeeper, and each cause looks like the others from the dashboards. History and trends tables that have grown past what the retention policy needs make every one of them worse, and the Zabbix server health template raises its alerts only at fixed utilization lines. Without a measured order of causes, settings get changed one at a time in production.
Environment
A Zabbix 7.0 LTS server and several proxies on virtual machines, with a PostgreSQL database that has grown over several years, thousands of hosts across network, server and application templates, and a mix of agent, SNMP and HTTP checks.
Approach
AceMQ engineers read the server's internal metrics first: process utilization, history cache and value cache usage and the item queue. They compare table growth with the retention actually needed, check whether the housekeeper keeps pace or whether TimescaleDB partitions are being dropped, and review proxy versions against the server. Only then do they propose changes, and each one is ranked by production risk and tested on a copy where the change is not trivially reversible.
Solution
- Review of process utilization, history cache and value cache usage and the delayed-item queue against the Zabbix server health template
- History and trends sizing by item type, with retention compared to what dashboards and triggers actually read
- Housekeeper behavior review, including per-item storage overrides and, where TimescaleDB is used, the settings that decide whether old partitions are dropped
- Proxy versions checked against the server's compatibility rule, and proxies on SQLite flagged for history loss at upgrade
- Cache size, poller count and history syncer recommendations, each tied to a measured symptom
- Timing of the schema upgrade to the next LTS, rehearsed on a copy of the database
Outcome
The engagement delivers a written assessment the monitoring team can act on without AceMQ: the causes ranked by production risk, the retention and housekeeping changes recommended, the cache and poller settings with the measurement behind each, and a rehearsed timing for the next LTS upgrade.
Technologies
PostgreSQLTimescaleDB
Linux
Grafana
Kubernetes
Amazon S3
Terraform
