24/7 replica set support with named senior engineers and no tier-1 triage

Overview

An analytics provider ran nightly ETL loads that generated far more oplog volume than steady-state traffic. Any secondary that fell behind during the load window rolled off the oplog and dropped into a full initial sync, which then competed with the next night's load. AceMQ took over replica set support with named senior engineers and direct escalation.

Challenge

The oplog was sized from the default percentage of disk at cluster creation and had never been revisited. Bulk updates that rewrote whole documents amplified oplog volume well beyond the size of the changed fields, and a single lagging secondary could trigger a resync cascade that left the set without a healthy read replica for most of a business day.

Environment

Managed MongoDB replica sets on AWS with nightly batch ingestion from an upstream clinical data warehouse.

Approach

AceMQ measured actual oplog churn during a full batch window rather than relying on averages, then sized the oplog to cover the worst observed window with headroom. In parallel, engineers reworked the batch writes to reduce oplog amplification and established a runbook so on-call staff could recover a lagging secondary without a full resync.

Solution

• Measured peak oplog generation rate across a complete batch window to size the oplog against worst case, not average
• Resized the oplog online and validated the resulting replication window under load
• Converted whole-document rewrites in the ETL to targeted field updates to cut oplog amplification
• Staggered batch load concurrency so secondaries could keep pace with primary write throughput
• Wrote a recovery runbook covering lagging secondaries, rollback files, and when a resync is genuinely required
• Set up replication lag and oplog window alerting tied to the 24/7 escalation path

Outcome

Technologies