Overview
This is a typical engagement for a mid-sized business that runs a core policy or billing system on Oracle Database Enterprise Edition with a physical standby in a second site. The standby was built years ago, and nobody on the current team has performed a failover. AceMQ and a partner Oracle DBA review whether the design would work on the day it is needed.
Challenge
Data Guard rarely fails at the moment of failover. It fails because of choices made earlier: a protection mode that allows more data loss than the business thinks, standby redo logs sized for an old redo rate, a lag alert nobody receives, and applications that connect to a fixed host name instead of a role-based service. The team believes they have disaster recovery, and the evidence is a configuration that shows SUCCESS in the broker.
Environment
Oracle Database 19c Enterprise Edition on Linux, a primary in one data center and a physical standby in another, RMAN backups to disk, and application servers that connect through a connection pool.
Approach
The review starts read-only and written findings come before any change. AceMQ compares what the configuration says with what the redo stream, the standby and the applications actually do, and measures a failover against the recovery targets the business states.
Solution
- Reviews the protection mode and redo transport settings against the recovery point the business has agreed
- Measures transport lag and apply lag over a normal business cycle and a heavy batch window
- Checks standby redo log size and count against the redo generation rate
- Checks whether Flashback Database is enabled so a failed primary can be reinstated as a standby
- Reviews how applications connect: role-based services, connection failover and retry behavior
- Writes a failover and switchover runbook and plans a rehearsal in a change window
Outcome
The engagement delivers a written readiness report that ranks each gap by the data loss or downtime it would cause, a tested failover and switchover runbook, lag and failover alerts routed to people who will act on them, and a rehearsal plan. The business learns whether its recovery targets are met by the design it owns.
Technologies
Linux
Kafka
Kubernetes
