Overview
A benefits administrator ran claims processing on MariaDB 10.6, which was about to pass community end of life. The team had postponed the upgrade because an earlier minor update had changed query plans without warning, and nobody wanted to repeat that on a regulated system.
Challenge
Moving from the 10.x series to 11.x changes the optimizer cost model, removes the InnoDB change buffer and, from 11.4, enables TLS on the server by default with clients required to verify the certificate. Each of those shows up as an application symptom, not a server error: a slower report, an unexpected tuning setting, a connection that suddenly fails.
Environment
Three-node MariaDB 10.6 Galera cluster on EC2 instances in a single AWS region, provisioned with Terraform, with asynchronous replicas for reporting.
Approach
AceMQ restored production into a staging copy and replayed a recorded query workload against both versions, comparing plans statement by statement. The rolling upgrade was then rehearsed node by node on that copy, including a rollback, before a production date was set.
Solution
- Captured the production slow log and a representative workload, then compared ANALYZE FORMAT=JSON output on 10.6 and 11.8
- Refreshed engine-independent statistics and fixed the small set of regressed queries with index changes
- Removed change buffer settings that no longer applied and reviewed the remaining InnoDB parameters
- Provisioned certificates and updated client connection strings so the 11.x TLS defaults did not break connections
- Rehearsed mariadb-upgrade node by node on the Galera copy, waiting for each node to resync before the next
- Documented the rollback path and the version freeze points in the Terraform modules
Outcome
The rehearsal found and fixed the regressed queries and the client connection failures before production was touched. The customer upgraded to 11.8 in a planned window with no application changes at cutover, ahead of the 10.6 end-of-life date.
Technologies
Terraform
Linux
Apache Spark
Redis Sentinel
