Registry deadlocks that only show up when everything restarts at once

Overview

WSO2 products lean heavily on shared registry and configuration databases. Under concurrent writes — a coordinated restart, a bulk API deployment, a large user store sync — lock contention in those tables can deadlock, and the visible symptom is a node that will not finish starting.

Challenge

After a planned maintenance restart, gateway nodes intermittently failed to complete startup, and bulk API deployments would hang partway. Database logs showed deadlocks on registry tables. The connection pool was sized well above what the database was configured to accept, so retry storms during startup made the contention worse rather than better.

Environment

WSO2 API Manager and Identity Server on-premises, sharing registry and user store databases on PostgreSQL, deployed across multiple nodes.

Approach

AceMQ correlated node startup sequences with database deadlock logs to identify the specific contended tables and the operations involved. We then staged node startup, corrected pool sizing against actual database limits, and moved bulk deployment to a controlled sequence rather than a parallel burst. AceMQ provides named senior engineers with 24/7 coverage and no tier-1 triage layer.

Solution

• Correlated startup sequences with database deadlock logs to identify contended registry tables
• Right-sized connection pools against the database's actual maximum connection limit
• Staged node startup so all nodes do not initialize registry state simultaneously
• Serialized bulk API deployment operations instead of issuing them in parallel
• Tuned database-side lock and statement timeouts so contention fails fast instead of hanging
• Documented a maintenance restart procedure that avoids reintroducing the contention

Outcome

Technologies