Overview
A typical engagement starts with a business application on several Tomcat nodes behind a load balancer. Users see intermittent 502 errors and occasional hangs at peak, restarting a node clears it for a day, and nobody has measured what the request threads wait on.
Challenge
Tomcat's HTTP connector ships with defaults the team has never revisited: 200 request threads, an accept queue of 100, 8192 connections on the NIO connector, and a keep-alive timeout that follows connectionTimeout unless it is set. If Tomcat closes idle connections before the load balancer does, the balancer reuses a connection that is already gone and returns a 502. Separately, when a slow database call holds threads, a database pool sized below the thread count turns every spike into a queue.
Environment
Three Tomcat 9 nodes on virtual machines behind a load balancer, a relational database with a connection limit, session state held in memory on each node, and application and access logs kept for a few days.
Approach
AceMQ engineers read server.xml and the load balancer's idle timeout side by side, take thread dumps during a hang and a heap histogram before and after a redeploy, and compare the database pool size with the connection limit. Changes are tried on one node first, with the load balancer's error rate as the measure. The session options are compared against how users actually log in, and a failover drill checks that a node restart does not log everyone out.
Solution
- Timeouts compared: connectionTimeout and keepAliveTimeout on Tomcat against the load balancer's idle timeout, so Tomcat does not close first
- Thread dumps during a hang, read for what the request threads wait on
- Database pool size set against the thread count and the database's connection limit
- Forwarded-header handling and the AJP connector reviewed
- Session options compared: sticky sessions, replication and an external store
- Failover drill with clients connected and a runbook for node replacement
Outcome
The engagement delivers a written review with ranked changes, the tested connector and pool settings, a session recommendation, the failover drill result and a runbook. The team keeps the thread dump procedure so the next hang is read in minutes. Application code changes, if the dumps point there, stay with the customer's developers.