A stable NGINX estate is mostly a matter of what you can see. When a customer calls with an outage, the questions below decide how quickly we can find the cause, and we put them in place during onboarding.
An access log format that carries the upstream. Add $upstream_addr, $upstream_status, $upstream_response_time, $request_time and $upstream_connect_time to the log format. With those, a slow request splits into time spent connecting, time the backend took and time spent sending the response to a slow client. Without them, every slow request looks like an NGINX problem.
Error log at the right level. The warn level shows upstream timeouts and connection failures. info is noisy. debug needs a debug-enabled binary and should be turned on for a single client address only (debug_connection).
A stub_status endpoint, or NGINX Plus API metrics. Active connections, reading, writing and waiting show whether you are near the connection ceiling. Scraping them into your monitoring gives a trend, not only a snapshot.
A configuration test in the deploy path. nginx -t before every reload catches syntax errors, but not logic errors such as a location block shadowing another. Reviewing the output of nginx -T, which prints the full effective configuration including includes, is how we find those. We keep that output in version control after each change so a diff shows what moved.
A rollback you have tested. A reload that fails leaves the old configuration running. A reload that succeeds with a bad configuration serves errors. Keeping the last known good configuration, and testing the rollback, turns a long outage into a short one.
We also check three platform settings that cause intermittent trouble: the number of worker processes against cores (worker_processes auto in most cases), the keepalive timeout to clients and upstreams so connections are reused without hoarding, and the TLS session cache so repeat handshakes are cheap. None is exciting. Each has turned up in a P1 root cause analysis.