Redis

Redis Troubleshooting: Production Issues and How to Fix Them

A

AceMQ Engineering Team

Redis Consulting & Support

Redis Troubleshooting: Production Issues and How to Fix Them

Most Redis production incidents come down to five things: memory hitting maxmemory, one O(N) command blocking the single-threaded event loop, eviction quietly deleting keys the application still needed, replication falling behind and forcing a full resync, or connections exhausting the client limit. Check them in that order before reading a log file.

Effective Redis troubleshooting is mostly about checking the right thing first. Below are the failures we see most often on enterprise estates, each with the command that confirms it rather than a description you have to translate into one.

Redis troubleshooting starts with three commands

Redis troubleshooting goes wrong when people reach for logs first. Redis logs are sparse by design — most of the failures below leave no log line at all. The server state does.

INFO                       # role, uptime, clients, persistence, replication
INFO memory                # used_memory, maxmemory, fragmentation
SLOWLOG GET 25             # commands that exceeded the slow-log threshold

Three questions, in this order:

  1. Is it out of memory? used_memory at or near maxmemory explains rejected writes, evictions, and a good share of "Redis is slow" reports.
  2. Did one command block the server? Redis executes commands on a single thread. One slow command stalls every client, and the slow log is the only place that fact is recorded.
  3. Is this node in the state you think it is? A replica that was promoted, or a primary that quietly became a replica after a failover, produces errors that look nothing like their cause.

Resist restarting. A restart clears the slow log, resets INFO counters, and discards the latency time series — converting a diagnosable problem into an anecdote.

1. Why does Redis return "OOM command not allowed"?

Problem. Writes fail with OOM command not allowed when used memory > 'maxmemory'. Reads still work, so monitoring says the service is up while the application is half-broken.

CONFIG GET maxmemory
CONFIG GET maxmemory-policy

If maxmemory-policy is noeviction, this is Redis doing exactly what it was told: refuse writes rather than delete data. Correct for a datastore, wrong for a cache.

Fix. Decide which one you are running. For a cache, set an eviction policy and let Redis reclaim space — we break the eight options down in Redis eviction policy. For a datastore, the error is a capacity signal; raising maxmemory on a host that cannot back it moves the failure from a clean error to a kernel OOM kill.

One trap ends more Redis troubleshooting sessions than it should: maxmemory set to 0 means no limit at all. An unbounded Redis grows until the kernel kills it, and the kernel does not send a polite error first.

2. Why is Redis slow when CPU and memory look fine?

Problem. Latency spikes with no memory pressure, no swap, and a mostly idle CPU. The cause is nearly always the same: Redis processes commands on one thread, so an O(N) command over a large collection holds that thread while every other client waits.

SLOWLOG GET 25
CONFIG GET slowlog-log-slower-than     # microseconds

The slow log records execution time only — it excludes time spent talking to the client — so an entry means the server genuinely blocked for that long. Each entry carries the duration in microseconds, the command arguments, and the client address and port, which is usually enough to identify the offending service without instrumenting it.

The usual offenders: KEYS *, SMEMBERS on a large set, HGETALL on an oversized hash, unbounded LRANGE key 0 -1, and long Lua scripts, which run atomically and block for their whole body.

Fix. Replace whole-collection reads with cursor-based iteration — SCAN, SSCAN, HSCAN, ZSCAN — which return bounded chunks instead of one blocking call. Then find out how the collection got that large:

redis-cli --bigkeys      # biggest keys by element count
redis-cli --memkeys      # biggest keys by bytes

Both walk the keyspace with SCAN, not KEYS, so they are safe in production. Add -i 0.01 to throttle them on a loaded instance.

Before you go further, turn latency monitoring on. latency-monitor-threshold defaults to 0, which disables the subsystem entirely — so LATENCY DOCTOR reports nothing and people conclude the tool is broken. It is the most common dead end in Redis troubleshooting.

CONFIG SET latency-monitor-threshold 100     # milliseconds

Redis then records spikes against named events: command, fork, expire-cycle, eviction-cycle, the aof-* family, and active-defrag-cycle. When the incident recurs, LATENCY LATEST and LATENCY DOCTOR tell you which one. LATENCY LATEST naming fork rather than command is the single most useful reading in this article, because it says the problem is persistence, not your queries.

3. Why does Redis stall during backups?

Problem. Regular, rhythmic latency spikes, often correlated with a backup window.

INFO persistence     # rdb_bgsave_in_progress, latest_fork_usec, rdb_last_bgsave_status
LATENCY HISTORY fork

Redis snapshots by forking a child process. That fork is the stall — fork(2) has to copy the process page tables, and the Redis documentation is explicit that it "can be time consuming if the dataset is big, and may result in Redis stopping serving clients for some milliseconds or even for one second if the dataset is very big and the CPU performance is not great." latest_fork_usec gives you the last fork duration. Hundreds of thousands of microseconds there is your latency.

Fix, in order of how often it is the right one: move snapshots to a replica so the fork lands on a node nobody queries; check the host, since fork cost is dominated by page-table size and by the hypervisor; or drop RDB entirely on an instance whose data is rebuildable. Persistence is the answer to more Redis troubleshooting tickets than query tuning is.

The fork also has a memory consequence that kills instances outright. Copy-on-write duplicates every page modified during the save, so process memory climbs toward double the dataset under write load. We investigated repeated crashes on a customer's cluster where reported utilisation never exceeded roughly 60 percent — the fork was the whole story, and the fix was kernel memory-overcommit tuning rather than any Redis setting. The same mechanism underpins the failures in Redis high availability.

Get help with Redis latency and crashes

Seeing Redis latency or crashes your metrics do not explain? These usually sit below Redis — in fork behaviour, allocator fragmentation, or the hypervisor, which is where Redis troubleshooting stops being a Redis exercise. AceMQ runs structured assessments that find them, and provides enterprise Redis support for production estates. Talk to an AceMQ engineer.

4. Why are keys disappearing before their TTL?

Problem. Keys vanish early, and the application starts writing defensive code around what it assumes is a data-integrity bug.

INFO stats     # evicted_keys, expired_keys, keyspace_hits, keyspace_misses

evicted_keys climbing means the instance is at its limit and reclaiming under the configured policy. expired_keys climbing is normal TTL behaviour and is not the same thing.

Fix. Rising evictions mean the instance is undersized, the policy is wrong for the access pattern, or a volatile-* policy is set on a keyspace where nothing carries a TTL — in which case it has no eviction candidates and behaves like noeviction while your config says otherwise.

Before tuning: redis-cli --hotkeys identifies frequently accessed keys, but it only works when maxmemory-policy is an *-lfu policy, because it reads the LFU counters. On an LRU instance it will not give you what you want.

5. Why is my replica lagging or resyncing?

Problem. Replicas fall behind, then perform a full resynchronisation — which forks the primary and produces a latency spike at the worst possible moment. Run on both nodes:

INFO replication     # role, master_link_status, master_repl_offset, connected_slaves

Compare the replica's offset against master_repl_offset on the primary, and confirm master_link_status is up.

Fix. Redis replicas reconnect and attempt a partial resync, replaying only the stream they missed — but only while that data is still in the primary's replication backlog. Exhaust the backlog through a long disconnection or a replica that cannot keep up, and Redis falls back to a full resync: background save, full dataset transfer, reload.

So the lever is the backlog. Size repl-backlog-size for the worst disconnection you expect to survive, not the average one. On a WAN or cross-AZ link the default is frequently an order of magnitude too small, and the symptom is a resync storm that looks like a network fault.

Two details catch teams out. Replicas ignore maxmemory by default — the primary drives eviction via replicated DEL, so a replica can exceed the primary's limit and hit a real out-of-memory condition. And a replica shut down with SHUTDOWN can partially resync; one that is killed cannot.

6. Why do clients get connection errors?

Problem. ERR max number of clients reached, or connection timeouts, against a healthy-looking server.

INFO clients               # connected_clients, blocked_clients
CONFIG GET maxclients      # read it — do not assume the default
CLIENT LIST                # per-connection age, idle time, last command

Do not trust a remembered default for maxclients. Redis lowers it at startup if the operating system file-descriptor limit cannot support the configured value, so the effective number on a containerised deployment is often far below what the config file says.

Fix. CLIENT LIST distinguishes two shapes. Thousands of connections with low age and high idle means the application opens a connection per request and needs a pool — an application fix, not a Redis one. A few connections with a huge age, one of them blocked, means something is holding a connection through BLPOP or similar and never releasing it.

If Redis is fronting a queue workload and you are seeing connection churn under load, the real question is whether Redis belongs on that path at all. We cover the trade-off in running Redis and RabbitMQ together.

7. Why does my Redis Cluster fail after a reshard?

Problem. CLUSTERDOWN Hash slot not served, or clients bouncing between MOVED and ASK redirections after a slot migration.

CLUSTER INFO                          # cluster_state, cluster_slots_assigned
CLUSTER SHARDS                        # slot ranges per shard
redis-cli --cluster check <host>:<port>

cluster_state:ok requires every one of the 16,384 hash slots to be assigned. An interrupted reshard leaves slots migrating or importing, and the cluster refuses to serve rather than serve inconsistently.

Fix. Let --cluster check report the open slots before touching anything, and use redis-cli --cluster fix to complete the migration rather than hand-assigning slots. Manual CLUSTER SETSLOT on a cluster you do not fully understand is how a stalled reshard becomes lost data.

Also confirm clients use cluster-aware mode (-c in redis-cli). A non-cluster client works fine against a cluster node until the first reshard moves a slot, then fails on exactly the keys that moved — a Redis troubleshooting case that looks like data loss and is not.

8. Why does the app time out when Redis is fast?

Because the timeout is not Redis. Measure the two halves separately:

redis-cli --latency                     # round-trip PING latency
redis-cli --intrinsic-latency 5         # host scheduler latency — run ON the Redis server

--intrinsic-latency does not touch Redis. It measures what the kernel and hypervisor can do, and must run on the machine hosting Redis. If intrinsic latency is already in the milliseconds, no Redis tuning will get you below it — the problem is the host, and on a virtualised deployment that usually means CPU steal.

If round-trip latency is sub-millisecond and the application still times out, the queue is client-side: pool exhaustion, a synchronous call on an event loop, or a client timeout set below the actual command duration.

When to escalate a Redis troubleshooting case

Most Redis troubleshooting is in-house work: eviction tuning, slow-command hunting, connection pooling, backlog sizing, snapshot scheduling.

Escalate when a wrong move costs more than a second opinion:

  • Repeated crashes with nothing in INFO or the logs that explains them
  • A cluster with unassigned slots after a failed reshard
  • Replicas that will not complete a sync no matter how many times they retry
  • Memory that keeps growing after FLUSHALL — a buffer or allocator problem, not a keyspace one
  • Any incident where you are about to force a state change without knowing which node holds authoritative data

Those share one trait: the evidence is destroyed by the obvious next step. AceMQ provides enterprise Redis support for production estates — incident response, cluster assessments, memory and latency forensics, and upgrade planning across open source Redis, Redis Enterprise, and managed cloud services.

FAQ

How do I start troubleshooting Redis?

Redis troubleshooting starts with INFO, then INFO memory and SLOWLOG GET 25. Those three answer whether the instance is out of memory, whether it is evicting, and whether one command has been blocking the event loop. Most incidents resolve there.

Why does Redis return OOM command not allowed?

The maxmemory limit is reached and maxmemory-policy is noeviction, so Redis rejects writes rather than deleting data. Reads keep working, which is why the application looks half-alive. Confirm with CONFIG GET maxmemory and CONFIG GET maxmemory-policy.

Why is Redis slow when CPU and memory look fine?

Almost always one O(N) command monopolising the single-threaded command loop. KEYS *, SMEMBERS on a large set, HGETALL on a huge hash, and unbounded LRANGE all block every other client until they finish. SLOWLOG GET names the offender.

Why is LATENCY DOCTOR empty?

Because latency monitoring is off by default — latency-monitor-threshold is 0. Run CONFIG SET latency-monitor-threshold 100 first, wait for the problem to recur, then run LATENCY DOCTOR. This is the most common dead end in Redis troubleshooting.

How do I find big keys in Redis safely?

redis-cli --bigkeys for element counts and redis-cli --memkeys for byte sizes. Both use SCAN rather than KEYS, so they are safe against a busy production instance. Add -i 0.01 to throttle the scan further.

How do I check Redis replication lag?

INFO replication on both nodes. Compare master_repl_offset on the primary with the replica's offset, and check master_link_status is up. A replica that falls far enough behind to exhaust the replication backlog forces a full resync, which forks the primary.

Why do clients get connection errors when Redis is running?

Either the client limit is reached — INFO clients versus CONFIG GET maxclients — or connections are leaking because the application opens a client per request instead of using a pool. CLIENT LIST shows age, idle time, and the last command per connection.

When should I escalate instead of restarting?

When a restart would destroy the evidence: repeated crashes with no cause in the metrics, a cluster with unassigned slots, replicas that will not sync, or unexplained data loss. Restarting first is how a recoverable incident becomes a permanent one.

Redis troubleshooting is mostly discipline: confirm the failure with a command before changing anything, and change one thing at a time.

Sources

Free Consultation

Get Expert Eyes on Your Redis Deployment

Whether you're troubleshooting a production incident, planning a migration, or want a second opinion on your architecture — our team is ready. No pitch, just answers.

Email Us