Redis Memory Problems: Why It Crashes and How to Size It

Redis crashes on memory for reasons the used_memory metric does not show: the RDB fork can nearly double resident memory under write load, allocator fragmentation inflates RSS above the data, and maxmemory only governs the keyspace — not replication buffers, client buffers, or the fork. Size for peak plus overhead, not for the dataset.

The pattern is consistent enough to be a signature. An instance sits at what looks like comfortable utilisation, nothing alerts, and then it dies. Nobody can reproduce it, because the Redis memory metric everyone is watching was never measuring the thing that killed it.

Less than most teams assume. The Redis documentation puts it plainly: to store user keys, "Redis allocates at most as much memory as the maxmemory setting enables (however there are small extra allocations possible)."

That sentence is doing a lot of work. maxmemory bounds the keyspace, which is only part of total Redis memory. It does not bound:

• The fork. A background save duplicates pages as they are modified. Nothing about maxmemory constrains that.
• Client output buffers. A slow consumer, a MONITOR session, or a large pub/sub fan-out accumulates data server-side waiting to be flushed.
• The replication backlog. Held on the primary so replicas can partially resync, and sized independently.
• Allocator overhead. Pages the allocator holds but Redis is not currently using.

Read your actual limits rather than trusting remembered defaults, which vary by version and by how the instance was provisioned:

CONFIG GET maxmemory
CONFIG GET maxmemory-policy
CONFIG GET client-output-buffer-limit
CONFIG GET repl-backlog-size

The practical consequence: an instance with maxmemory set to the full host RAM is not protected. It is configured to fail.

Two settings, and one of them silently disables the other.

maxmemory sets the ceiling. maxmemory-policy decides what happens at it. If the policy is noeviction, Redis rejects writes with OOM command not allowed when used memory > 'maxmemory' and keeps serving reads — a deliberately loud failure that is correct for a datastore and wrong for a cache.

The trap is the volatile-* family. Those policies only consider keys that carry a TTL. Set volatile-lru on a keyspace where nothing expires, and Redis has no eviction candidates at all: it behaves exactly like noeviction while your config says otherwise. We work through all eight options in Redis eviction policy.

One asymmetry worth knowing before you size a replica set: replicas ignore maxmemory by default. Eviction is driven by the primary, which propagates DEL as it evicts. Redis's own documentation warns that a replica "may end up using more memory than what is set via maxmemory" and tells you to make sure replicas "have enough memory to never hit a real out-of-memory condition before the master hits the configured maxmemory setting." A replica sized identically to its primary is undersized.

Fragmentation. The Redis memory ratio you want here is straightforward arithmetic:

INFO memory
# used_memory              — bytes Redis has allocated for data
# used_memory_rss          — resident set size: pages the OS has actually given the process
# mem_fragmentation_ratio  — used_memory_rss / used_memory

A ratio near 1.0 is healthy. Substantially above 1 means the allocator is holding pages that are not carrying data. That happens because, as the Redis docs state, "Redis will not always free up (return) memory to the OS when keys are removed. This is not something special about Redis, but it is how most malloc() implementations work." Delete 2GB from a 5GB instance and RSS will likely still read close to 5GB.

Two things follow, and the second is the one people miss:

Provision for peak. The documentation is explicit: "you need to provision memory based on your peak memory usage. If your workload from time to time requires 10GB, even if most of the time 5GB could do, you need to provision for 10GB."

The ratio lies after a peak. Redis says so directly — "the fragmentation ratio is not reliable when you had a memory usage that at the peak is much larger than the currently used memory," because RSS reflects the peak while used_memory reflects now. A ratio of 3.0 on an instance that recently shed most of its keyspace is expected behaviour, not a fault. Chasing it with defragmentation is wasted effort.

Where fragmentation is genuinely persistent under steady load, active defragmentation exists:

CONFIG GET activedefrag
MEMORY PURGE

Enable it deliberately and watch for the active-defrag-cycle event in LATENCY LATEST — defrag consumes CPU on the single command thread, so it trades memory for latency.

This is the failure that produces "it crashed and the metrics show nothing."

Redis snapshots by forking. The child writes the RDB while the parent keeps serving, and they share memory through copy-on-write. Every page the parent modifies during the save gets duplicated. Under sustained write load, total process memory climbs toward double the dataset — and none of that shows up in used_memory, because it is not keyspace. This is the single biggest gap between reported and real Redis memory consumption.

INFO persistence
# rdb_bgsave_in_progress   — is a save running right now
# latest_fork_usec         — how long the last fork() blocked the server
# rdb_last_bgsave_status   — ok, or the save is failing

We investigated repeated crashes on a customer's Redis cluster where reported utilisation never exceeded roughly 60 percent. Nothing in the Redis metrics explained it, and no alert threshold had been crossed. The cause was fork behaviour during backups, and the fix was kernel-level memory-overcommit tuning — not a Redis setting at all. The same mechanism sits behind several of the availability failures in Redis high availability.

The fork also has a latency cost independent of memory. The Redis persistence documentation notes that fork() "can be time consuming if the dataset is big, and may result in Redis stopping serving clients for some milliseconds or even for one second if the dataset is very big and the CPU performance is not great." latest_fork_usec gives you that number for the last save.

This one is worth describing in detail, because it is the version of the problem that no amount of Redis tuning would have fixed.

On an AceMQ engagement, a customer was hitting repeated crashes at around a 61 percent memory threshold. Same signature as above — plenty of headroom on paper, instance dead in practice. But the fork was only part of it. What we found in the workload:

Every device heartbeat triggered a full document reindex. The application stored device documents and indexed them for query. A heartbeat updated one trivial field — a timestamp — and the write path reindexed the entire document rather than the changed attribute. At fleet scale, that turned a lightweight liveness signal into continuous index churn, with the memory profile of a bulk reload running permanently.

Old indexes were lingering. Superseded index structures were not being reclaimed cleanly, so memory attributable to indexing crept up over time independently of the data volume. The keyspace looked stable; the process did not.

UUID fields were being indexed for no reason. High-cardinality opaque identifiers that nothing queried by, each one adding index structure the workload never read.

Stack those three and you get an instance whose real Redis memory demand is dominated by index maintenance, spiking exactly when write traffic spikes — which is also when a snapshot is most expensive. The 61 percent number was not a threshold in any meaningful sense. It was simply where the baseline sat when the next reindex-plus-fork overlap arrived.

The remediation we worked through with them was architectural, not configuration:

• Separate the heartbeat data from the indexed document. Liveness updates should not touch a structure that has to be reindexed. Split the volatile fields into their own key with its own lifecycle.
• Periodically drop and recreate indexes rather than relying on incremental reclamation to keep up with churn. A scheduled rebuild returns memory that incremental garbage collection was never going to reach at that write rate.
• Tune index garbage collection so reclamation keeps pace with the write rate instead of falling permanently behind it.
• Move reindex jobs off-peak, so bulk index work and peak write traffic stop competing for the same headroom.
• Stop indexing fields nothing queries. The UUIDs came out.

None of that is discoverable from INFO memory. If your Redis instance runs a query engine over documents that are written far more often than they are structurally changed, this is the first place to look — well before you touch maxmemory.

Redis dying at utilisation your metrics say is safe? Fork overhead, fragmentation, and index churn are all invisible to used_memory. AceMQ runs structured Redis memory assessments that find them, and provides enterprise Redis support for production estates. Talk to an AceMQ engineer.

Start broad, then narrow. All of these are safe against a busy production instance because they iterate with SCAN rather than blocking on KEYS.

MEMORY DOCTOR              # Redis's own read on whether memory looks unhealthy
MEMORY STATS               # breakdown by category: dataset, replication backlog, clients, overhead
redis-cli --memkeys        # biggest keys by bytes consumed
redis-cli --bigkeys        # biggest keys by element count, per type
redis-cli --keystats       # both, plus size distribution and a top-N list
MEMORY USAGE <key>         # exact bytes for one key

--bigkeys and --memkeys answer different questions and people conflate them. A set with ten million small integer members is enormous by element count and modest in bytes; a single string holding a 200MB serialised blob is the reverse. The first is a latency problem — any O(N) command over it blocks the server. The second is a memory problem. You need both readings.

Throttle either with -i 0.01 to sleep between SCAN batches on a loaded instance.

A pattern worth naming: write amplification from oversized values. Storing a whole object and rewriting it to change one field means every update pays the full serialised size in allocation churn and, if the instance persists, in copy-on-write pages. Hashes with per-field updates avoid that entirely. This is the same shape as the reindexing case above, one layer down.

Working rules for sizing Redis memory, in the order they matter:

1. Size for peak, not steady state. RSS tracks the high-water mark because the allocator does not readily hand pages back.
2. On an instance that snapshots under write load, keep the dataset around 50–60 percent of RAM. The remainder is fork headroom. Not 80 percent, and certainly not 100.
3. If an instance does not need snapshots, turn them off. A rebuildable cache does not need RDB, and removing the fork removes the entire failure class. Persist on a replica instead.
4. Alert on process RSS, not just used_memory. The metric that kills you is not the one Redis reports.
5. Size replicas independently. They ignore maxmemory by default and can exceed their primary.
6. Leave maxmemory well below host RAM. Setting it to the full box means the kernel OOM killer, not Redis, decides what happens next.

If you are using Redis as a queue and Redis memory pressure tracks queue depth, the sizing question is really an architecture question — a broker with disk-backed queues does not have this failure mode. We cover the trade-off in running Redis and RabbitMQ together.

For the wider diagnostic path around these symptoms — slow commands, replication stalls, connection limits — see our Redis troubleshooting guide.

• Redis — Memory optimization
• Redis — Persistence
• Redis — Replication
• Redis — Key eviction
• Redis — redis-cli reference