Redis BGSAVE Failed: Fork Cannot Allocate Memory Despite Free RAM - Why It Happens & VM Parameters to Fix

Table of Contents#

1. Understanding Redis BGSAVE and the Fork() System Call
2. Why "Fork Cannot Allocate Memory" Happens Despite Free RAM
3. Key VM Parameters to Fix the Issue
4. Step-by-Step Guide to Resolve the Error
5. Advanced Considerations & Best Practices
6. Conclusion
7. References

1. Understanding Redis BGSAVE and the Fork() System Call#

Before diving into the error, let’s clarify how BGSAVE works in Redis.

What is BGSAVE?#

BGSAVE is Redis’s background snapshotting mechanism. It creates a binary RDB file containing the entire dataset at a point in time, allowing Redis to restore data after a crash. Unlike the blocking SAVE command, BGSAVE runs in the background, so Redis remains responsive to client requests.

How BGSAVE Uses Fork()#

To achieve background saving, Redis uses the fork() system call to create a child process. The child process is an exact copy of the parent Redis process, including its memory. The child then writes the dataset to disk, while the parent continues serving clients.

Crucially, modern operating systems (like Linux) use a technique called Copy-On-Write (COW) to optimize this process. With COW, the parent and child initially share the same physical memory pages. Pages are only copied when either the parent or child modifies them. This means the child process doesn’t immediately consume additional memory equal to the parent—only modified pages are duplicated.

The Role of Fork() in BGSAVE#

For BGSAVE to work, the fork() call must succeed. If fork() fails, Redis cannot create the child process, and BGSAVE aborts with the "Cannot allocate memory" error.

2. Why "Fork Cannot Allocate Memory" Happens Despite Free RAM#

At first glance, the error seems contradictory: If there’s free RAM, why can’t Redis fork? The root cause lies in how the Linux kernel manages memory overcommit—a policy that determines whether the kernel allows processes to request more memory than is physically available.

The Problem: Linux Overcommit Memory Policy#

By default, Linux uses a conservative approach to memory overcommit. When a process calls fork(), the kernel must decide whether to allow the creation of the child process. To do this, it checks if there’s enough "available" memory to accommodate the child.

Here’s the catch: The kernel assumes the worst-case scenario for fork(). Even though COW means the child won’t immediately use memory equal to the parent, the kernel historically required that the system have enough free memory (or swap) to theoretically allocate a full copy of the parent’s memory.

For example:
If Redis is using 10GB of RAM, fork() tells the kernel, "I might need another 10GB (for the child’s copy)." The kernel, using its default overcommit policy, may block the fork() call if free RAM + swap is less than 10GB—even though COW ensures the child will only use a fraction of that.

Why Free RAM Isn’t Enough#

Suppose your system has 12GB of RAM, and Redis is using 10GB. Free RAM is 2GB. When Redis tries to fork(), the kernel checks if it can reserve 10GB (the parent’s size) for the child. Since 2GB < 10GB, the kernel rejects the fork() call, triggering the "Cannot allocate memory" error—even though the child will only use a tiny amount of new memory due to COW.

The Culprit: vm.overcommit_memory#

The Linux kernel’s behavior here is controlled by the vm.overcommit_memory sysctl parameter, which has three possible values:

Transparent Huge Pages (THP): A Hidden Aggravator#

Another factor that can worsen this issue is Transparent Huge Pages (THP), a Linux feature that uses 2MB memory pages (instead of the default 4KB) to improve performance. However, THP increases COW overhead: modifying a single byte in a 2MB huge page forces the entire 2MB page to be copied, drastically increasing memory usage during BGSAVE. This can lead to higher actual memory consumption post-fork, making the kernel’s initial reservation check even more likely to fail.

3. Key VM Parameters to Fix the Issue#

To resolve the "fork cannot allocate memory" error, we need to adjust Linux VM parameters to align with Redis’s COW-based BGSAVE workflow. The most critical parameters are:

1. vm.overcommit_memory#

As discussed, setting vm.overcommit_memory=1 tells the kernel to always overcommit memory. This is the gold standard for Redis because:

• We trust COW to minimize actual memory usage post-fork.
• Redis’s child process (for BGSAVE) is short-lived and only reads memory (it doesn’t modify pages, so COW overhead is minimal).

Setting vm.overcommit_memory=1 allows fork() to succeed even if the kernel can’t reserve the full parent memory size, as it assumes the process (Redis) won’t actually use all the requested memory.

2. vm.overcommit_ratio#

Relevant only when vm.overcommit_memory=2 (never overcommit), vm.overcommit_ratio defines the percentage of physical RAM that can be overcommitted. For example, if vm.overcommit_ratio=50, the kernel allows allocations up to RAM * 50% + swap.

Note: This is rarely used for Redis, as vm.overcommit_memory=1 is preferred.

3. Disabling Transparent Huge Pages (THP)#

To reduce COW overhead during BGSAVE, disable THP. THP can be disabled temporarily or permanently to prevent large page copies from exacerbating memory pressure.

4. Step-by-Step Guide to Resolve the Error#

Let’s walk through the steps to adjust these parameters and fix the BGSAVE error.

Step 1: Check Current VM Settings#

First, verify your current overcommit and THP settings:

# Check overcommit memory policy
sysctl vm.overcommit_memory
# Output: vm.overcommit_memory = 0 (default, problematic)
 
# Check overcommit ratio (if overcommit_memory=2)
sysctl vm.overcommit_ratio
 
# Check THP status
cat /sys/kernel/mm/transparent_hugepage/enabled
# Output example: [always] madvise never (THP is enabled)

Step 2: Temporarily Adjust vm.overcommit_memory#

To test the fix immediately (without rebooting), use sysctl -w:

sudo sysctl -w vm.overcommit_memory=1

This change takes effect immediately. Test BGSAVE afterward to confirm the error is resolved:

redis-cli bgsave
# Check Redis logs for success: "Background saving started by pid XXXX"

Step 3: Permanently Set vm.overcommit_memory#

To make the change survive reboots, add it to /etc/sysctl.conf:

sudo nano /etc/sysctl.conf

Add the line:

vm.overcommit_memory = 1

Save and exit, then apply the changes:

sudo sysctl -p /etc/sysctl.conf

Step 4: Disable Transparent Huge Pages (THP)#

To disable THP temporarily:

sudo echo never > /sys/kernel/mm/transparent_hugepage/enabled
sudo echo never > /sys/kernel/mm/transparent_hugepage/defrag

To make this permanent (persist across reboots), add the following to /etc/rc.local (or create a systemd service for modern systems):

# For /etc/rc.local (ensure executable with chmod +x /etc/rc.local)
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
  echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
  echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

For systemd-based systems (e.g., Ubuntu 20.04+), create a service file:

sudo nano /etc/systemd/system/disable-thp.service

Add:

[Unit]
Description=Disable Transparent Huge Pages (THP)
 
[Service]
Type=oneshot
ExecStart=/bin/sh -c "echo never > /sys/kernel/mm/transparent_hugepage/enabled && echo never > /sys/kernel/mm/transparent_hugepage/defrag"
 
[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl enable disable-thp
sudo systemctl start disable-thp

Step 5: Verify the Fix#

After making these changes, confirm vm.overcommit_memory is set to 1 and THP is disabled:

sysctl vm.overcommit_memory  # Should return 1
cat /sys/kernel/mm/transparent_hugepage/enabled  # Should return "always madvise [never]"

Then run redis-cli bgsave and check Redis logs (e.g., /var/log/redis/redis-server.log) for success.

5. Advanced Considerations & Best Practices#

Swap Space: A Safety Net#

If you must use vm.overcommit_memory=2 (e.g., due to strict security policies), ensure you have sufficient swap space. The kernel uses RAM + swap to calculate overcommit limits, so adding swap can increase the "available" memory pool and allow fork() to succeed.

Monitor Memory During BGSAVE#

Use tools like top, htop, or redis-cli info memory to monitor memory usage during BGSAVE. Look for spikes in RSS (Resident Set Size) or PSS (Proportional Set Size) to ensure COW overhead is minimal.

Optimize Redis Persistence#

• AOF vs. RDB: If BGSAVE (RDB) is problematic, consider using AOF (Append-Only File) persistence, which appends commands to a log instead of creating full snapshots. AOF uses fsync instead of fork() for durability.
• Hybrid Persistence: Redis 4.0+ supports hybrid persistence (AOF with RDB preamble), combining the best of both worlds.

Avoid OOM Killer Interference#

Even with overcommit enabled, ensure Redis has a reasonable maxmemory-policy (e.g., allkeys-lru) to evict old data when memory is full. This prevents the OOM killer from terminating Redis during high load.

6. Conclusion#

The "Redis BGSAVE failed: fork cannot allocate memory" error is not a reflection of actual free RAM but a result of Linux’s conservative memory overcommit policy. By adjusting vm.overcommit_memory=1 and disabling Transparent Huge Pages, you align the kernel’s behavior with Redis’s Copy-On-Write workflow, allowing fork() to succeed even when free RAM is limited.

Remember:

• vm.overcommit_memory=1 is the primary fix for most Redis deployments.
• Disabling THP reduces COW overhead during BGSAVE.
• Always test changes in staging before applying to production.

7. References#

• Redis Documentation: Background Saving
• Linux Kernel Documentation: Memory Overcommit
• Redis Labs: Fork Can’t Allocate Memory
• Transparent Huge Pages and Redis
• Linux Sysctl Tuning for Redis