Forecast Redis evictions to stop latency spikes early

Redis evictions are rarely a surprise if you watch the right signals. Memory pressure rises over hours, allocator fragmentation worsens, and then a latency spike hits when Redis scrambles to free space. You can forecast the pressure, get a clear time-to-limit, and act before users notice.

This guide explains the precursors to evictions, a practical forecast you can build from routine metrics, and the actions that reliably buy time. It closes with how to encode those actions into repeatable playbooks and how a predictive platform can automate the heavy lifting.

Signals that precede Redis evictions

Evictions begin when used_memory converges on maxmemory and your maxmemory_policy starts removing keys. Watch these signals with 10–30 second resolution.

• Headroom to maxmemory. Track headroom_bytes = max(0, maxmemory - used_memory). The slope of this series is your memory burn rate. Sustained negative headroom is where eviction storms come from.
• Eviction counters and rates. evicted_keys and its rate often tick up minutes before latency moves. A rising d(evicted_keys)/dt while headroom shrinks is a strong early warning.
• Fragmentation pressure. mem_fragmentation_ratio above ~1.4 means the allocator cannot pack objects tightly; effective free space is lower than it looks. Also compare used_memory_rss to used_memory to spot allocator overhead.
• Keyspace churn. Growth in dbX:keys, many expires set but slow expired_keys, and rising keyspace_misses point to TTL debt and cache leakage. These patterns increase burn without improving hit rate.
• Background work and copy-on-write. RDB snapshots and AOF rewrites can temporarily double memory via copy-on-write. If you are near the limit when BGSAVE or BGREWRITEAOF triggers, evictions spike even if steady-state looked safe.
• Latency sympathy. Queue depth or p99 latency increases in services that depend on Redis often appear shortly after eviction rate picks up. Plot them together to understand blast radius.

If maxmemory_policy is noeviction, pressure shows up as errors or OOM risk rather than key churn. The same forecasting math predicts the cliff; it just flags a different failure mode.

Build a simple eviction forecast

You do not need a complex ML pipeline. A robust slope over a rolling window gives actionable lead time if you handle fragmentation and background work.

1) Collect metrics with context

Scrape every 10–30 seconds. Minimum set: used_memory, maxmemory, evicted_keys, keyspace_hits, keyspace_misses, mem_fragmentation_ratio, used_memory_rss. Capture flags for BGSAVE/BGREWRITEAOF, replica resync, and compactions so you can exclude those windows from fitting.

2) Estimate burn rate

On a 60–120 minute rolling window, fit a robust line to used_memory with Theil–Sen or a median-of-differences slope to ignore spikes. A 90 minute default works well for diurnal traffic while remaining responsive to trend changes.

Example: growth_rate_bps = slope(used_memory over last 90m). Current headroom: H = max(0, maxmemory - used_memory_now).

3) Convert to time-to-limit

If growth_rate_bps > 0, compute ttl_seconds = H / growth_rate_bps, then cap horizon to 48h. If slope is negative or near zero, do not alert. Adjust for fragmentation: H_eff = H / max(1.0, mem_fragmentation_ratio) and recompute ttl_seconds.

Worked example: used_memory_now = 11.8 GiB, maxmemory = 12 GiB, so H = 0.2 GiB ≈ 214,748,364 bytes. If growth_rate_bps = 1.5 MiB/s ≈ 1,572,864 B/s, then ttl ≈ 136.6 s. With mem_fragmentation_ratio = 1.6, H_eff ≈ 134,217,727 B and ttl ≈ 85.4 s. Fragmentation can cut lead time by ~37%.

4) Sanity-check with correlated signals

Overlay the forecast with eviction rate, miss rate, and p99 latency. If evictions are rising while your headroom remains positive, fragmentation, copy-on-write from background work, or bursty allocator behavior are likely. If evicted_keys is flat and slope is noisy, suppress alerts.

5) Backtest and tune

Replay at least 7 days. Define a “sustained eviction” event as eviction rate >= X/s for Y seconds (for example 1/s for 300s). Measure precision/recall of alerts fired when ttl_hours <= 4. Tune window length, hold-down timers, and fragmentation adjustment until you reduce false alarms while catching events with hours of notice.

6) Alert condition

Start simple: alert when ttl_hours <= 4 AND (d(evicted_keys)/dt > 0 OR mem_fragmentation_ratio >= 1.4). Add a 5–10 minute hold-down to avoid flapping. Auto-resolve when ttl_hours >= 12 for 30 minutes and eviction rate returns to baseline.

Actions and playbooks to delay or avoid evictions

Predictions are only valuable if they drive safe, fast changes. Order actions by time-to-limit and risk. Encode each as a copy-paste runbook with prechecks, change, and rollback.

• Reduce memory footprint immediately (minutes). Trim payloads and store deltas. Prefer hashes with field updates over large strings. Enable compression at the app layer for bulky values. Lower TTLs on low-value namespaces first (for example, ephemeral sessions or dedupe caches).
• Clear TTL debt (minutes). If you have hours of headroom, temporarily increase active expire effort to flush stale keys. Verify hit rate and p99 latency during the change.
• Align eviction policy with reality (minutes). If most keys lack TTLs, volatile-lru will thrash a small subset. Consider allkeys-lru when the dataset behaves like a cache. Do not switch to noeviction under pressure.
• Right-size maxmemory with guardrails (minutes). If host memory allows, raise maxmemory in small increments and track mem_fragmentation_ratio and RSS delta. Set a hard cap to avoid host OOM.
• Scale the working set (tens of minutes). Split hot keyspaces, reshard high-growth slots in Redis Cluster, or move the bulkiest namespaces to a new node.
• Schedule heavy background work (hours). Shift AOF rewrite and RDB snapshots away from peak load or after you have raised headroom to avoid copy-on-write shocks.

Two pitfalls to avoid: do not chase one-off spikes; use a short hold-down so a single positive window does not trigger churn. And watch replicas during resync; a full sync can double memory briefly on the master and invalidate your forecast if ignored.

Example playbook snippets your on-call can run:

• Diagnose fast: redis-cli INFO memory, redis-cli INFO stats, then compute headroom and slope from your metrics store. Confirm mem_fragmentation_ratio and note background flags.
• Immediate relief valve: Propose a safe delta for CONFIG SET maxmemory (for example +512 MiB if host free RAM > 2 GiB and RSS/headroom trend is stable). Include CONFIG REWRITE and a rollback to the prior value.
• Prune low-value keys: Identify namespaces such as session:* or inflight:*. Sample with SCAN 0 MATCH session:* COUNT 1000 to estimate cardinality and memory, then apply a TTL policy change during a low-traffic window.
• Reshard with intent: Move the top N hot slots or migrate a large hash set. Validate by comparing per-shard hit rate, p99 latency, and evicted_keys rate pre/post.
• Verify and close: Define success as headroom > threshold for 30 minutes and eviction rate back to baseline. Auto-resolve the alert when conditions hold.

Codifying repeated steps pays off. Teams often capture multi-step actions in tools like FlyTrap so changes run the same way every time. Your Redis runbooks should be just as explicit.

How Foreseer predicts and explains eviction risk

If you prefer a self-hosted monitoring platform that handles forecasting end to end, Foreseer focuses on predictive signals rather than reactive alarms. Its agent gathers Redis telemetry at 10–30 second resolution and fits robust linear and rate-of-change models to memory headroom, eviction counters, fragmentation, and miss rate to estimate time-to-limit hours in advance.

When pressure builds, Foreseer correlates metrics within Redis and across neighboring services to explain the likely cause and blast radius. It distinguishes steady growth from copy-on-write spikes during BGSAVE, highlights keyspace churn patterns, and estimates how much headroom you recover by shortening TTLs or resizing maxmemory. The insight includes symptom, likely cause, impact, and copy-paste remediation steps tailored to your policy and topology. Insights auto-clear when triggering conditions resolve so dashboards stay accurate without manual cleanup.

Getting started is a single curl that installs a one-line agent on your VM. The agent auto-discovers Redis, Elasticsearch, Logstash, disks, and cloud metadata (AWS, GCP, Azure). Credentials live only on the VM. From one install, you get a cluster overview plus per-node drill-down with time-ranged, aggregation-aware charts. Role-based access provides Admin, Project Manager, and Viewer roles with per-VM grants and email invites with seat limits.

If your estate mixes predictive and traditional signals, keep this forecaster focused on headroom-to-limit problems like Redis evictions while you continue to track conventional events such as PostgreSQL replication lag in your existing alerting system.

Key takeaways

• A slope-based forecast on headroom, adjusted for fragmentation, gives practical warning before evictions.
• Correlate eviction rate, miss rate, and latency to reduce false alarms and scope impact early.
• Encode actions as runbooks with prechecks, change, and rollback so on-call can act in minutes.
• Foreseer applies per-service models, cross-metric correlation, and plain-English remediation to predict and explain eviction risk.