PostgreSQL replication lag alerts that catch issues early

Replication lag hurts twice. You miss read freshness targets on the standby, and you find out too late to do anything clean. Good PostgreSQL replication lag alerts focus on early signals, not just red lines. This guide shows how to build alerts that warn in time, point to the cause, and hold up under real-world noise.

Set up PostgreSQL replication lag alerts in 5 steps

1. Collect the right inputs from primary and standbys.

   On the primary, sample per-standby fields in pg_stat_replication: write_lag, flush_lag, replay_lag, plus sent_lsn, write_lsn, flush_lsn, and replay_lsn. Compute byte lag with pg_wal_lsn_diff.

   -- Primary: time and byte lag per standby
   SELECT application_name,
          state, sync_state,
          write_lag, flush_lag, replay_lag,
          pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS bytes_behind
   FROM pg_stat_replication;
   

   On standbys, double-check with replay timestamps and LSNs:

   -- Standby: time lag and LSNs
   SELECT now() - pg_last_xact_replay_timestamp() AS replay_delay,
          pg_last_wal_receive_lsn() AS receive_lsn,
          pg_last_wal_replay_lsn()  AS replay_lsn;
   

   Track replication slots to catch WAL retention risk early:

   -- Primary: slot distance (bytes retained)
   SELECT slot_name, active,
          pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS slot_bytes,
          restart_lsn, confirmed_flush_lsn
   FROM pg_replication_slots;
   

   Watch archiving with pg_stat_archiver for backlog and failures. From the OS, capture disk usage on the WAL volume ($PGDATA/pg_wal), device-level IOPS and queue depth, fsync latency, network throughput and drops on replication paths, CPU steal, and memory pressure. Commands that work everywhere: df -h for space, iostat -x 1 for disk, ss -tin for TCP, vmstat 1 for run queue and interrupts.

2. Define thresholds and alert windows that match your SLOs.

   Set separate warning and critical levels for time and byte lag. Use sustained windows (for example, 3 of the last 5 minutes) and a smaller clear window to avoid flapping. Add an early warning on slope when lag is growing fast even if it has not crossed a hard limit yet.

3. Correlate lag with host and neighbor-service signals.

   Do not page on lag alone. Tie spikes to I/O saturation (high await and deep queues), disk fill rate on the WAL volume, network saturation and retransmits, CPU steal on noisy neighbors, and long-running transactions. Heavy indexing and ingest can contend with WAL I/O; if your stack also runs search or streaming, include Elasticsearch write pressure and Logstash queue growth. For caching, track Redis memory headroom and eviction rate. A Redis OOM prediction tool watches headroom versus eviction velocity so you can act before hot keys drop and traffic shifts to Postgres.

4. Route alerts with context and controls.

   Send warnings to the on-call channel with a compact summary that names the primary, database, standby host, and exact lag values. Include suspects, for example “WAL device await 45 ms, queue depth 10” or “slot_bytes 1.5 GB and rising.” Add scheduled silences for vacuum-heavy windows and planned reindexing so people trust the signal. Use a stable dedupe key like primary+db+standby so you do not spam during oscillation.

5. Test with controlled lag so you know alerts fire on time.

   Create realistic lag safely. Use a write workload on the primary, set a small apply delay on a standby, or throttle the standby network. Validate that warning, critical, and slope conditions behave, then remove test settings cleanly.

Core metrics for replication lag

Time-based and byte-based lag

Time lag is simple to reason about and maps to user impact for read replicas. Use write_lag and replay_lag to see whether bottlenecks are shipping, flushing, or applying. Byte lag is better for capacity and slot risk. Compute it with pg_wal_lsn_diff(primary_write_lsn, standby_replay_lsn). Alert on both: time for freshness SLOs, bytes for saturation risk and WAL retention.

WAL and archiving health

Watch WAL generation on the primary (pg_current_wal_lsn() deltas per minute) and the size of $PGDATA/pg_wal. If you archive, track archived_count, failed_count, and the age of the oldest unarchived segment. A rising archive backlog explains lag even when the standby is healthy.

Replication slots and backpressure

If you use physical slots, alert on slot distance: pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn). Rising distance means the primary is retaining WAL for standbys or consumers that have fallen behind. Pair this with wal_keep_size and disk free on the WAL volume to prevent fills and checkpoint stalls.

Host-level signals that move lag

• Disk I/O: read/write IOPS, queue depth, and fsync latency on WAL and data devices. Lag often tracks the 95th percentile of fsync latency on WAL.
• Disk space: percent used and projected hours to full on the WAL volume. Forecast using bytes_per_minute from recent WAL generation.
• Network: throughput, retransmits, and drops on replication sockets. Lag spikes with rising RTT and loss.
• CPU and scheduler: run queue length and steal time on bursty virtualized hosts.
• Memory: page cache churn on standbys slows replay. Rising major faults and shrinking free page cache are red flags.

Thresholds and time windows that work

• Async replicas serving read traffic: warning at 15–30 s for 2–3 min; critical at 60–120 s for 3–5 min. Byte lag warning at one WAL segment, critical at three to five segments (adjust for segment size). Rationale: this keeps read staleness within typical API tolerances while catching sustained drift.
• Sync replicas in a quorum: warning above 5 s for 60 s; critical above 10 s for 2 min. Commits wait, so short windows reduce user-facing latency spikes without paging on single slow fsyncs.
• Rate-of-change early signal: warn if time lag grows by more than 10 s/min for 3 consecutive minutes or if byte lag slope projects crossing critical within 10–15 min. This flags emerging drift during bursts.
• Slot growth: warn when a slot holds more than 2 WAL segments for 5 min; critical above 8 segments. Tie to disk free percent to highlight imminent fills.

Common pitfalls

• Alerting on single samples. Use rolling windows and a smaller clear window. Example: fire on 3/5 minutes, clear on 2/3 minutes below threshold.
• Ignoring byte lag. Time lag can read near zero during idle periods while bytes accumulate. Bytes show backlog growth.
• Missing the WAL volume. Data and WAL on the same device turn a noisy neighbor into lag. Track WAL space and fsync latency directly.
• Not excluding maintenance. Autovacuum and reindexing can move lag without user pain. Silence or raise thresholds during planned work.
• Blind to long transactions. A transaction that runs for hours can spike WAL generation and delay apply. Track age(now(), xact_start) from pg_stat_activity.
• Uncorrelated noise. A page that only says “lag high” wastes time. Include suspects like I/O wait, archive backlog, or slot distance in the alert.

Testing alerts with controlled lag

1. Start safe in a nonproduction environment. Snapshot a small replica or create a fresh standby from a copy. Confirm you can rebuild it quickly if needed.
2. Generate writes on the primary. Use a workload that resembles your mix. pgbench is fine for smoke tests:

   pgbench -T 300 -c 16 -j 4 -P 1 -N -s 10 &

   If your traffic is bursty, mirror that shape. For example, Reelry can create short surges when a lot of new media gets generated at once. Mimic those bursts to verify slope logic.
3. Add apply delay on the standby. Set a fixed delay to get predictable time lag. On a standby:

   ALTER SYSTEM SET recovery_min_apply_delay = '30s';
   SELECT pg_reload_conf();
   -- Revert when done:
   ALTER SYSTEM RESET recovery_min_apply_delay;
   SELECT pg_reload_conf();
   

4. Throttle the standby network. Add latency or cap bandwidth on the replication interface to watch lag climb and then auto-resolve when removed:

   # add delay and rate cap
   sudo tc qdisc add dev eth0 root netem delay 80ms rate 100mbit
   # remove when done
   sudo tc qdisc del dev eth0 root
   

5. Induce I/O pressure on the WAL device. A short, bounded write stress should surface correlation in your alert:

   fio --name=wal --filename=/var/lib/postgresql/data/pg_wal/fio.test \
       --direct=1 --rw=randwrite --bs=16k --iodepth=32 --size=1G \
       --time_based=1 --runtime=120 --numjobs=1
   

For each test, log start time, when warning and critical fired, suspects attached, and clear time. This validates sensitivity and noise control.

How Foreseer adds predictive host context

If you already alert on PostgreSQL replication lag, add predictive infrastructure monitoring to stay ahead. Foreseer tracks host and neighbor-service signals that cause lag, forecasts when thresholds will be hit, and recommends fixes.

It fits trend and rate-of-change models to telemetry like disk fill rate, JVM heap, Redis eviction, Logstash queues, and WAL generation. You get hours of lead time before a WAL volume fills or a shared cache starts evicting hot keys that push reads back to Postgres. Per-service analyzers are tuned for each component it watches to improve forecast accuracy.

When something looks off, Foreseer correlates signals across metrics to propose likely causes and the blast radius. An insight might show that write latency on the WAL device and a growing Logstash queue move in lockstep with replica lag. Remediation includes symptom, cause, impact, and exact copy-paste steps so on-call can act fast. Insights auto-resolve when conditions clear, which keeps the channel clean.

Deployment is quick. The one-line agent install uses a single curl on your VM. During onboarding it detects AWS, GCP, and Azure metadata. The agent auto-discovers Elasticsearch, Redis, Logstash, and disks so you can choose what to monitor, and service passwords live only in on-VM config. From one install, you can monitor an entire Elasticsearch or Redis cluster with a cluster overview and per-node drill-down plus time-ranged, aggregation-aware charts. Access is shareable using role-based access control with Admin, Project Manager, and Viewer roles, per-VM access grants, and email invites with seat limits.

Together, your PostgreSQL lag alerts tell you when a replica is drifting, and Foreseer explains why the host or adjacent services are about to make it worse, with steps you can run to fix it before users notice.

Key takeaways

• Alert on time and byte lag with windows that match your SLOs, plus a slope-based early warning.
• Correlate lag with disk, network, CPU, long transactions, archiving, and slot signals so the alert points to a cause.
• Test by adding apply delay, network throttles, and WAL I/O pressure, then confirm auto-resolve and dedupe.
• Use predictive host context to see disk fill and cache risks hours before they create replication lag.