Incident prediction models for ops teams, explained simply

The worst part of a 3 a.m. page is not the alert. It is the lack of time to fix the root cause before users notice. Incident prediction flips that. By forecasting when a metric will hit a risky threshold, you move remediation into business hours with a plan you can execute.

This guide explains the modeling choices behind prediction-driven monitoring for self-hosted services (software you run on your own machines). You will see what to forecast, how far ahead to look, and how to produce explanations an on-call engineer can trust and act on, in plain English.

How incident prediction works in operations

Good ops (operations) models are simple, easy to explain, and fast on streaming telemetry (metrics and logs arriving in real time). You do not need complex ML (machine learning) to predict a disk filling up, a Redis shard approaching eviction, or a Logstash queue saturating. You need the right fit for the signal, a horizon that exceeds your repair time, and outputs that include cause and remedy.

Thresholds and trend lines

Start from a threshold that reflects user impact, then predict time to breach. For monotonic growth signals (steadily increasing) like disk usage or index segment counts, a robust straight-line fit over a recent window works well. Use a method that is resistant to outliers (for example, a median-based line or one that down-weights spikes), and compute confidence bands (how sure the forecast is).

• Example: disk_used_pct = 78, slope = +1.2 pct/hour, target = 90. Time-to-breach = (90 − 78) / 1.2 ≈ 10 hours. Include a 95 percent interval, e.g., 8.5 to 11.7 hours, so humans see uncertainty.
• Elasticsearch cluster monitoring: fit separate trends for JVM heap (Java memory), storage, and shard counts (data slices). If the fit shows heap pressure coinciding with peak indexing, schedule shard rebalancing or pipeline tuning before GC (garbage collection) stalls a node.

Rate of change and saturation clocks

Where burstiness (sudden spikes) matters, track velocity (how fast a metric is changing). For queues and buffers, estimate time to saturation from current backlog and net inflow (arrivals minus work done).

• Backlog clock: let b = current backlog, C = capacity, r = arrival rate, s = service rate. If r > s, ETA_to_full = (C − b) / (r − s). In words: if more is arriving than being processed, divide the remaining space by the extra arriving per unit time. Update r and s from short rolling windows to capture bursts without chasing noise.
• Logstash (data pipeline): if the pipeline deepens 15 percent every 10 minutes after a new feed starts, project when it crosses the backpressure threshold and preempt by throttling the source or adding workers.
• Redis: track used_memory, evictions, key creation and expire rates, and allocator fragmentation. Combine a rate-of-change model on used_memory with eviction spikes to forecast an OOM window (out of memory). Remediate by raising maxmemory, adding a shard (another node holding part of the data), or adjusting eviction policy before clients see timeouts.

Seasonality and baselines

Many workloads follow daily or weekly cycles (seasonality). You do not always need heavy seasonal models. A baseline by hour-of-week (0 to 167) is often enough:

• Compute rolling Q50 (median) and Q90 (high percentile) by hour-of-week and compare the current value to Q90 + k × IQR (interquartile range, the middle 50 percent spread). Flag elevated regimes and fit trends within that regime so predictions reflect today’s shape.
• Account for releases and calendar spikes by tagging events (for example, deploys or holidays). If a deploy changes the slope, cut the window and refit to avoid dragging in stale history.

Select horizons, thresholds, and SLO-aware triggers

Predictions are only useful if the lead time (heads-up) exceeds the time to remediate (fix or scale). Start from repair time, then pick thresholds and horizons (how far ahead you look) that balance signal and noise. If you tie alerts to your SLO (service level objective), they map to user impact, not just internal limits.

Lead time must exceed repair time

• Disks: if moving shards or expanding a filesystem takes 2 hours, target a 4 to 6 hour horizon. Alert on time-to-90 percent, not at 95 percent.
• Redis: if resizing a shard or adding a node is a 30 minute change, use a 60 to 90 minute horizon. Predict time to sustained eviction spikes, not just memory percent.
• Pipelines: if scaling Logstash workers requires a deploy, forecast saturation at least one deploy cycle ahead. Include a cool-off if autoscaling is in play to avoid flapping.

Pick thresholds tied to user impact

Map predicted thresholds to first-order symptoms, not technical maxima. For PostgreSQL replication lag, alert when predicted lag will break read-your-writes guarantees for replicas, not when replicas are fully stale. For latency SLOs, predict the time until p95 crosses the SLO budget, not just CPU percent.

Calibrate to risk appetite with backtests

Backtest (replay history) over 30 to 90 days of telemetry and incidents. Score predictions at the horizon you care about:

• Useful alert: predicted_breach_time − actual_breach_time ≥ required_lead_time.
• Precision and recall at horizon: count how many useful heads-ups you would have acted on versus false starts, not just how many threshold crossings occurred.
• Cost-weighted tuning: favor earlier warnings for high-blast-radius risks (wide impact), accept later warnings for low-impact capacity work.

Training, drift, and validation you can automate

Ops environments change. Treat models like code: version them, monitor them, and refresh them when reality moves.

Data windows and retraining

• Lookback: pick the shortest window that captures today’s workload. For capacity trends, a few days to 2 weeks is typical. Weight recent data higher with exponential decay (recent points count more).
• Retrain cadence: volatile metrics refresh every few minutes; slower capacity signals hourly. For counters, take rate and handle resets before fitting.
• Aggregation: standardize to 1 minute or 5 minute bins. Align windows to avoid leakage from the future (do not use data you would not have at alert time) when scoring.

Detecting drift

• Residuals: track MAE or MAPE (average forecast error) on a rolling window. Alert if error rises by X percent or exceeds a fixed band for Y consecutive windows.
• Distribution shift: compare recent data to the training window with simple statistical checks. For sudden regime shifts, run a change-point detector and refit immediately.
• Topology awareness: detect new shards, consumers, or version upgrades and trigger a scoped refit. Do not drag pre-change data into the new fit.

Backtesting and actionability

• Replay telemetry and compute whether alerts would have landed with enough lead time to prevent impact, not just whether the threshold was crossed.
• Score at decision time: a model with 60 percent overall accuracy can be more valuable than an 80 percent model if it fires 6 hours earlier with a clear remedy.
• Guardrails: if confidence bands are wide or inputs are stale, downgrade severity or suppress the alert with a rationale.

Make predictions actionable for on-call

A forecast without context is guesswork. Send the symptom, likely cause, user impact, and exact steps to fix, in plain English, with verifications.

From metric to message

• Symptom: Disk on es-data-3 will hit 90 percent in 5.3 hours at the current write rate. 95 percent interval: 4.6 to 6.2 hours.
• Cause: Indexing spike from ingest-pipeline-7 and shard imbalance on es-data-3.
• Impact: Heap and IO pressure will degrade indexing throughput and increase search latency for tenants A and B.
• Remediation: Move 2 shards off es-data-3, pause ingest-pipeline-7 for 20 minutes, or expand the volume by 200 GB. Include copy-paste CLI or API calls from your runbook.
• Verification: Confirm disk trend slope < 0.2 pct/hour for 30 minutes and p95 search latency back under target.

Define a standard alert payload with fields: metric, current value, slope, threshold, ETA, confidence interval, blast radius (who and what is affected), recommended actions, and verification checks. That mirrors explainable AI (systems that show their reasoning) elsewhere. Hiring teams use systems like Marxel to summarize why specific resumes match criteria and what changed the score. On-call engineers deserve the same clarity.

Foreseer’s approach to model selection

Foreseer focuses on models that deliver lead time for teams running self-hosted services. The platform applies streaming trend fits for linear and rate-of-change behaviors across disk fill rate, JVM heap, Redis evictions, and Logstash queue depth. Where diurnal cycles dominate, it blends a per-hour-of-week baseline with an EWMA trend (exponentially weighted moving average) so forecasts track today’s regime without overfitting.

Accuracy improves with context. Foreseer runs per-service analyzers: an Elasticsearch analyzer reasons about shard counts, segment growth, and heap differently than a Redis analyzer that weighs used_memory, eviction rate, and key churn. For cluster views, you get cluster and node monitoring with an overview and per-node drill-down plus time-ranged, aggregation-aware charts. That is especially useful during Elasticsearch cluster monitoring and capacity planning.

Predictions are correlated across metrics to expose cause and blast radius. If a Logstash queue deepens while Elasticsearch indexing latency rises, cross-metric correlation and lag analysis connect the dots so you see upstream pressure, not isolated alerts.

Insights are written in clear English by an LLM (large language model) that lays out symptom, cause, impact, and copy-paste remediation steps. When triggering conditions clear, insights auto-resolve so you do not chase ghosts.

Setup matches ops realities. A one-line agent install gets you running on a VM. During onboarding, the agent performs service auto-discovery for Elasticsearch, Redis, Logstash, and disk so you can choose what to monitor without writing scrape configs. Cloud metadata detection labels hosts across AWS, GCP, and Azure. Service credentials stay on the VM; Foreseer does not store them. Role-based access control supports Admin, Project Manager, and Viewer roles with per-VM grants and seat-limited email invites so you can share access safely.

If you prefer a self-hosted monitoring stack, the same ingredients apply: fit simple models to the right metrics, validate with backtests focused on lead time, and communicate actions with verification steps. That foundation scales from a single Redis node to a multi-tenant search cluster.

Key takeaways

• Use simple, interpretable models first. Linear trends and rate-of-change clocks cover most ops signals.
• Choose horizons from repair times and thresholds from user impact, not technical maxima.
• Continuously retrain, watch residuals and change-points for drift, and backtest on lead-time usefulness.
• Ship explanations with symptom, cause, impact, remedies, and verification checks the on-call can run.
• Apply per-service context and cross-metric correlation to reveal causes and blast radius before acting.

Predictive monitoring works when it buys time and reduces guesswork. Keep models simple, validate against your incidents, and deliver explanations humans trust.