Monitoring that predicts outages

Most monitoring tells you that something has already failed. That is necessary and it is not the same as useful. The signals worth alerting on are the ones that change before the failure, in a window long enough to act.

The distinction matters because every alert costs attention, and attention spent on alerts that arrive too late to help is attention not available for the ones that arrive in time.

Signals that lead

Capacity trending towards a limit

Disk filling is the classic example and still the most common avoidable outage. What matters is not the current percentage but the rate of change.

A disk at eighty percent that has been at eighty percent for six months is fine. A disk at sixty percent that was at forty percent last week will be full on Thursday. Alert on time-to-full rather than on a threshold, and the alert arrives with enough notice to be useful.

The same logic applies to database connection pools, thread pools, licence counts, IP address pool exhaustion in DHCP scopes, and certificate expiry. All of them fail suddenly after being fine for a long time, and all of them are predictable days or weeks ahead.

Latency before saturation

Systems slow down before they stop. Response time rising steadily while throughput stays flat is a reliable indication that something is approaching a limit: usually a lock, a connection pool, or a downstream dependency.

Alert on the ninety-fifth or ninety-ninth percentile rather than the average. Averages hide the onset of trouble, because the slow requests are a minority until suddenly they are not.

Error rate as a proportion, not a count

A count of errors is meaningless without volume. Ten errors an hour is fine at a million requests and alarming at fifty.

Alert on the ratio, and alert on the change in ratio. A rate that doubles is interesting even when the absolute number is small, because it usually means something changed.

Queue depth

Anything with a queue (message brokers, print spoolers, mail queues, job schedulers, replication) tells you about the health of whatever consumes it. A growing queue means consumption is slower than production, which is a problem that will become an outage if it continues.

Queue depth is one of the most reliable leading indicators available and it is frequently unmonitored.

Hardware predictive indicators

Drive health attributes, correctable memory errors, fan speed rising, power supply redundancy lost. Each of these is a warning that the component is degrading but has not yet failed.

Correctable memory errors are worth particular attention. They are silently corrected until they are not, and the transition to uncorrectable is often preceded by a rising rate.

Backup and job completion

Alert on the absence of success, not on the presence of failure. A backup job that stops running entirely reports nothing, and reporting nothing looks identical to everything being fine.

Any scheduled job that matters should have a check that fires when the expected completion does not arrive.

Certificate and credential expiry

Entirely predictable, entirely preventable, and a recurring cause of outages. Alert well ahead: thirty days is a reasonable first warning, with escalation as the date approaches. Include internal certificates, service account passwords with expiry, and API keys.

Signals that mostly make noise

Instantaneous CPU. A server at a hundred percent CPU for thirty seconds is usually a server doing its job. Sustained high CPU over a meaningful window is worth knowing about; a spike is not.

Memory used. Modern operating systems use available memory for caching by design. Alerting on memory used produces constant false positives. Alert on memory pressure, swap activity, or allocation failures instead.

Single ping failures. Networks drop packets. Require several consecutive failures before alerting, or alert on sustained loss rate.

Any individual service in a redundant cluster. If the design tolerates the loss of one node, losing one node is not an incident. It is a ticket. Alert on the cluster's capacity to tolerate the next failure, which is the thing that actually changed.

Everything the monitoring tool enables by default. Default rule sets are written for a hypothetical environment. Most of what they enable will not correspond to anything you care about.

Making alerts actionable

An alert that does not tell the recipient what to do produces a delay while they work it out, usually at an inconvenient hour.

Every alert should carry: what is wrong, which system, what the business impact is or might be, and a link to what to do about it. If nobody can write down what to do about it, that is a strong sign the alert should not exist.

Two habits keep the alert set healthy:

Review alerts that fired and required no action. These are candidates for tuning or deletion. An alert nobody acts on is training people to ignore alerts.

Review incidents that had no alert. This is the more important review and the one usually skipped. For each unmonitored failure, ask what signal would have shown it earlier, and whether that signal is available.

Where to start

If monitoring is currently thin, the highest-value additions in rough order:

1. Time-to-full on every filesystem that matters.
2. Certificate and credential expiry, with thirty days notice.
3. Backup and scheduled job success, alerting on absence.
4. Queue depth on anything with a queue.
5. Ninety-fifth percentile latency on the services users touch.
6. Hardware predictive indicators on physical infrastructure.

None of these require sophisticated tooling. All of them catch failures before users do, which is the only measure of monitoring that matters.