Written by Albert Friedman
on 

Alert fatigue: designing smarter alerts with PromQL

Alert fatigue is the slow leak that turns useful monitoring into background noise. When on-call teams receive pages for transient blips, duplicate failures, or low-value signals, responders stop trusting alerts — response time slips, real incidents get missed, and morale erodes. Prometheus + Alertmanager give a powerful toolbox to address this problem, but the key is design: focus alerts on user-visible symptoms, encode intent in PromQL, and use Alertmanager to route and suppress noise. (prometheus.io)

Why so many noisy alerts?

Core principles for smarter alerting

How PromQL and Alertmanager help — practical patterns

1) Symptom-based rate alerts (the basic building block) A symptom-level alert counts user-visible failures (for example 5xx HTTP responses) and compares their rate to total requests. Using rate() over a short window and then aggregating keeps the math correct and avoids miscounting due to counter resets. Example (conceptual):

# recording rule (precompute error ratio over 5m)
- record: job:slo_errors_per_request:ratio_rate5m
  expr: |
    sum(rate(http_requests_total{job="api",code=~"5.."}[5m]))
    /
    sum(rate(http_requests_total{job="api"}[5m]))

# alert (ticket-level or page-level depending on thresholds)
- alert: HighErrorRate
  expr: job:slo_errors_per_request:ratio_rate5m{job="api"} > 0.01
  labels:
    severity: page
  annotations:
    summary: "api: error rate > 1% over 5m"

Note: compute the rate first, then aggregate. Recording rules are the right place to precompute ratios used widely across dashboards and alerts. (prometheus.io)

2) Multi-window burn-rate alerts (reduce pages while keeping sensitivity) SLO-based burn-rate alerts reduce false positives by measuring how fast the service consumes its error budget. The multi-window pattern checks both a long window (to verify significance) and a short window (to ensure the problem is still active). For a 99.9% SLO, common guidelines use pairs like (1h & 5m) and (6h & 30m) with different burn-rate thresholds; pair results before paging. Example (abridged form based on recommended patterns):

# assume recording rules job:slo_errors_per_request:ratio_rate{5m,1h,6h,30m} exist

# page if both short+long windows exceed a high burn rate
- alert: ErrorBudgetBurnFast
  expr: |
    (
      job:slo_errors_per_request:ratio_rate1h{job="api"} > (14.4 * 0.001)
      and
      job:slo_errors_per_request:ratio_rate5m{job="api"} > (14.4 * 0.001)
    )
    or
    (
      job:slo_errors_per_request:ratio_rate6h{job="api"} > (6 * 0.001)
      and
      job:slo_errors_per_request:ratio_rate30m{job="api"} > (6 * 0.001)
    )
  labels:
    severity: page
  annotations:
    summary: "api is burning its 30-day error budget too fast"

This pattern prevents short spikes from generating long-lived pages and ensures quick reset after a fix. The method and example windows/thresholds draw on SRE guidance and community implementations. (sre.google)

Important nuance: avoid using Prometheus’ for: duration as the primary mechanism for SLO alerts. The SRE workbook explains that long durations can reduce precision or cause alerts that take too long to clear; the multi-window burn-rate approach gives both precision and fast resets. (sre.google)

3) Use recording rules and low-level smoothing

4) De-duplicate, group, and inhibit with Alertmanager Alertmanager can deduplicate identical alerts, group many instance-level alerts into a single notification, and inhibit lower-level alerts when a higher-level problem is firing (e.g., mute instance errors while a cluster outage is active). Combined with labels that capture service and owner, this turns many noisy alerts into a single, actionable page with context. (github.com)

Design details that matter

A note on risk-based and contextual prioritization Beyond rules and thresholds, recent work shows that risk-based/prioritized alerting (combining signal severity with business context) can greatly reduce the number of false or low-value alerts analysts must review. For security and SOC use cases, formal evaluations indicate strong benefits for prioritized approaches, suggesting similar gains could apply when enriching Prometheus alerts with ownership and business-risk metadata. (arxiv.org)

Example checklist (patterns to adopt in your ruleset)

Conclusion Alert fatigue is not just a nuisance — it undermines incident response and system reliability. Prometheus and PromQL make it possible to express intent (SLOs, burn rates, symptom-level checks) directly in your alerting logic, and Alertmanager turns rules into sensible, routed notifications. Combining multi-window burn-rate patterns, recording rules, and careful use of Alertmanager’s grouping/inhibition yields fewer, more actionable alerts — the kind that reclaim trust and keep teams focused on the problems that matter. (last9.io)

References (selected)