Skip to content

Alerting

Applies to: All subscriptions

Purpose

Decide what deserves to interrupt a person, who it should reach, and how urgently — and keep the alert set small enough that people still act on it.

Audience

Operations teams and whoever owns the customer's on-call rota.

Prerequisites

Reference

The test for whether something deserves an alert

An alert is a request to interrupt a human being. It earns that only if all three hold:

  1. Something is actually wrong, or will be soon.
  2. A person can do something about it now.
  3. Not acting has a consequence.

An alert failing any of these is noise, and noise is not harmless: it trains people to ignore the channel that carries the alerts that matter.

Severity

Severity Meaning Response Route
Critical Compliance or availability is at risk now Immediate, out of hours On-call
High Will become critical if unaddressed today Same business day Team channel plus email
Medium Needs attention this week Within days Email or ticket
Low Informational Routine review Dashboard only

The alert set

Critical:

Condition Why critical
Application unreachable Nobody can work
Database unreachable The application cannot start or serve
Oldest unregistered document past two-thirds of the reporting window The only unrecoverable condition in this list
Disk free below 10% Imminent failure
Certificate expires within 7 days A dated outage
Provider credentials expired All government operations have stopped

High:

Condition Why
Background jobs not completing Uploads silently stalled
Queue depth rising with no consumption The classic silent failure
Provider connection failing Generation has stopped
Documents received from an ERP at zero during business hours An integration has stopped, and deadlines are accruing
Write-back backlog age above four times the consumer interval The ERP is falling out of step
Government rejection rate rising sharply A systemic fault reaching the government
Validation rejection rate at twice baseline A systemic fault at the door
Disk free below 20% Days of runway
Certificate or provider credential expires within 30 days Time to act, comfortably

Medium:

Condition
Quarantined documents ageing beyond a day
e-Way Bills approaching expiry within the horizon
Dead-lettered write-back events present
Provider call volume approaching budget
Sustained resource utilisation above the warn threshold
Accounts unused for 90 days
Fewer than two administrators

Who receives what

Category Recipient
Infrastructure and availability Infrastructure team, on-call
Application errors Operations, then Complifly support if unresolved
Business and compliance conditions The tax or finance operations owner
Integration conditions The integration owner, and the ERP team
Security conditions The security team

Every recipient must be a distribution list. An alert addressed to an individual becomes an alert nobody receives the moment that person leaves, changes role, or takes leave — and nobody notices that it has stopped being received.

Every alert needs a runbook

An alert that fires at 2am to someone who does not know what to do is an alert that produces an escalation, not a fix. Each alert should carry:

Element Example
What it means "Background jobs are not completing"
First check "Is the worker process running? Is the queue reachable?"
Likely causes Worker stopped; queue unreachable; database contention
Link to the page Troubleshooting Center
When to escalate "If the worker and queue are both healthy"

Avoiding fatigue

Practice Reason
Alert on symptoms, not on every metric Ten alerts for one incident train people to mute the channel
Set thresholds from measured baselines Guessed thresholds are the main source of noise
Suppress during known maintenance Predictable noise is the fastest route to ignored alerts
Group related alerts One incident, one notification
Review monthly Delete alerts nobody acts on. An unactioned alert is a liability, not a safety net
Track false positives A rate above roughly one in five means the threshold is wrong

The monthly review is the practice most often skipped and the one that keeps the set healthy. An alert that has fired forty times and been acknowledged forty times without action should be deleted or re-tuned, not tolerated.

Validation

Check Method Pass condition
Every alert has been fired deliberately Cause each condition in a test environment It fires
Every alert reaches a person Fire and confirm Acknowledged by a human
Recipients are lists Review the configuration No individual addresses
Every alert has a runbook Review each Present and linked
Severities are correct Review Critical alerts genuinely warrant waking someone
False-positive rate is low Review a month of history Below roughly one in five
The compliance-deadline alert works Simulate an ageing document Fires with time to act
Nothing is muted permanently Review suppressions None indefinite

Troubleshooting

Symptom Cause Action
Alerts ignored Too many, or too many false positives Reduce the set; re-tune from measured baselines
An alert fires and nobody acts No owner, or no runbook Assign an owner; write the runbook
Alerts to a departed employee Individual recipients Use distribution lists
Ten alerts for one incident Alerting on every metric rather than symptoms Group them
Storm during maintenance No suppression window Suppress deliberately, for a bounded period
A real problem missed The condition is not alerted, or the threshold is too loose Add or tighten it, and review why it was missed
Critical alerts at 2am for non-urgent conditions Severity misassigned Re-classify. Critical means someone must act now
A channel muted permanently Fatigue has already won Rebuild the alert set from scratch with fewer, better alerts