Skip to content

Preventive Maintenance

Applies to: All subscriptions

Purpose

Collect the work whose only visible result is that nothing goes wrong. Preventive maintenance competes badly against reactive work precisely because its success is invisible — so it needs a schedule and an owner, not good intentions.

Audience

Infrastructure teams, database administrators, operations owners.

Prerequisites

  • Operations Guide, with routines being performed
  • Named owners for each item below

Reference

Database upkeep

Task Frequency Prevents
Index maintenance Weekly, or per fragmentation Gradual slowdown at constant volume
Statistics updates Weekly, and after bulk loads Sudden, severe plan regressions
Integrity checks Weekly Corruption discovered only at restore time
File growth review Monthly Disruptive growth events during business hours

The first two are the answer to "the system has got slower and nothing changed". Check maintenance before considering hardware; it is far cheaper and far more often the cause.

Expiry management

Everything here fails at a known future date, which makes each failure entirely preventable and entirely embarrassing.

Item Warn at Owner
TLS certificates 30 days Infrastructure
Provider credentials 30 days Operations, with the commercial owner
Identity provider secrets 30 days Identity team
Machine credentials Per rotation policy Integration owner
Contracts and licences Per the agreement Commercial

Each needs an owner, not just an alert. An alert with no owner produces a notification that everyone assumes someone else is handling.

Provider credentials deserve particular attention: they involve a third party, so 30 days is a floor rather than a comfortable margin, and their expiry stops all government operations at once.

Capacity headroom

Task Frequency Prevents
Compare growth against forecast Monthly Disk exhaustion
Review utilisation trends Monthly Saturation arriving without warning
Re-forecast after any volume change On change Planning against an obsolete baseline
Review retention settings Quarterly Growth nobody chose

Act on trends, not thresholds. A metric moving steadily towards its threshold is a planning decision; a metric that has crossed it is an incident.

Rehearsals

The most valuable preventive work, and the most often postponed.

Rehearsal Frequency Prevents
Restore Monthly, minimum quarterly Discovering at the worst moment that backups do not work
Upgrade Before each real upgrade Windows overrunning; unpleasant surprises
Rollback With the upgrade rehearsal A failed upgrade becoming an outage
Credential rotation Before it is needed A rotation that fails under time pressure
Year-end Before the real one The one exercise that cannot be retried casually

Every one of these is a procedure that people believe works because it is written down. A rehearsal is the difference between a documented procedure and a proven one.

Configuration drift

Task Frequency Prevents
Compare configuration across environments Monthly "It works in test" surprises
Confirm non-production has no production credentials Monthly A test document permanently registered at the government
Review feature flags against intent Quarterly Flags changed and forgotten
Review alert thresholds against baselines Quarterly Noise, or silence

The second row is the one with an unrecoverable consequence, and it is worth verifying by attempting a call rather than by reading configuration.

Knowledge maintenance

Task Frequency Prevents
Runbooks reflect reality Quarterly Procedures nobody can follow
The deputy performs the daily routine Monthly The routine lapsing during leave
Contact and distribution lists current Quarterly Alerts addressed to departed staff
Access review Monthly Access accumulating unnoticed

A single person holding all operational knowledge is an availability risk, and it is the one no amount of infrastructure redundancy addresses.

Scheduling it

Preventive work loses to reactive work unless it is protected:

Practice Reason
Put it in the calendar, with an owner Otherwise it happens when there is time, which is never
Automate what can be automated Maintenance jobs, alerts, checks
Track what was skipped Skipping is sometimes right; not knowing is not
Review after every incident Ask which preventive item would have avoided it

The last practice is where the schedule improves. Most incidents are traceable to a preventive item that was skipped, and naming it turns the incident into a schedule change rather than a lesson nobody records.

Validation

Check Pass condition
Maintenance jobs run and are monitored Job history clean; failures alert
Nothing expires within 30 days unowned Every item has a date and an owner
Growth tracked against forecast Within tolerance, or re-forecast
Restore rehearsed Within the last quarter
Upgrade and rollback rehearsed Before the last real upgrade
Environments free of production credentials Verified by attempting a call
The deputy is capable Demonstrated this month
Skipped items recorded Deliberately, with a reason

Troubleshooting

Symptom Cause Action
Gradual slowdown at constant volume Index or statistics maintenance overdue Run it before considering capacity
A certificate expired despite monitoring Alert with no owner Assign an owner, not just a check
Restore fails when needed Never rehearsed Rehearse quarterly at minimum
An upgrade overran its window Rehearsed at unrepresentative volume Rehearse at real volume and time it
A test document reached the government Non-production holding production credentials Treat as an incident. Verify separation by attempting a call
Preventive work never happens Not scheduled, or no owner Schedule it; assign owners
The routine stops during leave No deputy Name and practise with one
The same incident recurs The preventive item is still skipped Review after every incident and change the schedule