Skip to content

Health Endpoints

Applies to: All subscriptions

Purpose

Explain what the built-in health probe proves, what it does not, and how to build checks that catch the failures it cannot see. A green probe on a system that cannot process a document is worse than no probe at all.

Audience

Operations and infrastructure teams configuring monitoring.

Prerequisites

Reference

The built-in probe

GET /api/health

Returns HTTP 200 with a success body when the application is answering.

What it proves: the process is running, the port is open, the reverse proxy can reach it, and the routing works.

What it does not prove: that the database is reachable, that the queue is working, that background jobs are running, that the provider connection is valid, or that documents can be processed.

It is a liveness probe. Treat it as one.

Why a shallow probe is the right design

A health probe that checks every dependency has a failure mode of its own: a slow dependency makes the probe slow, the probe times out, the load balancer removes a healthy instance, and a minor degradation becomes an outage.

The shallow probe is correct for its purpose — load-balancer membership and process supervision. Depth belongs in monitoring checks that can afford to be slow and that alert rather than remove capacity.

Building the deeper checks

Configure these in your monitoring platform. Each targets a specific silent failure.

# Check Method Healthy Catches
1 Application liveness GET /api/health 200 within 2 seconds Process down, proxy misrouting
2 Database reachable Any authenticated list query Returns within your normal response time Database down, credentials expired, pool exhausted
3 Background jobs running Age of the oldest incomplete job Below your normal processing time Worker down, queue unreachable — the classic silent failure
4 Queue reachable Queue depth metric A number that rises and falls Queue down while everything else looks fine
5 Provider connection Scheduled connection test Success Credentials expired, egress address changed, provider outage
6 Write-back backlog Age of the oldest pending event Below your consumer's interval Consumer stopped, or a stuck head-of-queue
7 Documents received Count per hour during business hours Above zero An ERP that has stopped sending
8 Unregistered document age Oldest document awaiting registration Well inside the reporting window Documents accruing a statutory deadline
9 Certificate expiry Days remaining Above 30 An outage with a known date
10 Disk free Percentage free Above 20 percent Growth outrunning provisioning

Checks 3, 6, 7 and 8 are the ones that catch failures nothing else will. Do not omit them because they are harder to configure than an HTTP probe.

The synthetic transaction

The most valuable check is one that exercises the real path. In a non-production environment, on a schedule:

  ingest a test document
      -> confirm accepted
      -> generate against the sandbox provider
      -> confirm registered
      -> print
      -> confirm the output renders

This proves the whole chain in one signal. It cannot be run in production, because a document sent to the government is permanently registered and consumes a number — so run it in a non-production environment against the sandbox, and treat its failure as an early warning for production.

Probe intervals

Check Interval
Liveness 30 seconds
Database, queue 1 minute
Job age, write-back backlog 5 minutes
Documents received 15 minutes, during business hours
Provider connection 15 minutes
Unregistered age Hourly
Certificates, disk Daily
Synthetic transaction Daily

Do not probe the provider connection frequently. Each test may consume a billed call, and a high-frequency check against a third party is both expensive and impolite.

Validation

Check Method Pass condition
Liveness probe works Stop the application The probe fails within one interval
Database check works Stop the database The check fails; the liveness probe may still pass, which is expected
Job check works Stop the worker and submit a job Job age rises and the check fires
Queue check works Stop the queue The check fires while the application still looks healthy
Provider check works Point at a wrong endpoint The check fires
Backlog check works Stop the consumer Backlog age rises and the check fires
Documents-received check works Stop the ERP feed The check fires within one interval
Certificate check works Test against a soon-expiring certificate Fires with enough notice to act
Synthetic transaction works Break a step deliberately The failure is detected
Alerts reach someone Fire one Received and acknowledged by a person

Test every check by causing the failure it is meant to catch. A check that has never fired is a check nobody knows works.

Troubleshooting

Symptom Cause Action
Probe green, users blocked The probe is shallow by design Add the dependency checks above
Instances removed from the load balancer during a slow period The probe is too strict or too slow Keep liveness shallow and fast; put depth in monitoring
Job check fires constantly Threshold below normal processing time Measure the baseline and set the threshold above it
Provider check consumes billed calls Probing too often Reduce frequency to 15 minutes or lower
Documents-received check fires overnight Business-hours logic missing Restrict it to business hours
Backlog check never fires although the consumer is stopped Alerting on count rather than age A steady count hides a stuck queue. Alert on age
Synthetic transaction registers real documents Being run in production Run it only in non-production, against the sandbox
Certificate expired despite monitoring Threshold too short, or the alert unowned 30 days and a named owner