Skip to content

Performance and Availability

Applies to: All subscriptions

Purpose

Isolate the layer responsible when the system is slow or unavailable, and recognise the several failures that present as performance problems and are not.

Audience

Operations teams, infrastructure engineers, support.

Prerequisites

  • Access to monitoring across all four layers
  • Access to application and proxy logs

Steps

The decision tree

  Slow or down
        |
        v
  Does the application respond at all?
        |
        +-- No --> What does the browser show?
        |            |
        |            +-- Connection refused --> Proxy down, or DNS. -> Path A
        |            +-- 502 -----------------> Proxy up, cannot reach API. -> Path B
        |            +-- 504 -----------------> API too slow, or timeout too short. -> Path C
        |            +-- Certificate error ---> Certificate expired. -> Path D
        |
        +-- Yes -> Is everything slow, or one thing?
                     |
                     +-- Everything --> Which layer is saturated? -> Path E
                     |
                     +-- One thing ---> Which one?
                                          |
                                          +-- Government operations -> Provider, not
                                          |                             performance
                                          +-- Uploads ---------------> Worker or queue,
                                          |                             not performance
                                          +-- Printing --------------> Rendering resources
                                          +-- One report ------------> Query or data volume

Path A — nothing responds

Step Action
Checks Is the proxy running? Does DNS resolve? Is the host reachable?
Resolution Restart the proxy, or fix DNS or the network
Validation The sign-in page loads
Escalate when The proxy is running and the host is reachable and it still fails

Path B — HTTP 502

Step Action
Root cause The proxy is up and cannot reach the application
Checks Is the application process running? Is it listening on the expected port? Does the proxy target that port? Did the application refuse to start?
Resolution Start the application, or align the ports
Validation The health path returns success through the proxy
Escalate when The application is listening on the right port and the proxy still cannot reach it

Read the application's startup log. If it refused to start naming a database object, that is a missing migration, and it is working as intended.

Path C — HTTP 504

Step Action
Root cause The operation exceeded the proxy's timeout
Checks Which operation? Is it inherently slow — a government call, a bulk action, a large report? Is the timeout shorter than the slowest legitimate operation?
Resolution Raise the proxy timeout to accommodate the slow path, or reduce the operation's size
Validation The operation completes without a gateway error
Escalate when Ordinary operations time out

A 504 on a government operation does not mean it failed. It may have completed at the government while the proxy gave up. Check the document's status before retrying, or the retry becomes a permanent duplicate.

Path D — certificate error

Step Action
Root cause Expired, or an incomplete chain
Checks Expiry date. Full chain served. Correct hostname
Resolution Renew or fix the chain
Validation An external TLS checker passes
Escalate when Never — this is infrastructure. Prevent it with expiry monitoring

Browsers repair a missing intermediate certificate; ERP integrations do not. "Browsers fine, integration failing" is the signature of an incomplete chain.

Path E — everything is slow

Isolate the layer before changing anything.

Application Database Meaning Action
High Low Application tier saturated Add capacity, or scale out
Low High Database saturated, or maintenance overdue Check maintenance first — index and statistics work is cheaper than hardware
High High Genuine volume growth Re-size both
Low Low Waiting on something external Provider, network, mail. Not capacity

The final row is the most misdiagnosed. Both tiers idle while the system feels slow means it is waiting, not working. Adding hardware changes nothing.

The failures that look like performance and are not

Symptom Actual cause
Uploads never finish Worker down, or the queue unreachable. Not capacity
Government operations slow or failing Provider latency or outage. Not your infrastructure
Printing fails under load Rendering memory exhausted. A specific limit, not general capacity
One report is slow Data volume or a query, not the system
Slow only at month end Real load concentration. Size for the peak
Slow after an upgrade Statistics stale after schema changes. Run maintenance before assuming a regression
Gradually slower at constant volume Index fragmentation or stale statistics. Maintenance, not hardware

Two of these are worth internalising: check database maintenance before buying hardware, and both tiers idle means external waiting.

Validation

Check Pass condition
The layer was isolated Before any change was made
Maintenance checked before capacity Index and statistics state confirmed
Government slowness attributed correctly Not treated as an infrastructure fault
Timeouts sized for the slow path Not the median
A 504 on a government call was reconciled Status checked before retrying
The fix is verified Measured, not assumed

Troubleshooting

Symptom Cause Action
502 after a deployment The application did not start Read the startup log; it usually names the reason
504 on bulk operations Timeout shorter than the operation Raise it, or reduce batch size
Duplicates after gateway timeouts Retried without reconciling Query status before retrying
Gradual slowdown at constant volume Fragmentation or stale statistics Run maintenance
Sudden slowdown after a bulk load Stale statistics Update them on the affected tables
Periodic pauses Percentage-based file auto-growth Switch to fixed increments and pre-size
Everything idle but slow Waiting on an external dependency Check provider, network and mail
Month-end slowness Real concentration Size for the peak; averages hide it entirely
Bulk print fails, everything else fine Rendering memory Raise memory or reduce batch size
Site down overnight with no change Certificate expired Monitor expiry with 30 days of notice
Application returns after a reboot but the worker does not Supervisor configuration not persisted Save it and test with a real reboot