Skip to content

Performance and Availability

Applies to: All subscriptions

Purpose

Isolate the layer responsible when the system is slow or unavailable, and recognise the several failures that present as performance problems and are not.

Audience

Operations teams, infrastructure engineers, support.

Prerequisites

  • Access to monitoring across all four layers
  • Access to application and proxy logs

Steps

The decision tree

  Slow or down
        |
        v
  Does the application respond at all?
        |
        +-- No --> What does the browser show?
        |            |
        |            +-- Connection refused --> Proxy down, or DNS. -> Path A
        |            +-- 502 -----------------> Proxy up, cannot reach API. -> Path B
        |            +-- 504 -----------------> API too slow, or timeout too short. -> Path C
        |            +-- Certificate error ---> Certificate expired. -> Path D
        |
        +-- Yes -> Is everything slow, or one thing?
                     |
                     +-- Everything --> Which layer is saturated? -> Path E
                     |
                     +-- One thing ---> Which one?
                                          |
                                          +-- Government operations -> Provider, not
                                          |                             performance
                                          +-- Uploads ---------------> Worker or queue,
                                          |                             not performance
                                          +-- Printing --------------> Rendering resources
                                          +-- One report ------------> Query or data volume

Path A — nothing responds

Step Action
Checks Is the proxy running? Does DNS resolve? Is the host reachable?
Resolution Restart the proxy, or fix DNS or the network
Validation The sign-in page loads
Escalate when The proxy is running and the host is reachable and it still fails

Path B — HTTP 502

Step Action
Root cause The proxy is up and cannot reach the application
Checks Is the application process running? Is it listening on the expected port? Does the proxy target that port? Did the application refuse to start?
Resolution Start the application, or align the ports
Validation The health path returns success through the proxy
Escalate when The application is listening on the right port and the proxy still cannot reach it

Read the application's startup log. If it refused to start naming a database object, that is a missing migration, and it is working as intended.

Path C — HTTP 504

Step Action
Root cause The operation exceeded the proxy's timeout
Checks Which operation? Is it inherently slow — a government call, a bulk action, a large report? Is the timeout shorter than the slowest legitimate operation?
Resolution Raise the proxy timeout to accommodate the slow path, or reduce the operation's size
Validation The operation completes without a gateway error
Escalate when Ordinary operations time out

A 504 on a government operation does not mean it failed. It may have completed at the government while the proxy gave up. Check the document's status before retrying, or the retry becomes a permanent duplicate.

Path C2 — the IRN Workbench is slow to open, or does not show an older document

Step Action
Root cause Without ENTERPRISE_GRID_V2 the screen loads the newest 1,000 documents in one request and draws all of them: slow to open, slower still with many columns shown (the Full layout), and a registration with more than 1,000 documents cannot reach the older ones at all
Checks How many documents does the registration hold? Which layout is the user on? Is ENTERPRISE_GRID_V2 on for this GSTIN (Operational Controls, Super Admin)?
Resolution Have a Complifly Super Admin switch ENTERPRISE_GRID_V2 on for the GSTIN: the list then loads one page at a time and draws only the rows on screen. If it is already on and a filter by e-way bill status is slow, that is the one known slow path on very large registrations (see List Columns and Saved Layouts API); filter by IRN status or search first
Validation The IRN Workbench shows a page selector under the list and the header counts every document
Escalate when The control is on and a list with no e-way bill filter takes more than a few seconds

Path D — certificate error

Step Action
Root cause Expired, or an incomplete chain
Checks Expiry date. Full chain served. Correct hostname
Resolution Renew or fix the chain
Validation An external TLS checker passes
Escalate when Never — this is infrastructure. Prevent it with expiry monitoring

Browsers repair a missing intermediate certificate; ERP integrations do not. "Browsers fine, integration failing" is the signature of an incomplete chain.

Path E — everything is slow

Isolate the layer before changing anything.

Application Database Meaning Action
High Low Application tier saturated Add capacity, or scale out
Low High Database saturated, or maintenance overdue Check maintenance first — index and statistics work is cheaper than hardware
High High Genuine volume growth Re-size both
Low Low Waiting on something external Provider, network, mail. Not capacity

The final row is the most misdiagnosed. Both tiers idle while the system feels slow means it is waiting, not working. Adding hardware changes nothing.

The failures that look like performance and are not

Symptom Actual cause
Uploads never finish Worker down, or the queue unreachable. Not capacity
Government operations slow or failing Provider latency or outage. Not your infrastructure
Printing fails under load Rendering memory exhausted. A specific limit, not general capacity
One report is slow Data volume or a query, not the system
One list freezes for several seconds after a user chose more columns The browser drawing tens of thousands of cells — Layout Management's Full preset on IRN Workbench is about 70,000. Choose Default, or only the columns needed. Not server capacity
Slow only at month end Real load concentration. Size for the peak
Slow after an upgrade Statistics stale after schema changes. Run maintenance before assuming a regression
Gradually slower at constant volume Index fragmentation or stale statistics. Maintenance, not hardware

Two of these are worth internalising: check database maintenance before buying hardware, and both tiers idle means external waiting.

Path F — a platform health alert arrives

Different entry point from the tree above: nobody reported a problem, an automated email did. Complifly runs a fixed-rule health sweep every 5 minutes (see Logs and Observability); each of these is what the alert actually means, and what to check first.

Alert Meaning First check
SQL connectivity failure The application could not reach the database on two consecutive sweeps (10 minutes) Is SQL Server up? Is the connection string current? Path A/B above if the application itself is also unreachable
PM2 process restart loop One process restarted more than 3 times in a 10-minute window Read that process's startup log — a repeated crash almost always names its own cause on the way down
IRN / e-Way Bill failure spike More than 10% of generation attempts failed in the last 15 minutes Is one GSP down (see the GSP alert below), or is this systemic — a bad deploy, a data issue affecting many documents at once?
GSP success rate degraded A named provider's success rate fell below 50% over 10 minutes Check the provider's own status page first — this is very often their outage, not yours. e-Way Bill Failures / IRN Failures cover the document-level fallout
Audit log write failures At least one write to the compliance audit trail failed in the last hour Database capacity or connectivity — treat as urgent even at one occurrence; this is the compliance record
Elevated upload job failure rate More than 20% of upload jobs failed in the last hour Check the Job Spool Report for the common failure reason — usually a mapping template issue affecting a whole batch, not individual rows

Each condition alerts once when it starts, re-alerts at most every 30 minutes while it stays open, and sends exactly one resolution email when it clears — a repeated email for the same condition means it is still ongoing, not that something new happened.

Validation

Check Pass condition
The layer was isolated Before any change was made
Maintenance checked before capacity Index and statistics state confirmed
Government slowness attributed correctly Not treated as an infrastructure fault
Timeouts sized for the slow path Not the median
A 504 on a government call was reconciled Status checked before retrying
The fix is verified Measured, not assumed

Troubleshooting

Symptom Cause Action
502 after a deployment The application did not start Read the startup log; it usually names the reason
504 on bulk operations Timeout shorter than the operation Raise it, or reduce batch size
Duplicates after gateway timeouts Retried without reconciling Query status before retrying
Gradual slowdown at constant volume Fragmentation or stale statistics Run maintenance
Sudden slowdown after a bulk load Stale statistics Update them on the affected tables
Periodic pauses Percentage-based file auto-growth Switch to fixed increments and pre-size
Everything idle but slow Waiting on an external dependency Check provider, network and mail
Month-end slowness Real concentration Size for the peak; averages hide it entirely
Bulk print fails, everything else fine Rendering memory Raise memory or reduce batch size
Site down overnight with no change Certificate expired Monitor expiry with 30 days of notice
Application returns after a reboot but the worker does not Supervisor configuration not persisted Save it and test with a real reboot
Repeated "PM2 process restart loop" alerts for the same process A genuine crash loop, not sweep noise Read the startup log named in that process's most recent restart
"GSP success rate degraded" alert A named provider's own outage, more often than not Check the provider's status page before assuming an issue on your side
"Audit log write failures" alert, even once Database capacity or connectivity — treat as urgent This is the compliance record, not a diagnostic log