Rollback Procedure¶
Applies to: All subscriptions
Purpose¶
Return a deployment to its previous working state when an upgrade fails verification — and be clear about which parts of the world cannot be returned at all.
Audience¶
System administrators and implementation engineers, during an upgrade window.
Prerequisites¶
- The verified backup taken immediately before the upgrade
- The previous application artefacts and front-end build
- The previous environment configuration
- Knowledge of whether migrations have been applied
Read this before starting an upgrade, not during a failure. A rollback decided under time pressure by someone reading the procedure for the first time is how a failed upgrade becomes an outage.
Steps¶
First: which side of the line are you on?¶
Verification failed
|
v
Have migrations been applied?
|
+-- No ---> PATH A: swap the files back. Minutes
|
+-- Yes --> PATH B: restore the database. Longer, and it costs
any data written since the backup
Everything in the upgrade sequence before applying migrations is cheap to undo. Everything after is not. That is why the sequence delays migrations as long as it can.
Path A — migrations not yet applied¶
| # | Step |
|---|---|
| A.1 | Stop the application and worker |
| A.2 | Restore the previous application artefacts |
| A.3 | Restore the previous environment configuration |
| A.4 | Restore and publish the previous front-end build |
| A.5 | Start both processes |
| A.6 | Verify: health, sign-in, data, one document end to end |
| A.7 | Confirm to users |
Minutes, and no data loss. This is why keeping the previous front-end build is worth the disk it occupies.
Path B — migrations applied¶
| # | Step | Note |
|---|---|---|
| B.1 | Decide, explicitly | See the decision table below. Restoring loses anything written since the backup |
| B.2 | Stop the application and worker | Cleanly |
| B.3 | Notify integration owners | They must stop sending, or their documents will be lost with the restore |
| B.4 | Restore the database to the pre-upgrade backup | |
| B.5 | Restore the encryption key if it changed | Otherwise encrypted values are unreadable |
| B.6 | Restore the previous application artefacts and configuration | |
| B.7 | Restore and publish the previous front-end build | |
| B.8 | Start both processes | Read the startup log to the end |
| B.9 | Verify fully | Including the cross-tenant check |
| B.10 | Reconcile what was lost | Every document processed between the backup and the restore is gone from Complifly |
| B.11 | Confirm to users and integration owners |
Step B.10 is the one that requires care. Documents processed during the upgrade window exist in the ERP and no longer exist in Complifly, and they must be re-sent.
The decision: restore, or fix forward?¶
| Consider | Restore | Fix forward |
|---|---|---|
| The failure is understood and quickly fixable | Preferred | |
| The failure is not understood | Preferred | |
| Data may be inconsistent | Preferred | |
| Cross-tenant isolation failed | Always restore | Never |
| Little or no data written since the backup | Cheap | |
| Substantial data written since the backup | Costly — weigh carefully | Preferred if safe |
| The window is nearly over | Preferred — a known state beats an unknown one |
The isolation row is absolute. A cross-tenant visibility failure is a data-exposure defect: restore, then investigate. Do not attempt to fix it forward on a live system.
What cannot be rolled back at all¶
A restore returns your system. It does not return the world.
| Action | Reversible? |
|---|---|
| A document registered with the government | No. The number is consumed permanently |
| A cancellation at the government | No |
| An e-Way Bill generated | No. It can be cancelled within its window; it cannot be un-issued |
| A write-back event already delivered to an ERP | No. The ERP has it |
| An email already sent | No |
This is why an upgrade should be verified against the sandbox provider, never production. A verification document sent to production during an upgrade permanently consumes a real document number that a restore cannot recover.
After any rollback¶
| Step | Why |
|---|---|
| Record what failed, precisely | The next attempt depends on it |
| Reconcile documents lost to a restore | They exist in the ERP and not in Complifly |
| Confirm integrations resumed | And that they re-sent what was lost |
| Reproduce the failure in a non-production environment | Before attempting the upgrade again |
| Do not retry the same upgrade unchanged | It will fail the same way |
Validation¶
| Check | Pass condition |
|---|---|
| The system works | Health, sign-in, data, one document end to end |
| Isolation intact | Cross-tenant check performed |
| Encrypted values readable | Sensitive fields decrypt |
| Version consistent | Application and schema match |
| Configuration restored | Settings as before |
| Front end reverted | Users receive the previous bundle |
| Losses reconciled | Every document processed during the window accounted for |
| Integrations resumed | Confirmed by their owners |
Troubleshooting¶
| Symptom | Cause | Action |
|---|---|---|
| The application will not start after rollback | Version and schema mismatch | Ensure both were reverted, not only one |
| Encrypted values unreadable | The key was not restored | Restore the original key |
| Users see nothing after restore | Scope identifiers differ | Reconcile identifiers. Never disable isolation |
| Documents missing after restore | Processed after the backup | Expected. Re-send from the ERP |
| Duplicate rejections when re-sending | Some were registered before the restore | Registration is not undone by a restore. Reconcile by status first |
| Users still on the new front end | Previous build not republished, or caching | Republish and confirm cache headers |
| Rollback took longer than the window | Restore time never measured | Measure it. It is part of the window, not additional to it |
| The same upgrade fails again | Retried without a fix | Reproduce and fix in a non-production environment first |