Skip to content

Housekeeping

Applies to: All subscriptions

Purpose

Keep storage growth under control without deleting anything that must be retained — and make clear which of the two obligations wins when they conflict.

Audience

Infrastructure teams, database administrators, and the customer's compliance lead for the retention decisions.

Prerequisites

  • Sizing and Capacity
  • The customer's statutory retention obligation, confirmed by their tax and legal leads

Reference

What grows, and whether it may be trimmed

Data Grows with May be purged?
Documents and compliance records Documents processed No — statutory retention
Reference numbers, signed QR codes, acknowledgements Documents registered No — this is the evidence
Stored submission payloads Documents submitted No — evidence of what was sent
Raw ERP payloads Documents ingested Yes, after a configured period
Cached rendered documents Documents printed Yes — regenerable from stored data
Delivered write-back events Events emitted Yes, after a configured period
Dead-lettered write-back events Failures Yes, but see the warning below
Delivery attempt history Delivery attempts Yes, after a period
Upload files Uploads Yes, after processing and a grace period
Application logs Activity Yes, per log retention
Audit trail Activity Per the compliance requirement, not per convenience
Telemetry Activity Yes
Closed Attention Center findings — decisions and lapsed deadlines Findings closed with a recorded decision Only after 8 years — the record of what was decided and why
Closed Attention Center findings — fixed Findings that closed because the document was fixed Yes, after 1 year
Open Attention Center findings Open compliance problems No — never purged while open, however old

The first three rows are the reason storage forecasting matters: they cannot be trimmed to recover space, ever.

The rule when obligations conflict

Statutory retention wins over a data-minimisation preference.

Compliance evidence has a retention obligation that may well exceed what a privacy-minded policy would otherwise prefer. Any deletion policy must be checked against that obligation before implementation, not after a purge.

Verify: The retention period Complifly applies to compliance evidence, and its statutory basis, are not stated in this portal. Confirm before configuring any purge. Recorded as assumption C2 in the Assumptions Register.

The retention settings that matter

Setting Controls Consideration
Raw ERP payload retention How long the original ERP payload is kept Long enough to investigate a mapping dispute
Delivered event retention How long successfully delivered events are kept Long enough to answer "did you send it?"
Dead-letter retention How long failed events are kept Must exceed your realistic reaction time
Cached document retention How long rendered documents are cached Purely a performance and space trade-off
Print horizon How far back documents can be reprinted Affects what must remain retrievable

The dead-letter row is the one with a real trap. A weekend outage with a two-day retention purges the evidence before anyone returns to investigate. Set it against how quickly your team actually responds, including over holidays — not how quickly it intends to.

How a job decides to run

A retention or archive job runs only when two independent switches are both on. Changing one without the other is the most common reason a job "does nothing" or "runs when it should not".

Switch Where Who changes it Effect
Job environment flag (SESSION_REAPER_ENABLED, QUEUE_RETENTION_ENABLED, NOTIFICATION_RETENTION_ENABLED, AUDIT_ARCHIVE_ENABLED, QUARANTINE_ARCHIVE_ENABLED, PDF_CACHE_RETENTION_ENABLED, FINDING_RETENTION_ENABLED) Application server environment Operator, then restart Schedules the job
Retention policy enabled (table hk_retention_policy, one row per managed table) Database Operator, in SQL — no restart Lets the job touch that table. A missing or enabled = 0 row means the job skips the table and does not consume the day's run

The Observability > Housekeeping > Governance panel shows both in one column, per table: Active (both on), Off (both off), Enabled, not scheduled (policy on but the flag is off on that server — nothing will run, and the stale-job alert will fire after 48 hours), Scheduled, policy off (flag on, policy off — nothing runs by design). The flag is read from the server answering the request; confirm on each application server.

The notification ledger has a safety floor. compliance_notification_ledger (the "already told them" rows behind Attention Center emails) is purged under NOTIFICATION_RETENTION_ENABLED, 90 days by default. A value below 14 days in its policy row is ignored and raised to 14, with a warning in the log: deleting a row the email job can still match would let it send the same email again. Closed findings are unaffected.

Attention Center findings use two policy rows for one table. A closed finding is kept 8 years or 1 year depending on how it closed, so hk_retention_policy carries two rows whose table_name is a label, not a table: compliance_finding.COMPLIANCE (2,922 days) and compliance_finding.OPERATIONAL (365 days). Enable each separately; both are scheduled by FINDING_RETENTION_ENABLED. A finding under legal hold (entity type COMPLIANCE_FINDING) is skipped until the hold is released.

To turn a table on: UPDATE dbo.hk_retention_policy SET enabled = 1, updated_by = '<you>', updated_at = SYSUTCDATETIME() WHERE table_name = '<table>'; Changing updated_at also starts the stale-job grace period, so the alert does not fire before the first run.

When jobs run, and what happens after a failure

  • After start-up. Each enabled job runs once about 60 seconds after the server starts, then every 24 hours (weekly for the growth snapshot). Restarts no longer postpone a job by a full day. A job that already ran today (UTC) exits as "already claimed".
  • After a failure. A failed pass is retried automatically up to 3 times, after 15, 30 and 60 minutes. A failed run releases its claim for the day so a retry or another server can run. A DATA_INTEGRITY outcome is not retried — repeating it cannot repair duplicate keys; an engineer must resolve it. Tunable with HOUSEKEEPING_RETRY_MAX and HOUSEKEEPING_RETRY_DELAY_MS; see Environment Variables.
  • Alerts. HOUSEKEEPING_STALE_JOBS (P2): an enabled table with no successful run in 48 hours. HOUSEKEEPING_POLICY_VIOLATIONS (P1): an enabled table whose most recent run failed — it clears after the next successful run, usually the first retry.

Switching a job off, and what can be undone

To stop a job, set its policy row to enabled = 0. It takes effect on the next pass with no restart; a pass already running finishes first. To stop the growth snapshot, which has no policy row, set GROWTH_METRICS_ENABLED=false and restart. Switching a job off never brings back anything it already removed.

Job Effect on data Can you undo it?
Session cleanup, notification retention, queue retention Rows are deleted No. Only a database restore returns them. Confirm the retention window with your compliance lead before enabling
PDF cache retention Cached PDF is deleted Not byte-for-byte. The document is still available through Branded Print, rendered from the stored signed invoice. The official NIC-format print is unavailable once NIC's 48-hour print window has closed, whether or not the cache row exists
Audit archive, quarantine archive Rows move to sys_audit_log_archive / td_invoice_quarantine_*_archive and are deleted from the live table The data is kept, but there is no restore screen or tool. Putting rows back is a documented SQL procedure under row-level-security bypass, run by a Complifly engineer — raise it as a support request. Do not promise a customer an automatic restore. Place a legal hold first, or the next pass archives the row again
Growth snapshot Adds rows to arc_growth_metric and deletes its own snapshots older than 365 days Nothing to undo; the history is only monitoring data

A legal hold is the protection to use before a run, not a rollback after it: place the hold first (see Admin API) and the archive jobs skip that record. Holds are supported on quarantine headers (QUARANTINE_HEADER) and audit-log rows (SYS_AUDIT_LOG, by LogId) only.

The pattern for any purge

Step Action
1 Confirm the statutory position with the tax and legal leads
2 Establish what will be affected, precisely
3 Verify a backup exists and has been verified
4 Run against a non-production copy first
5 Verify the count against storage metadata, not against another scoped query
6 Run in production, in a maintenance window
7 Verify again, the same way
8 Record what was purged, when, and on whose authority

Step 5 exists because of how this database behaves: a query written without tenant scope returns zero rows and reports success. A purge written that way deletes nothing and reports success, and a verification query written the same way agrees with it. Read counts from storage metadata, which isolation policies do not filter.

Controlling growth without deleting

Options that reduce cost without touching retention:

Option Effect
Archive older data to cheaper storage Retained, retrievable, cheaper
Compress at the database level Smaller, transparent
Shorten cache retention Regenerable data only
Shorten raw payload retention Accepting a shorter investigation window
Move log retention to a log platform Off the database entirely

Archival is usually the right first move. It satisfies the retention obligation while removing the cost pressure.

Validation

Check Method Pass condition
Growth matches forecast Measure monthly Within 25 percent, or the forecast revised
Retention jobs run Inspect job history Completing successfully
Purges are effective Count against storage metadata before and after The expected reduction, verified independently
Nothing statutory purged Review the policy against the obligation Confirmed by the tax lead
Dead-letter retention adequate Compare against actual reaction times Exceeds them, including holidays
Archive retrievable Retrieve a sample Succeeds
Free space adequate Check every volume Above the warning threshold
Purges are recorded Review the change log What, when, and on whose authority

Troubleshooting

Symptom Cause Action
A purge reports success and frees nothing Scope not set; isolation filtered every row Set scope and verify against storage metadata
Storage growing faster than forecast Volume above plan, or retention longer than assumed Re-forecast; review retention deliberately
Disk fills despite retention jobs Jobs failing silently, or the growth is in non-purgeable data Check job history first; then check which category is growing
Dead-lettered events gone before investigation Retention shorter than reaction time Raise it
Cannot reprint an older document Beyond the print horizon Adjust the horizon deliberately, understanding the space cost
Evidence unavailable for an audit Purged, or archived without a tested retrieval path Test archive retrieval before relying on it
A purge deleted something needed Statutory position not confirmed first Restore from backup. Confirm the position before any future purge
A job's environment flag is on but nothing is deleted The table's retention policy is enabled = 0 (or missing) Enable the policy row (see How a job decides to run)
Stale-job alert on a table you just enabled The job's environment flag is off on that server, or the first run has not happened yet Check the flag; wait for the first run (about a minute after the next restart)
Logs filling the disk Log rotation absent, or verbose logging left on Configure rotation; return to normal log level