Webhooks and Events¶
Applies to: ERP Write-back · Which modules do I have?
Purpose¶
Help an integration team choose between pulling events and receiving pushed ones, build a consumer that survives real conditions, and recover when delivery has failed.
Audience¶
Integration developers and the customer's integration owner.
Prerequisites¶
- Write-back API — the contract
- Write-back enabled for the relevant registrations
- A subscription configured
Reference¶
Pull or push¶
| Pull | Push | |
|---|---|---|
| Direction | Your system calls Complifly | Complifly calls your endpoint |
| Inbound network path required | No | Yes, reachable and HTTPS |
| Pace control | Yours | Complifly's, with retry and circuit breaking |
| Failure visibility | Immediate to you | Through delivery logs |
| Operational complexity | Lower | Higher |
| Suits | Most integrations, restrictive networks, batch-oriented ERPs | Systems needing low latency and able to expose an endpoint |
Prefer pull unless latency genuinely requires push. It needs no inbound path, no endpoint to secure or keep available, and no certificate on your side. Both carry the same events and the same guarantees.
The properties both share¶
| Property | Consequence for your consumer |
|---|---|
| At-least-once delivery | Must be idempotent on the event identifier. This is not optional |
| Not retrospective | Enabling write-back does not backfill history |
| New event types appear over time | Log and skip unknown types rather than failing |
| Ordering is by sequence, not by wall clock | Use the sequence if order matters to you |
Building a pull consumer, step by step¶
Prefer this path. Work through the steps in order — each is testable on its own.
1. Store a cursor durably¶
Create a persistent record for the position in the feed, alongside your business data. A cursor held only in memory restarts from the beginning after every deploy, or from nowhere.
Check: restart the consumer process and confirm it resumes at the same position.
2. Fetch a page of events¶
GET /api/v1/integration/events?since={cursor}&limit=100
Authorization: Bearer <token>
Check: with a cursor of zero, events are returned. An empty first page usually means write-back was enabled after the documents were processed — it is not retrospective.
3. Deduplicate on the event identifier¶
For each event, look up event_id in your own processed-events store. If present, skip it.
Deduplicate on the event identifier, not the document number — one document produces several events, so keying on the document number silently drops the second and third.
Check: feed the same page twice and confirm the business effect happens once.
4. Apply the event and commit durably¶
Write the outcome against your own record — IRN, e-Way Bill number, cancellation — and record the event_id as processed, in the same transaction.
Check: kill the process between apply and commit; on restart, no event is half-applied.
5. Acknowledge — only after committing¶
POST /api/v1/integration/events/ack
{ "event_ids": ["…", "…"] }
Acknowledge after committing, never before. An event acknowledged and then lost to a crash is gone: the platform has been told you have it.
Check: reverse the order deliberately in a test environment and observe the event disappear. Then put it back.
6. Store the new cursor, and loop¶
Persist next_since and repeat on your schedule.
Check: run a full cycle and confirm the cursor advances and no event is seen twice.
7. Handle unknown event types by skipping, not failing¶
New event types appear over time. A consumer that throws on an unrecognised event_type stops the whole feed the day one is added.
Check: feed a synthetic event with an invented type; the consumer logs it and continues.
The shape of the finished loop:
loop on a schedule:
GET /events?since={cursor}&limit=100
for each event:
if already processed (event_id): skip
else: apply, then record it as processed
commit durably
POST /events/ack { event_ids: [...] } <-- only after committing
store next_since as the new cursor
Building a push consumer, step by step¶
Only take this path if latency genuinely requires it. It adds an inbound network path, an endpoint to secure, and an availability obligation.
1. Expose an HTTPS endpoint¶
Non-HTTPS delivery must never be enabled in production.
Check: the endpoint presents a valid certificate to a client outside your network.
2. Authenticate the caller¶
Verify that requests genuinely come from Complifly before acting on them. An unauthenticated write-back endpoint lets anyone mark documents as registered.
Check: an unsigned or wrongly signed request is rejected.
3. Acknowledge fast, process asynchronously¶
Return promptly and queue the work. A slow endpoint triggers retries, and the retries arrive while the first request is still running.
Check: the endpoint responds well inside the delivery timeout under realistic load.
4. Apply the same idempotency as pull¶
Steps 3, 4 and 7 of the pull procedure apply unchanged — deduplicate on event_id, commit durably, skip unknown types.
Check: deliver the same event twice; the business effect happens once.
5. Watch the circuit breaker¶
Complifly retries a failing endpoint with backoff and opens a circuit breaker after consecutive failures, closing it after a cooldown. That protects the dispatcher from an endpoint that is down. It does not protect you from missing events — that is what the dead-letter path and replay are for.
Check: take the endpoint down deliberately, confirm the breaker opens, restore it, and confirm the dead-lettered events replay.
Recovery¶
| Situation | Recovery |
|---|---|
| Consumer stopped for a while | Resume from the stored cursor. Nothing is lost |
| Cursor lost | Restart from zero and rely on idempotency to discard what you already have |
| Events dead-lettered after exhausting retries | An administrator replays them from the administration surface |
| Endpoint down for a long period | Fix it, then requeue. Preview the requeue first |
| Systems out of step for unknown reasons | Reconcile by document status rather than by replaying events blindly |
Dead-lettered events are retained for a configured period. Make sure that period exceeds your realistic reaction time — a weekend outage with a two-day retention loses the evidence before anyone looks. See Environment Variables.
Monitoring¶
The number that matters is the age of the oldest pending event, not the count. A steady count can hide a stuck head-of-queue; a rising age cannot be hidden.
| Signal | Alert when |
|---|---|
| Oldest pending age | Rising beyond your normal processing interval |
| Consecutive delivery failures | Threshold reached |
| Dead-letter count | Above zero |
| Events processed by your consumer | Falls to zero during business hours |
The last row is monitored on your side. Complifly cannot tell the difference between a stopped consumer and a quiet day.
Validation¶
| Scenario | Expected |
|---|---|
| Normal flow | Events pulled, applied, acknowledged, cursor advanced |
| Duplicate delivery | Processed once |
| Unknown event type | Logged and skipped, not fatal |
| Consumer restart | Resumes from the cursor; nothing lost or repeated |
| Cursor lost | Reprocesses harmlessly |
| Crash between apply and acknowledge | Event redelivered and handled idempotently |
| Push endpoint slow | Complifly retries; the consumer does not duplicate |
| Push endpoint down | Circuit opens; recovery works after the fix |
| Dead-lettered events | Replayed successfully by an administrator |
| Backlog age alert | Fires when the consumer is stopped |
Test the crash-between-apply-and-acknowledge case explicitly. It is the scenario that separates a consumer that works from one that appears to.
Troubleshooting¶
| Symptom | Cause | Action |
|---|---|---|
| No events at all | Write-back disabled, or no subscription | Check the flag first, then the subscription |
| No events for older documents | Not retrospective | Expected. Seeding is a separate exercise |
| The same event repeatedly | Not acknowledged, or acknowledgement failing | Check the acknowledgement response |
| Events processed twice | Not idempotent | Deduplicate on the event identifier |
| Events lost after a crash | Acknowledged before committing | Reverse the order |
| Backlog age rising, count steady | Head of queue stuck | Investigate the oldest event specifically |
| Push deliveries stopped | Circuit breaker opened | Fix the endpoint; delivery resumes after cooldown |
| Dead-lettered events gone | Retention shorter than your reaction time | Raise it |
| Requeue flooded the consumer | Requeued without preview | Preview first; pause the consumer before a large replay |
| Systems out of step | Events missed and never reconciled | Reconcile by document status |
Related Articles¶
- Write-back API — the endpoint contract
- The Integration Pattern — where events sit in the flow
- Write-back Failures — the diagnostic tree
- Key Metrics and Thresholds