External website; opens in a new tab.

Automation

What should happen when an integration fails halfway through?

Record durable state, bound retries and reconcile uncertain effects; never blindly replay a consequential write after a timeout.

Automation

Source review 21 November 2026

Suppose one system accepts a new order but the integration loses its connection before the reply arrives. Retrying may create a duplicate, while doing nothing may leave the customer waiting. A robust design treats that uncertainty as a normal possibility rather than assuming every failed network request means nothing happened.

Define a meaningful checkpoint

Track the business record, source event, attempted action and current known outcome. Use a stable identifier and an idempotency or deduplication strategy suitable for the destination. An event timestamp alone may not uniquely identify the intended work. Preserve enough state across restarts that a new process can recover without guessing from an operator’s memory.

Separate accepted, processing, completed, rejected and unknown states where the provider contract supports them. A provider may return quickly while performing the actual work later. Similarly, a delivered webhook can report a change which has already been seen or arrives out of order. The receiving system must check current state rather than apply every message blindly.

Retry safely and stop visibly

Use bounded retries with an appropriate delay/backoff for temporary faults. Do not retry a permanently invalid request forever or keep hammering an account that has revoked access. After the agreed limit, put the item into an operator-visible exception/dead-letter route with a safe summary and next action. Avoid logging secrets or whole confidential payloads simply to make diagnosis convenient.

Before retrying a financial, communication or privileged action, verify whether the effect already occurred. Query a safe provider reference where available or use the agreed reconciliation process. “Retry all failed jobs” is a dangerous button if a failed response can conceal a successful invoice or email.

Test recovery, not only delivery

Exercise duplicated and delayed events, timeouts after acceptance, expired credentials, invalid input, supplier outage and process restart. Confirm both source and destination after recovery. Test that an unrelated customer/project cannot receive the job or its error detail. Demonstrate that an operator can distinguish a current worker signal from a stale running record.

Document who owns exception review, how long records are kept and what changes need renewed approval. A good handover explains which operations are safe to replay and which require a separate consequential decision. It also includes a manual fallback so a supplier outage does not make the whole business process unknowable.

Reliable integration is not achieved by hiding errors behind a green dashboard. It comes from truthful state, a clear authority boundary and a recoverable path through the cases where the network cannot tell you exactly what happened.

Sources and review

Prepared by Yellow Brick Tech. General guidance; the source and test conditions for your own situation still matter.

Next review: 4 May 2027

Related services

How Yellow Brick Tech can help.

Next step

Need to apply this to a real situation?

Ask a question or choose the closest project route. A short conversation can often clarify the next step.