Most integration failures are not caused by the API. They are caused by assumptions about delivery, ordering and duplication that were never true.
An integration that works in testing and fails in production is usually not broken code. It is code written against an idealised model of the network: requests arrive once, in order, and succeed or fail cleanly. None of those things are reliably true.
Webhooks arrive more than once
Every serious provider retries webhook delivery, because they cannot distinguish between a response that was never sent and one that was lost in transit. That means your handler will receive duplicates. If receiving the same event twice credits an account twice, the integration is broken and it will take a customer complaint to find out.
The fix is not deduplication bolted on afterwards. It is designing every handler so that processing an event twice produces the same result as processing it once. That means checking whether this event identifier has already been applied, inside the same transaction that applies it.
Events arrive out of order
A 'payment succeeded' event can arrive before the 'payment created' event that logically precedes it. Handlers that assume ordering will create records in impossible states. Reliable integrations either tolerate out-of-order arrival by reconciling against the provider's current state, or explicitly buffer until prerequisites exist.
The provider will have an outage
Not might. Will. The question is what your system does during it. Synchronous calls to a third party inside a user-facing request mean their downtime is your downtime. Moving those calls behind a queue with retry and backoff converts an outage into a delay, which is a far better failure mode.
What reliable integrations have in common
- Idempotency keys on every state-changing request, in both directions
- Signature verification on inbound webhooks, without exception
- Retries with exponential backoff and a dead-letter destination
- Reconciliation against provider state on a schedule, not only on events
- Alerting on the absence of expected traffic, not just on errors
- Credentials scoped narrowly and rotatable without a deployment
The last one is worth emphasising: the most common integration alert is silence. A webhook endpoint that stops receiving traffic looks identical to a quiet business day unless you are explicitly monitoring for it.