Skip to content

Integration6 min read

Your webhooks will arrive twice

ARQON EngineeringARQON Singapore

Any integration works on the happy path. Whether you can forget it exists depends on how it behaves on a bad day.

Most of the integration bugs we're asked to fix weren't bugs on launch day. The code did exactly what it was written to do. What changed was the world around it. A vendor resent a webhook. A token expired over a long weekend. Two people edited the same customer record a second apart.

An external API is a distributed system you have no control over. Treating it like a function call that always returns is the most common reason integrations quietly corrupt data.

Assume every message arrives twice, or never

Most webhook providers promise at-least-once delivery, which is a polite way of saying duplicates are your problem. Every handler we write is idempotent: record the event ID before processing, and do nothing if the same ID shows up again. It costs one table and one index. Skip it and, now and then, an invoice goes out twice, and the first person to notice is your customer.

  • Store an idempotency key for every incoming event and every outbound write.
  • Space retries with exponential backoff and jitter, and stop at a ceiling.
  • Send messages you can't process to a dead-letter queue, where a person can inspect and replay them.
  • Never let a sync overwrite a newer value with an older one. Compare a version or timestamp before writing.

Drift will happen, so reconcile

Even a well-built integration drifts. Webhooks get dropped during a vendor outage. Someone edits the database by hand. A nightly job that compares both sides and reports the differences turns silent data corruption into a short list you can read over coffee. On a freight dispatch project, this report is why the planners came to trust the new system. Every difference arrived in writing, every morning.

Make failure visible

An integration without a dashboard hears about its problems from customers. At a minimum, track events received, processed and failed, queue depth, and the age of the oldest unprocessed message. If you set one alert, set it on that last number. It starts climbing before anything else goes wrong.

Filed under Integration · 2026.05.27

Back to the journal

(→)Keep reading

More from the journal.

Integration

· 4 min

When no-code integrations should become code

No-code tools are a great way to start. Here are the signs you've outgrown them, and how to migrate without rebuilding everything at once.

AI

· 7 min

Build the eval before the feature

The fastest way to lose three months on an AI project is to ship the demo first and work out whether it's right later.

Practice

· 5 min

The handover is the product

If your team can't ship on its own three months after we leave, we've failed, whatever else we delivered.