Why the same event arrives twice
Duplicate delivery is not one bug with one fix. It is four independent mechanisms with the same symptom, and one handler has to survive all four — so the answer is a property of your write path, not a patch on whichever mechanism you noticed first.
Your endpoint committed the change, then timed out. You wrote the payment,
then spent a few more seconds sending a receipt, and the provider gave up
waiting before your 200 arrived. From its side the delivery failed. From yours
it succeeded completely.
The provider retried after a genuine failure. A redeploy, an exhausted connection pool, a dependency down. Some of those failures happen after a partial write.
Somebody pressed resend. Support staff replay events from a provider dashboard while investigating a discrepancy. At your endpoint that is indistinguishable from a retry, and it can arrive days after the original.
Your consumer restarted mid-processing. With an at-least-once queue between the endpoint and the business logic, an unacknowledged message is redelivered after a crash — a duplicate your provider never sent and never sees.
Only the second is a failure in the ordinary sense. The other three happen while everything works as designed, which is why “we will fix the timeouts” is not a strategy. When one gets through, one payment posts to the ledger twice, sends two confirmation emails, decrements stock twice, or fulfils a digital order twice. The email is harmless and immediately visible; the ledger entry is costly and silent until reconciliation.
Handling it correctly
The rule is short: deduplicate inside the same transaction that changes business state. Anything else leaves a window.
Use a unique constraint, not a read-then-write check. The intuitive implementation asks the database whether the event was already processed and inserts a record if it was not. Two deliveries handled at the same time — two workers, or a retry overlapping a slow first attempt — both read “absent” before either writes, and both proceed. A unique constraint on the deduplication key moves that decision into the database: exactly one writer wins, and the other gets a violation it can safely treat as “already done”. Insert that row in the same transaction as the ledger entry; if they can commit separately, they eventually will.
Choose the key carefully. The provider’s event ID is the obvious first move, and it is correct for the plain case: the same event redelivered carries the same ID. It is not sufficient. Some providers emit more than one event object for a single underlying occurrence, each with its own event ID, and to that key they are two different events. Key on the identifier of the object the event is about together with the event type instead. The provider notes below record where each provider documents this.
Return the success status before the slow work. Acknowledge, then process asynchronously. This shrinks the timeout-retry mechanism, the most common source of duplicates. It does not remove the need for idempotency; it stops you manufacturing extra duplicates yourself.
One neighbouring hazard, named so it is not confused with this one: several different events can be generated for one business transition, and they are not guaranteed to arrive in the order they were created. Those are not duplicates and deduplication will not help. They need a handler that does not depend on arrival order.
Testing it, and why a naive test passes
A duplicate the provider actually sends reuses the exact same raw body and event ID and carries a freshly computed, independently valid signature with its own timestamp. Both properties matter, and a test that gets either one wrong is worse than no test, because it reports success.
- Deduplicating on the signature header. Each delivery is signed independently, so the header differs between the original and the duplicate. Key on it and you never recognise a duplicate at all — and a harness that replays a recorded request byte for byte will never reveal that, because there the headers do match.
- Replaying a recorded request verbatim. This exercises a case the provider never produces. Depending on the signature tolerance window it passes trivially or fails for a reason unrelated to idempotency.
The scenario below delivers the same event twice the way a provider does: one immutable body and event ID, one fresh signature per attempt. Assert its two counts separately — two deliveries, one logical event. An integration that records two payment events because it received two deliveries has already lost.
What this scenario does not prove
The duplicate is delivered sequentially: the second request is sent only after the first completed and returned a success status. That is the common shape, and it is not the hard case. Two deliveries processed in parallel, by two workers or by a retry overlapping a slow handler, can both pass a read-then-write check and both commit. A unique constraint survives that; “select, then insert if absent” does not, and this scenario will not tell you which one you have.
It does not demonstrate the two-event-objects hazard either. The key above uses the object identifier together with the event type, rather than the event ID alone, because a provider can emit two separate event objects — each with its own event ID — for one occurrence. This scenario emits one event per canonical transition, so it delivers one event ID twice and never two IDs for one payment. That half of the recommendation rests on the provider documentation cited below, not on the run.