How to Design Idempotent Salesforce Webhooks & Ingestion Layers

Why retries create duplicates

Most webhook providers deliver at least once. If your endpoint doesn't return a success response quickly, because of a timeout, a 5xx error or a dropped connection, the sender tries again. Salesforce's own outbound messaging documentation warns that a message can be delivered more than once and out of order. Many payment, ecommerce and support platforms describe the same behaviour.

The problem is that a failure doesn't always mean nothing happened. A typical sequence:

1. Webhook evt_123 arrives: "payment succeeded".
2. Your handler inserts a Payment__c record and starts updating the related invoice.
3. The work takes longer than the sender's timeout, or the response is lost on the network.
4. The sender retries evt_123. Your handler inserts a second Payment__c.

Then reports double-count revenue and automation sends two receipts. Somebody in finance ends up reconciling by hand. Manual replays, middleware retries and users re-running failed jobs produce the same effect. Any system that retries needs a receiver that can tolerate it.

Idempotency keys and upsert on external ID

Two mechanisms work together.

1. An idempotency key per event. This is a stable, unique identifier for the delivery intent. Use the provider's event ID where there is one (evt_123). If there isn't, derive a deterministic key from the source system, entity type, entity ID and version or timestamp, for example shop:order:98765:v4. Avoid keys generated at receive time, such as random UUIDs or arrival timestamps, because a retry would get a new one.

2. Upsert on an external ID. The business record carries the source system's own identifier in a custom field marked External ID and Unique, for example Payment__c.Stripe_Charge_Id__c. Upserting on that field updates the existing record instead of creating a new one:

List<Payment__c> payments = PaymentMapper.fromEvents(events);
List<Database.UpsertResult> results =
    Database.upsert(payments, Payment__c.Stripe_Charge_Id__c, false);

for (Integer i = 0; i < results.size(); i++) {
    if (!results[i].isSuccess()) {
        IngestionErrors.record(events[i], results[i].getErrors());
    }
}

The false (allOrNone) flag means one bad record doesn't roll back the other 199. The Unique attribute matters: without it, parallel inserts can each create a record before either can see the other. With it, the database rejects the second insert, and you can treat that as "already processed".

These two keys answer different questions. The event key asks: have I handled this delivery before? The external ID asks: which record does this delivery describe? You need both. A payment can generate several distinct events (created, succeeded, refunded), each with its own event ID but the same charge ID.

Ingestion-layer pattern: queue, validate, upsert

Don't do the business processing inside the webhook request. Separate it into three stages.

Stage 1: receive and acknowledge. Verify the signature, check the basic shape of the payload, store it durably keyed by the idempotency key, and return 2xx quickly. Public webhook senders usually can't run an OAuth flow, so in many designs a thin middleware receiver (for example API Gateway and Lambda) does this step. It then writes to a queue, or to a staging object in Salesforce through an authenticated integration user.

POST /webhooks/payments
    if not verifyHmacSha256(rawBody, header["Signature"], secret): return 401
    event = parse(rawBody)
    if not event.id: return 400
    stored = idempotencyStore.putIfAbsent(event.id, status = "RECEIVED", payload = rawBody)
    if not stored: return 200        // duplicate delivery: acknowledge, do nothing
    queue.send(event.id)
    return 202

Stage 2: validate and transform. A worker reads from the queue, checks the payload against a schema, maps it to Salesforce fields, and rejects anything permanently invalid to a dead-letter queue instead of retrying it forever.

Stage 3: upsert in bulk. Process events in batches, upsert on external IDs, and record the outcome per event. If the staging happens inside Salesforce, a Queueable job can do this:

public with sharing class InboundEventProcessor implements Queueable {
    public void execute(QueueableContext ctx) {
        List<Inbound_Event__c> events = [
            SELECT Id, Event_Id__c, Payload__c, Source_Timestamp__c, Attempts__c
            FROM Inbound_Event__c
            WHERE Status__c = 'Received'
            ORDER BY Source_Timestamp__c ASC
            LIMIT 200
        ];
        if (events.isEmpty()) {
            return;
        }

        List<Payment__c> payments = new List<Payment__c>();
        for (Inbound_Event__c evt : events) {
            payments.add(PaymentMapper.fromPayload(evt.Payload__c));
        }
        List<Database.UpsertResult> results =
            Database.upsert(payments, Payment__c.Stripe_Charge_Id__c, false);

        for (Integer i = 0; i < events.size(); i++) {
            events[i].Status__c = results[i].isSuccess() ? 'Processed' : 'Failed';
            events[i].Attempts__c = (events[i].Attempts__c == null ? 0 : events[i].Attempts__c) + 1;
        }
        update events;
    }
}

Inbound_Event__c.Event_Id__c is itself a unique external ID, so a duplicate insert at receive time fails with DUPLICATE_VALUE. That failure is your deduplication check. Keep the processor bulkified, because a webhook burst behaves just like a data load. The SOQL 101 refactoring guide covers the same principles.

Handling out-of-order events. Store the source's last-modified timestamp or version on the target record, for example Source_Updated_At__c, and skip any event older than what's already stored. Otherwise, a delayed "pending" event can overwrite a "succeeded" status that arrived first.

Protecting side effects. Upsert keeps records unique, but Flows and triggers still fire on every update. Have emails, invoice creation and outbound callouts fire on a state transition (old value ≠ new value), not on every save, so a replayed event doesn't repeat them.

Testing checklist

Before go-live, check that each of these holds:

• Same event twice: delivering an identical payload twice leaves exactly one record and one set of side effects.
• Concurrent duplicates: sending the same event in parallel (say, 10 simultaneous requests) doesn't create duplicates. Expect one success and the rest either acknowledged or rejected as duplicates.
• Different events, same entity: created, updated and refunded events for one charge update a single record in the right order.
• Out-of-order delivery: an older event arriving after a newer one doesn't overwrite newer data.
• Partial batch failure: one invalid record in a batch of 200 fails alone, and the other 199 commit.
• Timeout after commit: simulate the receiver processing the event but the response never reaching the sender, then confirm the retry is harmless.
• Invalid signature: tampered or unsigned payloads are rejected and never stored.
• Bulk burst: several hundred events in quick succession stay within governor and API limits.
• Replay: re-running events from the store or DLQ is safe and produces the same end state.