How Do You Build a Webhook Consumer That Remains Correct When Delivery Is Not?

Webhooks look deceptively simple. A provider sends an HTTP request, your application receives some JSON, you inspect the event type, update a database record, return 200 OK, and move on. On the happy path, that model is easy to understand and easy to implement.

The difficulty appears when the transport behaves like a real distributed system. The network can time out after your database transaction has already committed. The same event can arrive twice, or two copies can arrive concurrently. An update can arrive before the event you expected to precede it. Your application can update local state and then crash before sending an email. It can also be unavailable long enough that you are no longer certain which events were missed. At that point, a webhook endpoint stops being merely an HTTP-controller problem and becomes a correctness problem across failure boundaries.

I ran into this distinction while building Featherly, a marketplace application with Stripe billing and Connect payouts across multiple payment flows. The system handled nine webhook event types, including signature verification and retry-safe idempotency. Parsing the payload was not the interesting part. The harder question was:

How can a webhook consumer produce correct business state when the transport delivering events is inherently imperfect?

The answer is not simply “use an idempotency key.” A reliable webhook consumer needs several layers of protection: authentication, durable acceptance, duplicate suppression, idempotent state transitions, asynchronous processing, safe handling of external side effects, observability, and eventually reconciliation. The important idea tying those mechanisms together is that delivery attempts are unreliable, but your business invariants should not be.

The Wrong Mental Model: “Stripe Calls My API Once”

The easiest webhook architecture to imagine looks like this:

The diagram quietly assumes four things: the provider sends one request, your application receives it, your code executes successfully, and the response reaches the provider. But every arrow is a failure boundary. A request can reach your application while the response does not make it back; a database transaction can commit before the process crashes; a downstream service can succeed while your application believes it failed.

Consider a payment event where the database commit succeeds, but the acknowledgement disappears on the network:

From the provider's perspective, the first delivery failed because it never received the acknowledgement. From your application's perspective, the operation succeeded because the database committed. Both sides are behaving correctly according to the information available to them, and that ambiguity is exactly why retries exist.

Stripe's engineering team describes the same class of failure when discussing idempotent APIs: an operation may succeed while the connection breaks before the caller receives the result, leaving the caller uncertain whether retrying is safe. A webhook consumer therefore needs a different mental model:

Delivery attempts are not business operations.

A provider may attempt delivery multiple times. Your system must make those attempts converge on one correct business outcome.

Treat Webhooks as At-Least-Once Delivery

Even if a provider tries hard to avoid duplicates, your application should behave as though an event may arrive more than once. Shopify explicitly tells webhook consumers to expect possible duplicate deliveries and provides a webhook ID for deduplication. It also warns that webhook ordering is not guaranteed and recommends reconciliation rather than assuming every event will arrive successfully. GitHub similarly recommends tracking delivery identifiers and supports redelivery of missed webhook deliveries.

A safer contract for your application is therefore:

An event may arrive:
- once
- multiple times
- later than expected
- concurrently with another copy
- after your system partially processed it

That assumption changes the architecture. Instead of designing around a perfect sequence of requests, you design each stage so that duplicates, delays, crashes, and retries remain recoverable.

Verify Before You Trust

The first requirement is not idempotency; it is authenticity. A public webhook endpoint accepts requests from the internet, so any reliability mechanism built on an unverified event is already on shaky ground. Without signature verification, anyone capable of reaching the endpoint could attempt to manufacture events.

For Stripe, signature verification depends on the exact raw request body, the Stripe-Signature header, and the endpoint secret. Stripe specifically warns that modifying or parsing the body before verification can cause verification to fail. In NestJS, a simplified setup might look like this:

// main.ts
const app = await NestFactory.create(AppModule, {
  rawBody: true,
});

await app.listen(3000);

And then:

@Post("/webhooks/stripe")
async handleStripeWebhook(
  @Req() req: RawBodyRequest<Request>,
  @Headers("stripe-signature") signature: string,
) {
  const event = this.stripe.webhooks.constructEvent(
    req.rawBody!,
    signature,
    process.env.STRIPE_WEBHOOK_SECRET!,
  );

  // Only process the event after verification succeeds.
}

The important architectural ordering is straightforward:

Do not deserialize the payload, trust an event ID, and only later ask whether the request was legitimate. Security should surround the reliability mechanism, not follow it.

Separate “Received” From “Processed”

One of the most useful design changes is separating acceptance from processing. Stripe recommends returning a successful 2xx response quickly, before performing complex logic that might time out. GitHub makes a similar recommendation: respond quickly and move longer processing into an asynchronous queue or worker.

A stronger architecture therefore looks like this:

The endpoint's responsibility becomes deliberately small: authenticate the event, extract its identity, durably record it, and acknowledge receipt. The worker performs the expensive work later. This reduces how much application logic sits inside the provider's retry boundary and gives your own system a durable point from which processing can resume.

Implementation note

A fast acknowledgement is useful only after the event has crossed a durable boundary. Returning 200 before you can recover the event trades provider retries for the possibility of silent loss.

Build an Inbox, Not a “Have I Seen This?” Query

The most obvious deduplication implementation is also the easiest one to get wrong under concurrency:

const existing = await events.findOne({ eventId });

if (existing) {
  return;
}

await events.insertOne({
  eventId,
  status: "pending",
});

Two requests can execute the lookup at the same time:

Request A: event not found
Request B: event not found

Request A: insert
Request B: insert

The check and the write are separate operations, so both requests can observe “not found” before either insert becomes visible. What you actually need is a database-enforced uniqueness guarantee:

await webhookEvents.createIndex({ provider: 1, eventId: 1 }, { unique: true });

Then the delivery handler can attempt the insert and let the datastore arbitrate duplicates atomically:

try {
  await webhookEvents.insertOne({
    provider: "stripe",
    eventId: event.id,
    eventType: event.type,
    payload: event,
    status: "pending",
    receivedAt: new Date(),
  });
} catch (error: any) {
  if (error.code === 11000) {
    // Already accepted this delivery.
    return { received: true };
  }

  throw error;
}

Now uniqueness is determined by the database rather than by the timing between two JavaScript requests. This pattern is often described as an inbox:

(provider, event_id) -> unique

The inbox gives you something more useful than “I think I saw this event before.” It gives you a durable statement that your system has accepted responsibility for the event.

Why persist before returning 200?

Imagine this instead:

queue.add(event);
return { received: true };

If queue.add() only writes to process memory and the application crashes immediately afterward, the provider believes the event was accepted while your application has lost it. The rule I prefer is:

Do not acknowledge a webhook until it exists somewhere durable enough that your own recovery mechanisms can find it.

That durable location could be a database, message broker, or managed queue. The exact technology matters less than the guarantee.

Deduplication Is Not the Same as Idempotency

Suppose you store evt_123, detect every repeated copy of evt_123, and ignore it. That solves one problem, but not the whole problem. Two different events can still represent business operations affecting the same entity:

evt_100 -> invoice.payment_succeeded
evt_101 -> customer.subscription.updated

Event-ID deduplication prevents the same delivery from being processed twice; it does not guarantee that the business operation itself is safe. For example:

if (event.type === "invoice.payment_succeeded") {
  await credits.increment(userId, 100);
}

If another logically equivalent event or retry path reaches that operation under a different event identifier, the customer's balance could still be incremented twice. A more durable design gives the business operation its own identity:

await creditLedger.updateOne(
  {
    operationKey: `invoice:${invoiceId}:credit`,
  },
  {
    $setOnInsert: {
      userId,
      amount: 100,
      source: invoiceId,
      createdAt: new Date(),
    },
  },
  {
    upsert: true,
  },
);

Now the invariant becomes:

One credit per invoice

rather than:

One execution per webhook ID

That distinction is central to reliable webhook design. Transport idempotency protects deliveries; business idempotency protects invariants. The first asks whether you have already accepted this event. The second asks whether running this business operation again can change the result incorrectly.

Out-of-Order Events Are a State-Machine Problem

Duplicates are only one source of uncertainty. Ordering is another. Shopify explicitly notes that webhook events can arrive out of order—even an update can arrive before a corresponding creation event. If your handler simply translates each event into a database assignment, arrival order can accidentally become business truth:

switch (event.type) {
  case "payment.failed":
    order.status = "FAILED";
    break;

  case "payment.succeeded":
    order.status = "PAID";
    break;
}

Suppose the provider created these events in this order:

payment.failed    created at 12:00:01
payment.succeeded created at 12:00:05

but the network delivers them like this:

12:00:06 -> payment.succeeded
12:00:08 -> payment.failed

A naïve last-write-wins implementation ends with FAILED, even though the payment eventually succeeded. Solving that requires reasoning about state rather than blindly trusting arrival order, and the right strategy depends on the domain.

Strategy 1: Validate state transitions

Represent allowed state movement explicitly:

PENDING -> PROCESSING -> PAID
                    \-> FAILED

Then validate a requested transition against the current state:

const allowedTransitions = {
  PENDING: ["PROCESSING", "PAID", "FAILED"],
  PROCESSING: ["PAID", "FAILED"],
  PAID: [],
  FAILED: ["PROCESSING", "PAID"],
};

The exact transition table is domain-specific, but the important shift is that an event proposes a transition; it does not automatically get to overwrite state.

Strategy 2: Compare authoritative versions or timestamps

If the provider supplies meaningful versioning information, process only state that is newer than what you have already observed. This can prevent an older event from overwriting newer state, provided the version or timestamp actually represents an ordering you can trust for that resource.

Strategy 3: Retrieve current state

For particularly important transitions, treat the webhook as a notification that something changed rather than the absolute source of truth. The consumer receives an event and then asks the authoritative API for the resource's current state. That costs another network call, but it can simplify consistency when events describe a resource that may have changed several times before you process the notification.

There is no universally correct answer. That is a recurring theme in distributed systems: stronger correctness often costs latency, throughput, additional network calls, or implementation complexity.

The Hardest Bug Is the Half-Completed Side Effect

Now consider a handler that does three things:

await orders.markPaid(orderId);
await email.sendReceipt(customer);
await analytics.trackPurchase(orderId);

What happens if the process crashes after markPaid() but before sendReceipt()? Retrying the entire handler may safely re-run the database operation if it is idempotent, but external side effects are harder. The email might have been sent even if your application never received a success response. On retry, you may send it twice. If the external call transfers money instead of sending email, the same ambiguity becomes much more serious.

This is why webhook reliability eventually leads to the classic dual-write problem. You are trying to coordinate two things atomically:

Database state
+
Another system

but you do not have one transaction that spans both. One common approach is an outbox-style design:

Instead of doing this:

await database.update();
await externalService.call();

you commit both the business update and the fact that an external action must happen inside the same database transaction:

business update
+
"external action must happen"

A separate dispatcher then performs the external action. This does not magically create exactly-once delivery to another system, but it moves the uncertain boundary somewhere you can observe, retry, and recover intentionally. If the destination supports idempotency keys—as Stripe itself does for mutating API calls—you can combine the outbox operation ID with the destination's idempotency mechanism. Stripe documents idempotency keys specifically to make retries safe when the result of an API request is uncertain.

Exactly-Once Is Usually the Wrong Goal

Webhook systems tempt engineers into saying, “I need this event to execute exactly once.” The problem is that HTTP cannot give your application that property by itself. If acknowledgement can be lost, the sender cannot know whether the receiver completed processing, so it eventually has to choose between two imperfect options:

Do not retry:
    risk losing work

Retry:
    risk duplicates

Reliable systems generally choose retries because lost work is often worse than duplicate attempts that the receiver can make safe. That means the useful goal is not necessarily exactly-once delivery. It is closer to:

At-least-once delivery with effectively-once business behavior.

You accept duplicates as part of the transport model and design operations so repeated execution converges on the same valid business state. Stripe's engineering guidance makes this point indirectly: retries become safe when operations are idempotent, and backoff plus jitter prevents retry behavior from making failures worse. That is a much more practical reliability model than pretending the network can eliminate ambiguity.

Retries Need Their Own Reliability Design

Retries are not free. Suppose a worker fails and immediately retries, then retries again, and again. Now imagine 20,000 events failing because the database is overloaded. If every worker retries at roughly the same time, the recovery mechanism becomes a positive feedback loop:

database overloaded
      ↓
requests fail
      ↓
workers retry
      ↓
more database traffic
      ↓
database becomes more overloaded

Retries can improve reliability during transient failures, but they can also amplify outages. A disciplined retry strategy should account for retry count, exponential backoff, jitter, maximum attempts, dead-letter or failed-event state, and a path to manual or reconciliation-based recovery.

For example:

function retryDelay(attempt: number): number {
  const base = Math.min(1000 * 2 ** attempt, 60_000);
  const jitter = Math.random() * 1000;

  return base + jitter;
}

Stripe's engineering team specifically recommends exponential backoff and randomness to avoid synchronized retry storms. Just as importantly, retries should not continue forever. Eventually an event may need to move to a state such as:

FAILED_PERMANENTLY

or:

NEEDS_RECONCILIATION

A webhook consumer needs a way to admit, “Automatic processing could not establish the correct state.” That is healthier than endlessly retrying an operation that will never succeed, and it gives operators a concrete set of failures to inspect.

Reconciliation Is the Safety Net

Even well-designed webhook infrastructure should not assume the event stream is perfect. Shopify explicitly recommends periodic reconciliation because deliveries can be missed or mishandled. For a payment system, a reconciliation job might periodically compare provider state with local state:

Provider payment state
        vs.
Local payment state

For example:

This changes the reliability model from “every event must work perfectly” to something much more resilient:

Events keep us current quickly.
Reconciliation eventually repairs drift.

Webhooks optimize freshness; reconciliation protects correctness. That distinction is especially useful for workflows where being temporarily stale is acceptable but silently remaining wrong is not.

Observability Should Follow the Event Through the System

A webhook endpoint that returns 200 can still be completely broken. Events may be entering the inbox while workers have stopped processing them. One event type may have started failing after a schema or application change. Duplicate deliveries may suddenly increase. Your oldest pending webhook may have been waiting for 40 minutes while the ingress endpoint remains perfectly healthy.

This is why request-ID structured logging was useful in my own backend work: observability becomes much easier when the provider event identifier follows the workflow through every stage. A useful log might look like:

{
  "provider": "stripe",
  "eventId": "evt_123",
  "eventType": "invoice.payment_succeeded",
  "stage": "processing",
  "attempt": 2,
  "orderId": "ord_456",
  "durationMs": 184,
  "result": "success"
}

Metrics I would want include:

webhooks_received_total
webhooks_duplicate_total
webhooks_signature_failed_total
webhooks_processing_failed_total
webhooks_processing_latency
webhooks_pending_count
oldest_pending_webhook_age
webhooks_dead_lettered_total
reconciliation_drift_total

The most useful alert may not be:

HTTP error rate > 5%

but rather:

oldest pending event > 10 minutes

The first tells you about an endpoint. The second tells you whether the business workflow is actually progressing. For asynchronous systems, that difference matters.

Putting It Together

A robust webhook architecture might look like this:

Notice how little responsibility remains in the HTTP endpoint itself. That is intentional. The request is only the transport boundary: authenticate, durably accept, acknowledge. The business workflow lives beyond it, where processing can be retried, observed, reconciled, and guarded by domain-level invariants.

The architecture also separates different kinds of correctness instead of expecting one mechanism to solve everything. Signature verification establishes authenticity. The inbox protects durable acceptance and duplicate delivery. Business idempotency protects domain invariants. State-transition logic handles ordering. The outbox makes side effects recoverable. Retries handle transient failure, while failed-event state and reconciliation cover the cases automation cannot safely resolve.

The Invariants Matter More Than the Framework

It is easy to frame webhook reliability as a Stripe problem, a NestJS problem, a Kafka problem, or an AWS problem. It is none of those at its core. The deeper design problem is deciding which properties your system must preserve when execution is repeated, interrupted, delayed, or reordered.

For a marketplace, the invariants might be:

A payment must never create two bookings.
One invoice must never grant credit twice.
A payout must never execute twice for the same obligation.
An event already accepted must survive a process restart.
A duplicate delivery must not change the final business result.
A delayed event must not overwrite newer authoritative state.
Failed processing must remain observable and recoverable.

Once those invariants are explicit, implementation choices become easier to evaluate. Ask of every design:

If this line executes twice, what happens?

If this process dies immediately after this write, what happens?

If the response disappears, what happens?

If this event arrives tomorrow, what happens?

If two copies execute concurrently, what happens?

If the event never arrives, how would I know?

Those questions expose reliability problems much faster than staring at the happy path because each one forces you to consider a specific failure boundary and the state left behind.

What I Would Build Today

For a webhook-driven backend handling financially meaningful operations, my baseline architecture would be:

1. Verify the provider signature using the untouched request body.
2. Persist an event identifier in a datastore with a database-enforced uniqueness constraint.
3. Acknowledge only after durable acceptance.
4. Move expensive processing outside the HTTP request lifecycle.
5. Make important business operations idempotent independently of webhook IDs.
6. Represent state transitions explicitly instead of blindly applying events in arrival order.
7. Use idempotency keys or durable operation identifiers for external side effects where possible.
8. Retry transient failures with bounded exponential backoff and jitter.
9. Keep permanently failed events visible rather than silently discarding them.
10. Run reconciliation against authoritative systems for workflows where correctness matters enough.
11. Measure processing lag and workflow completion, not just endpoint availability.

That is more infrastructure than:

@Post("/webhook")
handle(event) {
  // update database
}

But the additional complexity exists for a reason. You are not protecting the system from the normal case; you are protecting it from the uncertain case. The goal is not to make every webhook handler elaborate. It is to add the mechanisms that correspond to the failures your business cannot afford to mishandle.

Final Thought

The most important lesson I learned from building webhook-driven payment flows is that reliability rarely comes from preventing failure. Networks fail, processes crash, responses disappear, requests are duplicated, events arrive late, and dependencies become unavailable. Reliable systems are designed so those failures do not change the correctness of the final outcome.

That is what idempotency is really about. It is not merely “ignore the same webhook twice,” “store the event ID in Redis,” or “retry three times.” It is the broader idea that repeated, delayed, interrupted, or partially completed execution should still converge toward a valid business state.

Once you view webhook handling through that lens, the architecture changes. The webhook endpoint is no longer just a controller; it becomes the entrance to a small distributed system. The question stops being:

“Did my endpoint return 200?”

and becomes:

“Can this system still reach the correct state when delivery is not?”

That is the question production systems actually have to answer.

References

• Stripe Engineering — Designing robust and predictable APIs with idempotency. Discusses ambiguous network failures, idempotency keys, retries, exponential backoff, jitter, and why reliable API operations must tolerate uncertainty. Stripe: Designing robust and predictable APIs with idempotency

• Stripe Documentation — Receive Stripe events in your webhook endpoint. Covers webhook signature verification, raw-body requirements, and returning successful responses before expensive processing. Stripe webhook documentation

• Stripe API Reference — Idempotent requests. Documents Stripe's idempotency-key behavior for safely retrying mutating requests. Stripe idempotent requests documentation

• Shopify Developer Documentation — About webhooks. Documents non-guaranteed webhook ordering, reconciliation recommendations, event identity, and webhook delivery behavior. Shopify webhook documentation

• Shopify Developer Documentation — Verify webhook deliveries. Covers signature verification and handling duplicate webhook deliveries using persistent delivery identifiers. Shopify webhook verification documentation

• GitHub Documentation — Best practices for using webhooks. Recommends fast acknowledgement, asynchronous processing, unique delivery tracking, and recovery of missed deliveries. GitHub webhook best practices