How to Design a Webhook System That Delivers 10 Million Events a Day

Most webhook implementations start the same way: fire an HTTP POST when something happens and hope the other side responds. That works fine at low volume. But at 10 million events per day, every assumption in that naive design will break. This walkthrough from KodeKloud builds a production-grade webhook delivery pipeline in five incremental versions, showing exactly where each one fails and what to fix next.

Version 1: The Naive POST

The simplest approach is to POST directly to the customer's URL from your backend when an event happens, then wait for a 200 OK. Two problems surface immediately. If the customer's endpoint is slow or down, your request handler blocks waiting on their server. And if that request times out, the event is gone because you never persisted it. You have coupled your system's availability to your customer's uptime, and you have no durability guarantee.

Version 2: Write First, Send Later (Transactional Outbox)

The fix is to decouple event creation from event delivery. When an event occurs, write it to a transactional outbox table as part of your database transaction. A separate background worker reads pending events from the outbox and posts them to customer endpoints. This gives you two wins: the request handler returns immediately without waiting on external servers, and if delivery fails, the event is still safe on disk. The outbox pattern is a well-established approach in distributed systems for exactly this reason.

But this introduces a new bottleneck. One worker processes all customers sequentially. If a single customer's endpoint responds slowly, every other customer's events pile up behind it.

Version 3: Fair-Share Dispatcher with Per-Customer Lanes

The solution is a dispatcher that tracks in-flight requests per customer and enforces a cap. If Customer A has already hit their limit (say 5 out of 5 slots full), the dispatcher skips them and routes events for Customer B and Customer K to the worker pool instead. Customer A's slowness only stalls Customer A's lane. Everyone else keeps moving.

The visual from the video makes this concrete: a table tracks each customer's in-flight count against their cap. Customer A shows 5/5 with a stop indicator. Customer B at 2/5 and Customer K at 1/5 both get green checkmarks and continue processing normally.

Version 4: Retry Logic with Exponential Backoff

Once events reach the worker pool, each worker posts with a short timeout (around 10 seconds) and handles the response in three ways:

200 OK: Mark the event as delivered.
Timeout or temporary error (5xx): Retry a few times with increasing wait intervals (exponential backoff).
Permanent error (4xx like a bad URL): Fail fast and stop retrying, because the request will never succeed.

This distinction between transient and permanent failures is the key design decision at this layer. Retrying a 404 wastes resources. Giving up on a 503 too early loses events.

Version 5: Dead Letter Queue and Replay UI

After all retry attempts are exhausted, the event moves to a dead letter queue (DLQ). The event is parked, not deleted. A replay UI (a dashboard) lists all failed webhooks and lets support staff or the customers themselves click a replay button to re-trigger delivery. This closes the loop on reliability: no event is silently dropped, and recovery does not require engineering intervention.

HMAC Signing for Authenticity

One detail that sits outside the delivery pipeline but is equally important: every outbound webhook is signed with an HMAC. The customer can verify the signature to confirm the event actually came from your system and was not spoofed by a third party hitting their endpoint.

The Full Architecture at a Glance

The final pipeline flows through five components in sequence:

Backend emits events and writes them to the Outbox (durable, write-first).
The Dispatcher reads from the outbox and enforces fair-share routing per customer.
The Worker Pool posts signed events with short timeouts.
Workers retry transient failures with exponential backoff.
Permanently failed events land in the Dead Letter Queue with a replay UI for manual recovery.

Key Takeaways

Never POST inline from your request handler. Decouple event creation from delivery using a transactional outbox so events survive failures.
Isolate customers from each other. A per-customer dispatcher with in-flight caps prevents one slow endpoint from degrading your entire platform.
Distinguish transient from permanent errors. Retry 5xx responses with backoff, but fail fast on 4xx errors that will never resolve.
Dead letter queues preserve failed events. Combined with a replay UI, they let you recover without code changes or manual database work.
Sign every webhook with HMAC. Customers need a way to verify that incoming events are authentic and untampered.