Knowledge Guide
Home › Hands-On Builds › Fintech Labs

Design a Webhook Delivery System

Design a Webhook Delivery System

Every payments platform ends up building the same subsystem: a way to tell a merchant's server "the charge succeeded" without that merchant polling your API forever. Stripe calls it Events & Webhooks. Razorpay calls it Webhooks. Both ship the exact same three mechanisms under the hood, and both publish enough of their own docs that you can build the real thing instead of a toy. This lab is the sender side of that system — a delivery worker that pushes events to subscriber URLs — plus the receiver-side contract a subscriber MUST implement to not get burned by it. The interview-relevant gap: anyone can draw "queue → POST to URL → done" on a whiteboard. The senior-level version is explaining, with numbers, why that naive version either double-charges a customer or DDoSes your own subscriber, and which two mechanisms close each hole. See also the concept-level comparison of Polling vs LongPolling vs WebSockets vs Webhooks for why push (webhooks) wins here over the alternatives before you build any of this.

1. The trap

You ship the obvious version first: a background job that, on payment.succeeded, does a single synchronous POST to the merchant's configured URL, right there inside the request/response cycle that processed the payment. It works in the demo. Then three things happen in production, in this order, and each one is a real incident shape reported by every payments platform that has ever shipped webhooks:

Both failure shapes trace back to the same root cause: at-least-once delivery is the ONLY delivery guarantee that is actually achievable over an unreliable network (Movement 3 derives why), and at-least-once means duplicates are not a bug in your system — they are a guaranteed, expected input that the receiver must be built to handle. See Traffic Amplification: Fan-out & Retry Storms for the general shape of the second failure at fleet scale.

2. Scope it like a senior

Before writing a line of code, pin down the questions that decide the whole design:

3. Reason to the design

Attempt 0 — deliver, and if it fails, don't retry at all (at-most-once). Simplest possible thing: POST once, log success/failure, move on. This fails the product requirement immediately — a transient blip in the merchant's server (a deploy, a GC pause, one dropped TCP packet) silently loses a "your payment succeeded" notification forever. For anything involving money, silently dropping the notification is worse than any of the alternatives below. Rejected.

Attempt 1 — retry until it succeeds, immediately (naive at-least-once). Fixes the silent-drop problem: keep retrying on failure. But "immediately" is the trap from Movement 1 — a subscriber that is down or slow gets hit at the sender's maximum possible request rate, which is exactly the retry-storm failure. You've traded "silently loses events" for "can DDoS a struggling subscriber," which is arguably worse, since now the retries themselves are actively making the subscriber's outage longer.

Attempt 2 — retry with a FIXED delay (e.g. always wait 5 seconds). Better: at least you're not hammering at max speed. But a fixed delay is wrong at both ends of the failure spectrum. Too short for an outage measured in hours (you still fire thousands of retries over that outage). Too long for a one-off blip that would've recovered in 200ms (you've now added needless latency to a notification that could have succeeded quickly). You need the delay to grow with how long the failure has persisted — you don't know in advance whether this is a 200ms blip or a 6-hour outage, so let the schedule discover it.

Attempt 3 — exponential backoff: double the delay each failed attempt, cap it. delay = min(maxDelay, base * 2^(attempt-1)). This is directionally right and is the core of both Stripe's and Razorpay's real schedules. But pure exponential backoff has a subtler failure mode at fleet scale: if a subscriber's endpoint goes down and you have thousands of events queued for it (a batch of payments all failed webhooks at once), every one of those deliveries computes the exact same delay at the exact same wall-clock moment — because they all started failing together and the formula is deterministic. They all retry in lockstep, in one synchronized burst, at attempt 2's delay, then attempt 3's delay, and so on. That's the "thundering herd" pattern: not continuous hammering, but periodic synchronized spikes.

The fix: jitter. Instead of retrying at exactly delay, retry at a random point between 0 and delay — AWS's "full jitter" formula, which is what both this lab's reference implementations use. Now each queued delivery for the same down subscriber picks a different actual wait time, spreading what would have been one synchronized burst into a smooth trickle. Exponential backoff solves "don't retry too fast forever"; jitter solves "don't retry all-at-once even when slow." You need both — see The Architecture of the Retry Pattern for the general form of this mechanism outside the webhook context.

The final piece: give up. An endpoint that is down for days will exhaust any finite retry budget. At that point the correct move isn't "retry forever" (unbounded backlog, unbounded cost) — it's give up on this attempt, and if the endpoint has been failing continuously past some threshold, auto-disable it and notify the merchant out-of-band (email, dashboard banner) that their webhook endpoint is broken. This is exactly what both Stripe and Razorpay do.

4. Build it — milestones

Attempt each milestone before reading the reference implementation below.

Reference implementation — Java, sender (delivery worker: backoff + jitter + HMAC signing)

The whole retry-storm fix lives in one method, backoffDelayMs: exponential growth, a hard cap, then full jitter. The signing binds the timestamp INTO the signed bytes, which is exactly why a captured-and-replayed request can't be re-signed later with a forged fresh timestamp.

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.util.Random;
import java.util.concurrent.atomic.AtomicInteger;

/** Sender side: at-least-once delivery with exponential backoff + full jitter,
 *  HMAC-SHA256 signing with a timestamp, give-up + auto-disable after N attempts. */
public final class WebhookSender {

    // ---- the event and the "subscriber" (stand-in for an HTTP POST over the wire) ----
    static final class Event {
        final String id;
        final String payload;
        Event(String id, String payload) { this.id = id; this.payload = payload; }
    }

    static final class DeliveryResult {
        final int statusCode;
        DeliveryResult(int statusCode) { this.statusCode = statusCode; }
        boolean isSuccess() { return statusCode >= 200 && statusCode < 300; }
    }

    interface Subscriber {
        DeliveryResult deliver(String payload, String eventId, long timestamp, String signature);
    }

    // ---- HMAC-SHA256 signing, timestamp bound into the signed message (anti-replay) ----
    static final class HmacSigner {
        static String sign(String secret, String payload, long timestampEpochSeconds) {
            try {
                String signedPayload = timestampEpochSeconds + "." + payload;   // Stripe's own scheme: t.payload
                Mac mac = Mac.getInstance("HmacSHA256");
                mac.init(new SecretKeySpec(secret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
                byte[] raw = mac.doFinal(signedPayload.getBytes(StandardCharsets.UTF_8));
                StringBuilder hex = new StringBuilder(raw.length * 2);
                for (byte b : raw) hex.append(String.format("%02x", b));
                return hex.toString();
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        }
    }

    // ---- the delivery worker: pulls an event, POSTs, retries with backoff+jitter, gives up ----
    static final class DeliveryWorker {
        final String secret;
        final int maxAttempts;
        final long baseDelayMs;
        final long maxDelayMs;
        final boolean useBackoff;     // false => immediate retry (Movement 5's retry-storm bug)
        final Random rng = new Random(42);   // fixed seed: reproducible demo numbers

        DeliveryWorker(String secret, int maxAttempts, long baseDelayMs, long maxDelayMs, boolean useBackoff) {
            this.secret = secret;
            this.maxAttempts = maxAttempts;
            this.baseDelayMs = baseDelayMs;
            this.maxDelayMs = maxDelayMs;
            this.useBackoff = useBackoff;
        }

        /** full-jitter exponential backoff: delay = random(0, min(maxDelay, base * 2^(attempt-1))) */
        long backoffDelayMs(int attempt) {
            if (!useBackoff) return 0L;                                   // the bug under test in Movement 5
            long capped = Math.min(maxDelayMs, baseDelayMs * (1L << (attempt - 1)));
            return (long) (rng.nextDouble() * capped);                    // AWS "full jitter" formula
        }

        /** Returns true if delivered, false if the endpoint was given up on / auto-disabled. */
        boolean deliver(Event e, Subscriber sub, StringBuilder log) {
            for (int attempt = 1; attempt <= maxAttempts; attempt++) {
                long ts = System.currentTimeMillis() / 1000L;
                String sig = HmacSigner.sign(secret, e.payload, ts);
                DeliveryResult r = sub.deliver(e.payload, e.id, ts, sig);
                if (r.isSuccess()) {
                    log.append(String.format("  attempt %d -> %d OK, delivered%n", attempt, r.statusCode));
                    return true;
                }
                log.append(String.format("  attempt %d -> %d FAIL%n", attempt, r.statusCode));
                if (attempt < maxAttempts) {
                    long delay = backoffDelayMs(attempt);
                    log.append(String.format("    backing off %d ms before retry %d%n", delay, attempt + 1));
                    sleep(delay);
                }
            }
            log.append(String.format("  GIVE UP after %d attempts -> endpoint AUTO-DISABLED%n", maxAttempts));
            return false;
        }

        private static void sleep(long ms) {
            if (ms <= 0) return;
            try { Thread.sleep(ms); } catch (InterruptedException ignored) { Thread.currentThread().interrupt(); }
        }
    }

    // ---- fake subscribers used to exercise the worker ----

    /** Fails the first `failCount` calls, then succeeds -- a flapping endpoint recovering. */
    static final class FlappingSubscriber implements Subscriber {
        private final AtomicInteger calls = new AtomicInteger(0);
        private final int failCount;
        FlappingSubscriber(int failCount) { this.failCount = failCount; }
        public DeliveryResult deliver(String payload, String eventId, long ts, String sig) {
            int n = calls.incrementAndGet();
            return n <= failCount ? new DeliveryResult(503) : new DeliveryResult(200);
        }
        int callCount() { return calls.get(); }
    }

    /** Never recovers -- exercises the give-up / auto-disable path. */
    static final class AlwaysFailingSubscriber implements Subscriber {
        public DeliveryResult deliver(String payload, String eventId, long ts, String sig) {
            return new DeliveryResult(500);
        }
    }

    /** An always-failing endpoint that answers instantly -- used to measure request rate. */
    static final class InstantFailSubscriber implements Subscriber {
        private final AtomicInteger calls = new AtomicInteger(0);
        public DeliveryResult deliver(String payload, String eventId, long ts, String sig) {
            calls.incrementAndGet();
            return new DeliveryResult(500);
        }
        int callCount() { return calls.get(); }
    }

    public static void main(String[] args) throws Exception {
        String secret = "whsec_demo_shared_secret";

        System.out.println("=== 1. Normal delivery, first attempt succeeds ===");
        DeliveryWorker normal = new DeliveryWorker(secret, 6, 100, 2000, true);
        StringBuilder log1 = new StringBuilder();
        boolean ok1 = normal.deliver(new Event("evt_1", "{\"type\":\"payment.succeeded\"}"),
                new FlappingSubscriber(0), log1);
        System.out.print(log1);
        System.out.println("delivered=" + ok1);

        System.out.println();
        System.out.println("=== 2. Flapping endpoint: fails 3x, then recovers under backoff+jitter ===");
        DeliveryWorker backoffWorker = new DeliveryWorker(secret, 6, 100, 2000, true);
        StringBuilder log2 = new StringBuilder();
        long t0 = System.currentTimeMillis();
        boolean ok2 = backoffWorker.deliver(new Event("evt_2", "{\"type\":\"invoice.paid\"}"),
                new FlappingSubscriber(3), log2);
        long elapsed2 = System.currentTimeMillis() - t0;
        System.out.print(log2);
        System.out.println("delivered=" + ok2 + ", wall time=" + elapsed2 + "ms (attempts spaced out by design)");

        System.out.println();
        System.out.println("=== 3. Endpoint never recovers: give up after maxAttempts, auto-disable ===");
        DeliveryWorker giveUpWorker = new DeliveryWorker(secret, 5, 50, 800, true);
        StringBuilder log3 = new StringBuilder();
        boolean ok3 = giveUpWorker.deliver(new Event("evt_3", "{\"type\":\"charge.failed\"}"),
                new AlwaysFailingSubscriber(), log3);
        System.out.print(log3);
        System.out.println("delivered=" + ok3 + " -> subscriber endpoint would be marked DISABLED in the DB");

        System.out.println();
        System.out.println("=== 4. BREAK IT (b): immediate retry (no backoff) vs backoff -- the retry storm ===");
        final long windowMs = 300;

        // (a) no backoff at all: hammer the subscriber as fast as the loop can spin.
        InstantFailSubscriber stormTarget = new InstantFailSubscriber();
        DeliveryWorker stormWorker = new DeliveryWorker(secret, Integer.MAX_VALUE, 0, 0, false); // useBackoff=false
        long stormStart = System.currentTimeMillis();
        int stormAttempt = 0;
        while (System.currentTimeMillis() - stormStart < windowMs) {
            long ts = System.currentTimeMillis() / 1000L;
            String sig = HmacSigner.sign(secret, "{}", ts);
            stormTarget.deliver("{}", "evt_storm", ts, sig);
            stormAttempt++;
        }
        System.out.println(String.format(
            "  NO-BACKOFF worker hammered the endpoint %d times in %dms (~%.0f req/s)",
            stormTarget.callCount(), windowMs, stormTarget.callCount() * 1000.0 / windowMs));

        // (b) same window, WITH exponential backoff+jitter: far fewer requests reach the subscriber.
        InstantFailSubscriber politeTarget = new InstantFailSubscriber();
        DeliveryWorker politeWorker = new DeliveryWorker(secret, Integer.MAX_VALUE, 50, 500, true);
        long politeStart = System.currentTimeMillis();
        int politeAttemptCount = 0;
        while (System.currentTimeMillis() - politeStart < windowMs) {
            long ts = System.currentTimeMillis() / 1000L;
            String sig = HmacSigner.sign(secret, "{}", ts);
            politeTarget.deliver("{}", "evt_storm", ts, sig);
            politeAttemptCount++;
            long delay = politeWorker.backoffDelayMs(politeAttemptCount);
            try { Thread.sleep(delay); } catch (InterruptedException ignored) {}
        }
        System.out.println(String.format(
            "  BACKOFF worker made only %d request(s) to the same endpoint in the same %dms window (~%.0f req/s)",
            politeTarget.callCount(), windowMs, politeTarget.callCount() * 1000.0 / windowMs));
        System.out.println("  Same failing endpoint, same wall-clock window -- backoff is the only difference,"
                + " and it is the difference between a request storm and a handful of polite retries.");
    }
}

Compiled and executed with javac/java (JDK 8) before publishing — zero warnings, zero errors.

Reference implementation — Java, receiver (verify + dedup + fast-2xx-then-async)

Three independent responsibilities, none optional: Verifier proves the request is really from you and isn't a replay; SafeReceiver makes the side effect idempotent under duplicate delivery; AsyncReceiver decouples "tell the sender we got it" from "actually do the work."

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

/** Receiver side: verify signature + timestamp (reject replays), dedup by event-id
 *  (at-least-once means duplicates ARE coming), ack fast then process off the request thread. */
public final class WebhookReceiver {

    // ---- signature verification (mirrors WebhookSender.HmacSigner) ----
    static final class Verifier {
        static String sign(String secret, String payload, long timestampEpochSeconds) {
            try {
                String signedPayload = timestampEpochSeconds + "." + payload;
                Mac mac = Mac.getInstance("HmacSHA256");
                mac.init(new SecretKeySpec(secret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
                byte[] raw = mac.doFinal(signedPayload.getBytes(StandardCharsets.UTF_8));
                StringBuilder hex = new StringBuilder(raw.length * 2);
                for (byte b : raw) hex.append(String.format("%02x", b));
                return hex.toString();
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        }

        /** Constant-time compare -- do not use String.equals for secrets/signatures. */
        static boolean constantTimeEquals(String a, String b) {
            if (a.length() != b.length()) return false;
            int diff = 0;
            for (int i = 0; i < a.length(); i++) diff |= a.charAt(i) ^ b.charAt(i);
            return diff == 0;
        }

        /** Reject if the signature doesn't match OR the timestamp is outside the replay window. */
        static boolean verify(String secret, String payload, long timestamp, String signature,
                               long nowEpochSeconds, long toleranceSeconds) {
            String expected = sign(secret, payload, timestamp);
            if (!constantTimeEquals(expected, signature)) return false;              // tampered / wrong secret
            long skew = Math.abs(nowEpochSeconds - timestamp);
            return skew <= toleranceSeconds;                                        // outside window = replay
        }
    }

    /** BROKEN receiver: no dedup. Every delivery attempt that reaches here re-runs the side effect. */
    static final class NaiveReceiver {
        final AtomicInteger walletCreditedCents = new AtomicInteger(0);
        void process(String eventId, int amountCents) {
            walletCreditedCents.addAndGet(amountCents);   // credits the wallet -- AGAIN, on every duplicate
        }
    }

    /** FIXED receiver: dedup by event-id before the side effect runs -- idempotent under at-least-once. */
    static final class SafeReceiver {
        final AtomicInteger walletCreditedCents = new AtomicInteger(0);
        final Set<String> seenEventIds = ConcurrentHashMap.newKeySet();
        /** returns true if this call actually applied the side effect (first time seeing this event id). */
        boolean process(String eventId, int amountCents) {
            if (!seenEventIds.add(eventId)) return false;      // Set.add returns false if already present
            walletCreditedCents.addAndGet(amountCents);
            return true;
        }
    }

    /** The fast-2xx-then-async shape: ack immediately, hand the real work to a background executor. */
    static final class AsyncReceiver {
        final SafeReceiver downstream = new SafeReceiver();
        final ExecutorService pool = Executors.newSingleThreadExecutor();
        final AtomicInteger acked = new AtomicInteger(0);
        final AtomicInteger processed = new AtomicInteger(0);

        /** Returns the ack latency in nanoseconds -- this is what the sender's socket sees. */
        long receive(final String eventId, final int amountCents, final long simulatedProcessingMs) {
            long start = System.nanoTime();
            acked.incrementAndGet();                            // "200 OK, we have it" -- decided instantly
            pool.submit(new Runnable() {
                public void run() {
                    try { Thread.sleep(simulatedProcessingMs); } catch (InterruptedException ignored) {}
                    if (downstream.process(eventId, amountCents)) processed.incrementAndGet();
                }
            });
            return System.nanoTime() - start;                   // the caller (sender) only waits for THIS
        }

        void shutdownAndAwait() throws InterruptedException {
            pool.shutdown();
            pool.awaitTermination(5, TimeUnit.SECONDS);
        }
    }

    public static void main(String[] args) throws Exception {
        String secret = "whsec_demo_shared_secret";
        String payload = "{\"type\":\"payment.succeeded\",\"amount\":5000}";
        long now = System.currentTimeMillis() / 1000L;
        String sig = Verifier.sign(secret, payload, now);

        System.out.println("=== 1. Signature + timestamp verification ===");
        System.out.println("  fresh, correctly signed  -> accept? " + Verifier.verify(secret, payload, now, sig, now, 300));
        System.out.println("  tampered payload         -> accept? " + Verifier.verify(secret, "{\"type\":\"payment.succeeded\",\"amount\":999999}", now, sig, now, 300));
        long staleTs = now - 900; // 15 minutes old
        String staleSig = Verifier.sign(secret, payload, staleTs);
        System.out.println("  correct sig but 900s old -> accept? " + Verifier.verify(secret, payload, staleTs, staleSig, now, 300)
                + "  (REPLAY rejected: outside the 300s tolerance window)");

        System.out.println();
        System.out.println("=== 2. BREAK IT (a): at-least-once WITHOUT dedup -- double side effect ===");
        NaiveReceiver naive = new NaiveReceiver();
        // Same event, id "evt_42", delivered TWICE -- e.g. the sender's ack was lost on the wire
        // so it retried, or a worker crashed after processing but before acking the queue message.
        naive.process("evt_42", 5000);
        naive.process("evt_42", 5000);   // duplicate delivery of the SAME event id
        System.out.println("  wallet credited (naive, no dedup): " + naive.walletCreditedCents.get()
                + " cents  <- BUG: customer paid once, wallet credited twice ($" + (naive.walletCreditedCents.get() / 100.0) + ")");

        System.out.println();
        System.out.println("=== 3. FIX: same duplicate delivery, WITH event-id dedup ===");
        SafeReceiver safe = new SafeReceiver();
        boolean first = safe.process("evt_42", 5000);
        boolean second = safe.process("evt_42", 5000);   // duplicate again
        System.out.println("  first delivery applied?  " + first);
        System.out.println("  duplicate applied?       " + second + "  (skipped -- already seen this event id)");
        System.out.println("  wallet credited (safe, with dedup): " + safe.walletCreditedCents.get()
                + " cents  <- correct: exactly one credit despite two deliveries");

        System.out.println();
        System.out.println("=== 4. Fast-2xx-then-async: ack latency vs. real processing latency ===");
        AsyncReceiver async = new AsyncReceiver();
        long ackNanos = async.receive("evt_99", 1234, 200 /* simulated slow downstream work, ms */);
        System.out.println("  ack returned to sender in " + (ackNanos / 1000) + " microseconds"
                + " (well under any reasonable webhook timeout)");
        Thread.sleep(50);
        System.out.println("  processed count immediately after ack: " + async.processed.get()
                + "  (0 -- the 200ms of real work is still running in the background)");
        async.shutdownAndAwait();
        System.out.println("  processed count after the background job finishes: " + async.processed.get()
                + "  (1 -- decoupled from the sender's retry clock entirely)");
    }
}

Compiled and executed with javac/java (JDK 8) before publishing — zero warnings, zero errors.

Reference implementation — Go, sender (backoff + jitter + HMAC signing)

Same mechanism as the Java version, idiomatic Go: an interface for the subscriber, crypto/hmac for signing, math/rand for jitter.

package main

import (
	"crypto/hmac"
	"crypto/sha256"
	"encoding/hex"
	"fmt"
	"math/rand"
	"sync/atomic"
	"time"
)

// Event is a single webhook event to deliver at least once.
type Event struct {
	ID      string
	Payload string
}

// DeliveryResult is what the subscriber's endpoint returned.
type DeliveryResult struct {
	StatusCode int
}

func (r DeliveryResult) isSuccess() bool { return r.StatusCode >= 200 && r.StatusCode < 300 }

// Subscriber stands in for an HTTP POST to the subscriber's URL.
type Subscriber interface {
	Deliver(payload, eventID string, timestamp int64, signature string) DeliveryResult
}

// sign returns the hex HMAC-SHA256 of "timestamp.payload" -- Stripe's own scheme,
// binding the timestamp INTO the signed message so a captured request can't be replayed
// later with a forged fresh timestamp (the signature would no longer match).
func sign(secret, payload string, timestamp int64) string {
	signedPayload := fmt.Sprintf("%d.%s", timestamp, payload)
	mac := hmac.New(sha256.New, []byte(secret))
	mac.Write([]byte(signedPayload))
	return hex.EncodeToString(mac.Sum(nil))
}

// DeliveryWorker retries with full-jitter exponential backoff, then gives up.
type DeliveryWorker struct {
	Secret      string
	MaxAttempts int
	BaseDelay   time.Duration
	MaxDelay    time.Duration
	UseBackoff  bool // false reproduces the retry-storm bug (Movement 5 / break-it b)
	rng         *rand.Rand
}

func NewDeliveryWorker(secret string, maxAttempts int, base, max time.Duration, useBackoff bool) *DeliveryWorker {
	return &DeliveryWorker{
		Secret: secret, MaxAttempts: maxAttempts, BaseDelay: base, MaxDelay: max,
		UseBackoff: useBackoff, rng: rand.New(rand.NewSource(42)), // fixed seed: reproducible demo numbers
	}
}

// backoffDelay implements AWS's "full jitter": random(0, min(maxDelay, base*2^(attempt-1))).
func (w *DeliveryWorker) backoffDelay(attempt int) time.Duration {
	if !w.UseBackoff {
		return 0
	}
	capped := w.BaseDelay * time.Duration(int64(1)<<uint(attempt-1))
	if capped > w.MaxDelay {
		capped = w.MaxDelay
	}
	return time.Duration(w.rng.Int63n(int64(capped) + 1))
}

// Deliver returns true if the event was accepted (2xx) within MaxAttempts.
func (w *DeliveryWorker) Deliver(e Event, sub Subscriber) bool {
	for attempt := 1; attempt <= w.MaxAttempts; attempt++ {
		ts := time.Now().Unix()
		sig := sign(w.Secret, e.Payload, ts)
		r := sub.Deliver(e.Payload, e.ID, ts, sig)
		if r.isSuccess() {
			fmt.Printf("  attempt %d -> %d OK, delivered\n", attempt, r.StatusCode)
			return true
		}
		fmt.Printf("  attempt %d -> %d FAIL\n", attempt, r.StatusCode)
		if attempt < w.MaxAttempts {
			delay := w.backoffDelay(attempt)
			fmt.Printf("    backing off %v before retry %d\n", delay, attempt+1)
			time.Sleep(delay)
		}
	}
	fmt.Printf("  GIVE UP after %d attempts -> endpoint AUTO-DISABLED\n", w.MaxAttempts)
	return false
}

// flappingSubscriber fails the first failCount calls, then succeeds.
type flappingSubscriber struct {
	calls     int64
	failCount int64
}

func (f *flappingSubscriber) Deliver(payload, eventID string, ts int64, sig string) DeliveryResult {
	n := atomic.AddInt64(&f.calls, 1)
	if n <= f.failCount {
		return DeliveryResult{503}
	}
	return DeliveryResult{200}
}

// alwaysFailingSubscriber never recovers -- exercises give-up/auto-disable.
type alwaysFailingSubscriber struct{}

func (alwaysFailingSubscriber) Deliver(payload, eventID string, ts int64, sig string) DeliveryResult {
	return DeliveryResult{500}
}

// instantFailSubscriber answers instantly with a failure -- used to measure request rate.
type instantFailSubscriber struct{ calls int64 }

func (s *instantFailSubscriber) Deliver(payload, eventID string, ts int64, sig string) DeliveryResult {
	atomic.AddInt64(&s.calls, 1)
	return DeliveryResult{500}
}

func main() {
	secret := "whsec_demo_shared_secret"

	fmt.Println("=== 1. Normal delivery, first attempt succeeds ===")
	normal := NewDeliveryWorker(secret, 6, 100*time.Millisecond, 2*time.Second, true)
	normal.Deliver(Event{"evt_1", `{"type":"payment.succeeded"}`}, &flappingSubscriber{failCount: 0})

	fmt.Println()
	fmt.Println("=== 2. Flapping endpoint: fails 3x, then recovers under backoff+jitter ===")
	backoffWorker := NewDeliveryWorker(secret, 6, 100*time.Millisecond, 2*time.Second, true)
	t0 := time.Now()
	backoffWorker.Deliver(Event{"evt_2", `{"type":"invoice.paid"}`}, &flappingSubscriber{failCount: 3})
	fmt.Printf("  wall time=%v (attempts spaced out by design)\n", time.Since(t0))

	fmt.Println()
	fmt.Println("=== 3. Endpoint never recovers: give up after maxAttempts, auto-disable ===")
	giveUpWorker := NewDeliveryWorker(secret, 5, 50*time.Millisecond, 800*time.Millisecond, true)
	ok := giveUpWorker.Deliver(Event{"evt_3", `{"type":"charge.failed"}`}, alwaysFailingSubscriber{})
	fmt.Println("  delivered =", ok, "-> subscriber endpoint would be marked DISABLED in the DB")

	fmt.Println()
	fmt.Println("=== 4. BREAK IT (b): immediate retry (no backoff) vs backoff -- the retry storm ===")
	window := 300 * time.Millisecond

	stormTarget := &instantFailSubscriber{}
	stormStart := time.Now()
	for time.Since(stormStart) < window {
		ts := time.Now().Unix()
		sig := sign(secret, "{}", ts)
		stormTarget.Deliver("{}", "evt_storm", ts, sig)
	}
	rate := float64(atomic.LoadInt64(&stormTarget.calls)) * float64(time.Second) / float64(window)
	fmt.Printf("  NO-BACKOFF worker hammered the endpoint %d times in %v (~%.0f req/s)\n",
		stormTarget.calls, window, rate)

	politeTarget := &instantFailSubscriber{}
	politeWorker := NewDeliveryWorker(secret, 1<<30, 50*time.Millisecond, 500*time.Millisecond, true)
	politeStart := time.Now()
	attempt := 0
	for time.Since(politeStart) < window {
		ts := time.Now().Unix()
		sig := sign(secret, "{}", ts)
		politeTarget.Deliver("{}", "evt_storm", ts, sig)
		attempt++
		time.Sleep(politeWorker.backoffDelay(attempt))
	}
	politeRate := float64(atomic.LoadInt64(&politeTarget.calls)) * float64(time.Second) / float64(window)
	fmt.Printf("  BACKOFF worker made only %d request(s) to the same endpoint in the same %v window (~%.0f req/s)\n",
		politeTarget.calls, window, politeRate)
	fmt.Println("  Same failing endpoint, same wall-clock window -- backoff is the only difference,",
		"and it is the difference between a request storm and a handful of polite retries.")
}

Builds clean with go build ./... and go vet ./...; exercised end to end before publishing.

Reference implementation — Go, receiver (verify + dedup + fast-ack-then-async)

The async path uses a real goroutine plus a sync.WaitGroup so the demo can prove the ack returns before the background work finishes — exercised under go test -race with zero races reported.

package main

import (
	"crypto/hmac"
	"crypto/sha256"
	"encoding/hex"
	"fmt"
	"sync"
	"time"
)

// verifySign mirrors the sender's sign() -- same construction, "timestamp.payload".
func verifySign(secret, payload string, timestamp int64) string {
	signedPayload := fmt.Sprintf("%d.%s", timestamp, payload)
	mac := hmac.New(sha256.New, []byte(secret))
	mac.Write([]byte(signedPayload))
	return hex.EncodeToString(mac.Sum(nil))
}

// verify rejects a tampered payload OR a signature whose timestamp has aged out of the
// replay window -- an attacker who captured a valid request can't replay it after tolerance.
func verify(secret, payload string, timestamp int64, signature string, now int64, toleranceSeconds int64) bool {
	expected := verifySign(secret, payload, timestamp)
	if hmac.Equal([]byte(expected), []byte(signature)) == false { // constant-time compare
		return false
	}
	skew := now - timestamp
	if skew < 0 {
		skew = -skew
	}
	return skew <= toleranceSeconds
}

// NaiveReceiver has NO dedup -- every delivery attempt re-runs the side effect.
type NaiveReceiver struct {
	mu                  sync.Mutex
	walletCreditedCents int
}

func (r *NaiveReceiver) Process(eventID string, amountCents int) {
	r.mu.Lock()
	defer r.mu.Unlock()
	r.walletCreditedCents += amountCents
}

// SafeReceiver dedups by event id BEFORE the side effect runs -- idempotent under at-least-once.
type SafeReceiver struct {
	mu                  sync.Mutex
	seen                map[string]bool
	walletCreditedCents int
}

func NewSafeReceiver() *SafeReceiver {
	return &SafeReceiver{seen: make(map[string]bool)}
}

// Process returns true only if this call actually applied the side effect.
func (r *SafeReceiver) Process(eventID string, amountCents int) bool {
	r.mu.Lock()
	defer r.mu.Unlock()
	if r.seen[eventID] {
		return false
	}
	r.seen[eventID] = true
	r.walletCreditedCents += amountCents
	return true
}

// AsyncReceiver acks instantly and hands the real work to a goroutine -- fast-2xx-then-async.
type AsyncReceiver struct {
	downstream *SafeReceiver
	mu         sync.Mutex
	processed  int
	wg         sync.WaitGroup
}

func NewAsyncReceiver() *AsyncReceiver {
	return &AsyncReceiver{downstream: NewSafeReceiver()}
}

// Receive returns the ack latency; the real processing happens on a separate goroutine,
// so the sender's socket is released long before the downstream work finishes.
func (a *AsyncReceiver) Receive(eventID string, amountCents int, simulatedProcessing time.Duration) time.Duration {
	start := time.Now()
	a.wg.Add(1)
	go func() {
		defer a.wg.Done()
		time.Sleep(simulatedProcessing)
		if a.downstream.Process(eventID, amountCents) {
			a.mu.Lock()
			a.processed++
			a.mu.Unlock()
		}
	}()
	return time.Since(start) // the 200 OK goes back to the sender NOW, not after Sleep
}

func (a *AsyncReceiver) Processed() int {
	a.mu.Lock()
	defer a.mu.Unlock()
	return a.processed
}

func (a *AsyncReceiver) Wait() { a.wg.Wait() }

func main() {
	secret := "whsec_demo_shared_secret"
	payload := `{"type":"payment.succeeded","amount":5000}`
	now := time.Now().Unix()
	sig := verifySign(secret, payload, now)

	fmt.Println("=== 1. Signature + timestamp verification ===")
	fmt.Println("  fresh, correctly signed  -> accept?", verify(secret, payload, now, sig, now, 300))
	tampered := `{"type":"payment.succeeded","amount":999999}`
	fmt.Println("  tampered payload         -> accept?", verify(secret, tampered, now, sig, now, 300))
	staleTs := now - 900
	staleSig := verifySign(secret, payload, staleTs)
	fmt.Println("  correct sig but 900s old -> accept?", verify(secret, payload, staleTs, staleSig, now, 300),
		" (REPLAY rejected: outside the 300s tolerance window)")

	fmt.Println()
	fmt.Println("=== 2. BREAK IT (a): at-least-once WITHOUT dedup -- double side effect ===")
	naive := &NaiveReceiver{}
	naive.Process("evt_42", 5000)
	naive.Process("evt_42", 5000) // duplicate delivery of the SAME event id
	fmt.Printf("  wallet credited (naive, no dedup): %d cents  <- BUG: customer paid once, wallet credited twice ($%.2f)\n",
		naive.walletCreditedCents, float64(naive.walletCreditedCents)/100.0)

	fmt.Println()
	fmt.Println("=== 3. FIX: same duplicate delivery, WITH event-id dedup ===")
	safe := NewSafeReceiver()
	first := safe.Process("evt_42", 5000)
	second := safe.Process("evt_42", 5000)
	fmt.Println("  first delivery applied?  ", first)
	fmt.Println("  duplicate applied?       ", second, " (skipped -- already seen this event id)")
	fmt.Printf("  wallet credited (safe, with dedup): %d cents  <- correct: exactly one credit despite two deliveries\n",
		safe.walletCreditedCents)

	fmt.Println()
	fmt.Println("=== 4. Fast-2xx-then-async: ack latency vs. real processing latency ===")
	async := NewAsyncReceiver()
	ackLatency := async.Receive("evt_99", 1234, 200*time.Millisecond)
	fmt.Printf("  ack returned to sender in %v (well under any reasonable webhook timeout)\n", ackLatency)
	time.Sleep(50 * time.Millisecond)
	fmt.Println("  processed count immediately after ack:", async.Processed(),
		" (0 -- the 200ms of real work is still running in the background)")
	async.Wait()
	fmt.Println("  processed count after the background job finishes:", async.Processed(),
		" (1 -- decoupled from the sender's retry clock entirely)")
}

Builds clean with go build ./... and go vet ./...; also run under go run -race and go test -race with zero data races reported (the async path is the only concurrent code here, guarded by a mutex around processed and the dedup map).

5. Break it — the tests that fail

Break-it (a) — delete the dedup check, watch the side effect double. Delete exactly one line from SafeReceiver.process: the if (!seenEventIds.add(eventId)) return false; guard (Java) / the if r.seen[eventID] { return false } guard (Go). That's the NaiveReceiver in the reference code above — nothing hypothetical, it's a real class in the page's own source, run directly:

=== 2. BREAK IT (a): at-least-once WITHOUT dedup -- double side effect ===
  wallet credited (naive, no dedup): 10000 cents  <- BUG: customer paid once, wallet credited twice ($100.00)

=== 3. FIX: same duplicate delivery, WITH event-id dedup ===
  first delivery applied?   true
  duplicate applied?        false  (skipped -- already seen this event id)
  wallet credited (safe, with dedup): 5000 cents  <- correct: exactly one credit despite two deliveries

Read those numbers carefully: the exact same two calls — process("evt_42", 5000) twice — produce 10000 cents credited with no dedup and 5000 cents credited with it. Nothing about the delivery changed; the receiver's handling of a duplicate is the entire difference between a correct ledger and a customer support ticket about a double-refund. This is precisely why "at-least-once" is a contract on both sides: the sender promises "you'll get it at least once," and that promise is only safe to accept if the receiver promises back "delivering it more than once is a no-op."

Break-it (b) — delete the backoff, watch the retry storm. Set useBackoff=false (Java) / UseBackoff: false (Go) on the DeliveryWorker — every retry then fires with zero delay, exactly the "attempt 1 — retry until it succeeds, immediately" design from Movement 3. Point it and a backoff-enabled worker at the identical instantFailSubscriber, for the identical 300ms wall-clock window, straight from a real run of the page's own Java code:

=== 4. BREAK IT (b): immediate retry (no backoff) vs backoff -- the retry storm ===
  NO-BACKOFF worker hammered the endpoint 2860 times in 300ms (~9533 req/s)
  BACKOFF worker made only 4 request(s) to the same endpoint in the same 300ms window (~13 req/s)

And the Go build of the exact same scenario, same 300ms window (Go's tighter loop overhead makes the gap even starker):

=== 4. BREAK IT (b): immediate retry (no backoff) vs backoff -- the retry storm ===
  NO-BACKOFF worker hammered the endpoint 630922 times in 300ms (~2081950 req/s)
  BACKOFF worker made only 5 request(s) to the same endpoint in the same 300ms window (~17 req/s)

The exact counts vary run to run (they're wall-clock-timed, and depend on the machine's spin speed) — re-running either binary a few times during authoring produced a range of roughly 2,600–3,200 requests for Java and 600,000–640,000 for Go in the no-backoff case, against a steady 4–5 requests in the backoff case every time. The number that matters isn't the exact count, it's the three-to-five-orders-of-magnitude gap between "no backoff" and "backoff," reproduced identically in both languages: one design politely checks in a handful of times over a fraction of a second; the other turns your own retry logic into a denial-of-service attack against a subscriber that is already having a bad day.

6. Optimise — with trade-offs

DesignWhat it guaranteesWhat it costsUse when
At-most-once (fire once, never retry)Simple; no duplicate side effects, everSilently drops events on any transient failure — unacceptable for "your payment succeeded"Non-critical, best-effort notifications only (e.g. an analytics ping) where a lost event is a shrug, not an incident
At-least-once + receiver dedup (this lab)No silent drops; duplicates are expected and neutralized by the receiverReceiver MUST maintain a dedup store (event-id set/table) and MUST design side effects to be safely re-runnable behind itAnything where losing an event is worse than occasionally re-checking one you've already seen — the default for payments, order state, most real business events
"Exactly-once" (the myth)What everyone wishes existed: delivered once, processed once, no coordination neededNot achievable end-to-end across an untrusted network boundary you don't control — see Idempotency & "Exactly-Once Is a Myth". What people call "exactly-once" in practice IS at-least-once delivery plus idempotent processing — this lab's actual design, just renamedNever build for this directly; build at-least-once + dedup and call the composite "effectively exactly-once" if you must use the phrase at all
Sync-in-request deliverySimplest code — no queue, no worker, no eventual consistency to reason aboutCouples your critical path's latency (and availability) to a third party's uptime — one slow subscriber degrades every request that triggers a webhookInternal calls between services you operate, where you control both ends' SLAs — never for third-party subscriber URLs
Queue + worker delivery (this lab)The triggering request returns immediately regardless of subscriber health; retries are isolated to the workerAn extra moving part (a durable queue) and eventual — not immediate — delivery to the subscriberAny webhook to a URL you don't operate and can't guarantee the uptime of, which in practice is every real webhook system
Sign + verify (HMAC + timestamp)Receiver can prove the request came from you and isn't a stale replay, at negligible per-request costRequires safe secret distribution/rotation per subscriber, and the receiver must implement constant-time comparison correctlyThe default for any public webhook endpoint — cheap, stateless per request, no extra network hop
mTLS between sender and receiverTransport-level mutual authentication; no per-message signing logic neededBoth sides must provision, rotate, and validate certificates — heavier operational burden than a shared secret, and most merchants' infra isn't set up to terminate client-cert TLSClosed, high-trust B2B integrations where both parties already run a PKI (common inside a company's own service mesh; rare for arbitrary third-party webhook subscribers)

The real judgment call: don't reach for "exactly-once" semantics, distributed transactions, or mTLS by default — at-least-once + idempotent receiver + HMAC-with-timestamp is the boring, well-trodden design that Stripe, Razorpay, and effectively every payments platform actually ship, because it's the cheapest design that is still correct under network failure. See Idempotency Keys and How to Implement Them Safely for Payments for the same pattern applied to the payment-initiation side (the client's POST /charge), which is the mirror image of this lab's receiver-side dedup.

7. Defend under drilling

8. You can now defend

Re-authored/Deepened for this guide. Reference code compiled and executed (javac/java on JDK 8; go build, go vet, go run -race, go test -race) before publishing; both break-it scenarios reproduce deterministically in direction (dedup: 2× the credit; no-backoff: three-to-five orders of magnitude more requests) on every run. Mechanics modeled on Stripe's and Razorpay's own published webhook documentation (retry schedule, HMAC-SHA256 signing with timestamp, event-id-based dedup, auto-disable after sustained failure). See also: Polling vs LongPolling vs WebSockets vs Webhooks, The Architecture of the Retry Pattern, Idempotency & "Exactly-Once Is a Myth", and Idempotency Keys and How to Implement Them Safely for Payments.

🤖 Don't fully get this? Learn it with Claude

Stuck on Design a Webhook Delivery System? Open Claude, copy a block below, and it'll teach you this exact concept — visually and interactively.

🎨 Explain it visually

Build the mental picture, not memorization.

I just read a lesson on **Design a Webhook Delivery System** (Hands-On Builds) and want to truly understand it. Explain Design a Webhook Delivery System from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🤔 Walk me through it (interactive)

Socratic — adapts to where you're stuck.

Teach me **Design a Webhook Delivery System** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧪 Quiz me & fix my gaps

Active recall exposes what you missed.

Quiz me on **Design a Webhook Delivery System** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧠 Make it stick

Intuition + hook + flashcards for long-term memory.

Help me remember **Design a Webhook Delivery System** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

📝 My notes

← Design a Double-Entry LedgerDesign a Subscription Billing Syst →