Knowledge Guide
Home › Hands-On Builds › LLD Katas

Design a Notification Service

Design a Notification Service

"Design a Notification Service" is one of the most durable LLD rounds at payment companies — Razorpay lists it among its own recurring interview problems — because a payment platform's entire trust story rides on this one mechanism: telling a merchant or customer "your payment went through" reliably, over whichever channel (push, SMS, email) is actually reachable right now, without stalling the checkout request that triggered it. Anyone can draw the class diagram — a Notification, a Channel per medium — in ninety seconds. What separates a pass from a strong pass is what happens when the push provider times out at 2am, or the caller's retry fires twice, or a slow SMS gateway ends up sitting in the critical path of a payment confirmation. This kata builds all three, in Java and Go, from an empty file: a Strategy-based channel abstraction with an automatic fallback chain, an Observer/pub-sub layer deciding who gets notified, and the reliability crux most candidates never reach: async queue-backed delivery, retry with backoff, and a dedup guard that stops a redelivered job from sending the same message twice. (For the distributed, HLD-scale version of this same problem — sharded queues, a preferences store, per-provider rate limits — see the Designing Notification Service system-design page; this kata is the in-process mechanism that page's "notification dispatcher" box is actually hiding.)

The Trap

The obvious first cut: when a payment succeeds, notify the customer inline, in the same request that just charged their card.

chargeCard(order);
smsGateway.send(customer.phone, "Payment of ₹499 received"); // <-- right here, in the response path
return ResponseEntity.ok(receipt);

This works in the demo. It breaks in production for two independent reasons, and both reappear as this kata's break-it tests.

First — it blocks the caller on a system it doesn't control. SMS/push gateways are third-party HTTP calls: P50 latency is fine (150–250ms), but the P99 tail for a telco SMS aggregator under load is genuinely bad — 5–8 seconds is normal, not exceptional. Inline, that means roughly 1 in 100 payment-confirmation requests now takes 5–8 seconds to return, even though the card was already charged and the response has nothing left to wait on except a notification. The customer stares at a spinner for a business action that already succeeded.

Second, and worse — that same stall causes a real double-send. The client (a mobile app or a browser tab) doesn't know an SMS call is quietly hanging server-side; it just sees the request taking too long and times out at, say, 5 seconds, then retries the entire "confirm payment" call because nothing came back. But the server's SMS call might complete a moment later, at the 6-second mark — just after the client gave up. The retry re-enters the same handler, which calls smsGateway.send(...) a second time. The customer's phone buzzes twice: "Payment of ₹499 received" … "Payment of ₹499 received." Two real SMSes, one real payment, and a merchant paying for both.

Now swap in a push-only version with no fallback path at all: the customer's phone is simply unreachable (flight mode, dead battery, an expired push token), the push call throws, and there is no code anywhere that tries a second channel. That confirmation is just gone. Silently. Nothing retries it, nothing dead-letters it for someone to notice — the exception is caught, logged, and the request returns 200 OK because the payment succeeded. The customer was never told, and nobody finds out until they call support asking where their money went.

Scope it like a senior

Before writing a line of dispatch code, pin the contract down:

Answer: 3 channels behind a common Strategy interface with a per-notification fallback order, an Observer/pub-sub fan-out layer for "who gets notified," async queue-backed dispatch so the caller never blocks on a provider, bounded retry with exponential backoff, and dedup by a stable notification id — at-least-once delivery made safe, not fake exactly-once.

Reason to the design

Simplest thing that could work is the trap itself: call one channel, synchronously, inline in the business-logic handler. Why it fails, in two distinct ways shown above: it blocks the caller on a system outside its control (the tail-latency stall), and it has no recourse when that one channel fails (the silent loss). Fixing these one at a time is how the real design falls out.

Step 1 — make the channel a Strategy. Wrap each medium behind one Channel interface (EmailChannel, SmsChannel, PushChannel) so calling code depends on an abstraction, not three concrete APIs. This alone doesn't fix anything yet — whoever calls Channel.send() still picks exactly one channel and still has nothing to fall back on if that one throws. The silent-loss bug survives untouched.

Step 2 — chain the channels. Give each notification a preference order (["push","sms","email"]) and try them in sequence until one confirms delivery. Now a down push provider no longer loses the notification — it falls through to SMS. This kills the silent-loss bug for good, as long as at least one channel in the chain is actually up. But it's still invoked inline, in the caller's request thread — and now the worst case is worse: if push AND sms are both degraded, the caller is blocked for the sum of both timeouts before ever reaching email. Chaining channels made reliability better and caller latency worse, in the same line of code.

Step 3 — decouple delivery from the request. Don't run the fallback chain in the caller's thread at all: enqueue a Notification job and return immediately; a background worker pool drains the queue and runs the fallback chain there. Caller latency becomes O(1) regardless of how slow or how many channels get tried. This is a genuine trade, not a strict win — the caller no longer knows, at the moment it returns, whether the notification actually went out (see the sync-vs-async row in the trade-off table below; some notifications, like a login OTP whose next screen needs the code already delivered, actually want the synchronous guarantee back).

Step 4 — the queue's own reliability creates a new bug. A queue is only worth having because it's at-least-once: if a worker crashes mid-delivery, or its acknowledgment to the broker is lost, the job gets redelivered rather than silently dropped. That's exactly the property that made async delivery reliable in step 3. But it's also exactly what causes a double-send: if the redelivered job had, in fact, already been successfully sent before the crash/ack-loss, redelivering it means sending the same real message twice — the retry-storm bug from the Trap, now happening for a completely different (and completely legitimate) reason. The fix isn't to fight the queue's at-least-once guarantee — it's to make delivery itself idempotent: track notification IDs already confirmed delivered, and short-circuit any duplicate before it ever reaches a real channel. Now redelivery (or a client-side retry) is always safe.

A separate axis: who gets notified. Naive: OrderService directly calls NotificationService.notify(...) for every interested party it happens to know about. Every new consumer of "an order shipped" — analytics, a partner webhook, an audit log — now requires editing OrderService. The Observer/pub-sub fix: OrderService just publish("ORDER_SHIPPED", data)s to an event bus and never references a subscriber; a NotificationFanOut observer subscribes independently and turns one event into N per-user Notification jobs, each honoring that user's own channel-preference order. New subscribers cost zero changes to the publisher — the whole point of the pattern.

Build it — milestones

Attempt-first: the contract is Channel.send(Notification) (the Strategy), FallbackDispatcher.deliver(Notification) → the channel name that delivered or null/"", Dispatcher.notifyAsync(Notification) (enqueues, never blocks), and EventBus.publish(Event) (fans out via Observer). Try each milestone before reading the reference implementation below.

Reference implementation — Java

One file, no external dependencies beyond the JDK. NaiveNotifier is kept ONLY so the break-it tests can reproduce the two classic failure modes on real code — it is never how production code should be structured. Dispatcher's dedupEnabled constructor flag exists purely so the break-it test can construct an intentionally-unsafe instance on the exact same class, the same pattern used for concurrency bugs elsewhere in this guide (an "unsafe twin," never called from real code). Save as Notif.java.

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;

/** Channel is the delivery STRATEGY -- one implementation per medium (email/SMS/push).
 *  A real implementation wraps a provider SDK (SES, Twilio, FCM); here each is a
 *  controllable fake so the kata's tests can flip a channel healthy/unhealthy on demand. */
interface Channel {
    String name();
    /** Attempt delivery. Returns true on CONFIRMED success, throws on failure. */
    boolean send(Notification n) throws ChannelException;
}

/** Thrown by a Channel when delivery fails (provider down, device offline, bad token, ...). */
final class ChannelException extends Exception {
    ChannelException(String msg) { super(msg); }
}

/** A renderable message: a pattern with {{placeholders}}, resolved against variables. */
final class Template {
    private final String pattern;
    Template(String pattern) { this.pattern = pattern; }
    String render(Map<String, String> vars) {
        String out = pattern;
        for (Map.Entry<String, String> e : vars.entrySet()) {
            out = out.replace("{{" + e.getKey() + "}}", e.getValue());
        }
        return out;
    }
}

/** One notification job: an idempotency key, the rendered text, and the user's channel
 *  preference order (tried in sequence, most-preferred first, until one confirms). */
final class Notification {
    final String id;                  // idempotency / dedup key
    final String userId;
    final String text;
    final List<String> channelOrder;  // e.g. ["push","sms","email"]
    Notification(String id, String userId, String text, List<String> channelOrder) {
        this.id = id;
        this.userId = userId;
        this.text = text;
        this.channelOrder = channelOrder;
    }
}

/** A controllable fake channel: toggle healthy/unhealthy, optionally sleep to simulate a
 *  slow provider, and count real send attempts so tests can assert on them directly. */
final class FakeChannel implements Channel {
    private final String channelName;
    private final AtomicBoolean healthy = new AtomicBoolean(true);
    private final AtomicInteger attempts = new AtomicInteger(0);
    private final long simulatedLatencyMs;

    FakeChannel(String channelName) { this(channelName, 0L); }
    FakeChannel(String channelName, long simulatedLatencyMs) {
        this.channelName = channelName;
        this.simulatedLatencyMs = simulatedLatencyMs;
    }

    void setHealthy(boolean h) { healthy.set(h); }
    int attempts() { return attempts.get(); }

    public String name() { return channelName; }

    public boolean send(Notification n) throws ChannelException {
        attempts.incrementAndGet();
        if (simulatedLatencyMs > 0) {
            try { Thread.sleep(simulatedLatencyMs); } catch (InterruptedException ie) { Thread.currentThread().interrupt(); }
        }
        if (!healthy.get()) throw new ChannelException(channelName + " provider unavailable");
        return true;
    }
}

/** THE NAIVE V1 -- what most people ship first. Synchronous, single channel, no fallback.
 *  Kept here ONLY so the break-it tests can reproduce the two classic failure modes on
 *  real code; never model production notification delivery this way. */
final class NaiveNotifier {
    private final Channel channel;
    NaiveNotifier(Channel channel) { this.channel = channel; }
    /** Blocks the CALLING thread until the channel call returns or throws. No fallback:
     *  if this channel fails, the notification is gone -- nothing else will try it. */
    boolean notify(Notification n) throws ChannelException {
        return channel.send(n);
    }
}

/** Tries each channel in Notification.channelOrder in turn until one confirms delivery.
 *  This is the reliability crux: a failed channel falls through to the next, instead of
 *  the notification silently dying with that one provider. */
final class FallbackDispatcher {
    private final Map<String, Channel> channels = new HashMap<String, Channel>();

    FallbackDispatcher(List<Channel> chs) {
        for (Channel c : chs) channels.put(c.name(), c);
    }

    /** Returns the name of the channel that delivered, or null if EVERY channel in the
     *  notification's preference order failed. */
    String deliver(Notification n) {
        for (String chName : n.channelOrder) {
            Channel c = channels.get(chName);
            if (c == null) continue;
            try {
                if (c.send(n)) return chName;
            } catch (ChannelException e) {
                // fall through to the next channel in the chain
            }
        }
        return null;
    }
}

/** Async, queue-backed dispatch with retry-with-backoff and a dedup guard.
 *  notifyAsync() returns as soon as the job is enqueued -- NOT once it is delivered --
 *  so a slow or down provider never blocks the caller. A background worker pool drains
 *  the queue, runs the fallback chain, and on total failure requeues with exponential
 *  backoff up to maxRetries before giving up to a dead-letter list.
 *
 *  dedupEnabled exists ONLY to let the break-it test demonstrate the double-send bug on
 *  this exact class by constructing it with dedupEnabled=false; production code must
 *  always use dedupEnabled=true. */
final class Dispatcher {
    private final FallbackDispatcher fallback;
    private final boolean dedupEnabled;
    private final int maxRetries;
    private final Set<String> delivered = ConcurrentHashMap.newKeySet();     // dedup store
    private final Map<String, Integer> attemptCounts = new ConcurrentHashMap<String, Integer>();
    private final List<Notification> deadLetters = Collections.synchronizedList(new ArrayList<Notification>());
    private final BlockingQueue<Notification> queue = new LinkedBlockingQueue<Notification>();
    private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(2);
    private final ExecutorService workers;
    private volatile boolean running = true;

    Dispatcher(FallbackDispatcher fallback, int workerCount, int maxRetries, boolean dedupEnabled) {
        this.fallback = fallback;
        this.maxRetries = maxRetries;
        this.dedupEnabled = dedupEnabled;
        this.workers = Executors.newFixedThreadPool(workerCount);
        for (int i = 0; i < workerCount; i++) {
            workers.submit(new Runnable() { public void run() { workerLoop(); } });
        }
    }

    /** Non-blocking: enqueues and returns immediately. This is the whole point -- the
     *  caller (e.g. a checkout request handler) never waits on a notification provider. */
    void notifyAsync(Notification n) {
        queue.offer(n);
    }

    private void workerLoop() {
        while (running) {
            Notification n;
            try {
                n = queue.poll(200, TimeUnit.MILLISECONDS);
            } catch (InterruptedException ie) {
                Thread.currentThread().interrupt();
                return;
            }
            if (n == null) continue;
            process(n);
        }
    }

    /** Package-visible so a test can re-submit the SAME Notification directly, simulating
     *  a message queue's at-least-once redelivery (e.g. the worker crashed, or its ack was
     *  lost, after the notification had already been successfully delivered). */
    void process(final Notification n) {
        if (dedupEnabled && delivered.contains(n.id)) {
            return;   // DEDUP GUARD -- this line is the entire fix for the double-send bug below
        }
        String deliveredVia = fallback.deliver(n);
        if (deliveredVia != null) {
            delivered.add(n.id);
            return;
        }
        int attempts;
        synchronized (attemptCounts) {
            Integer prev = attemptCounts.get(n.id);
            attempts = (prev == null ? 0 : prev) + 1;
            attemptCounts.put(n.id, attempts);
        }
        if (attempts > maxRetries) {
            deadLetters.add(n);
            return;
        }
        long backoff = backoffMillis(attempts);
        scheduler.schedule(new Runnable() { public void run() { queue.offer(n); } }, backoff, TimeUnit.MILLISECONDS);
    }

    /** Exponential backoff: 50ms, 100ms, 200ms, 400ms, ... */
    static long backoffMillis(int attempt) {
        return 50L << (attempt - 1);
    }

    List<Notification> deadLetters() { return deadLetters; }
    Set<String> delivered() { return delivered; }

    void shutdown() {
        running = false;
        workers.shutdownNow();
        scheduler.shutdownNow();
    }
}

/** A domain event some other part of the system publishes -- e.g. OrderService fires
 *  "ORDER_SHIPPED", PaymentService fires "PAYMENT_RECEIVED". The publisher has no idea
 *  who (if anyone) is listening. */
final class Event {
    final String type;
    final Map<String, String> data;
    Event(String type, Map<String, String> data) {
        this.type = type;
        this.data = data;
    }
}

interface EventObserver {
    void onEvent(Event e);
}

/** The pub-sub bus. Publishers call publish(); they never reference a subscriber
 *  directly. This is what lets NotificationFanOut (or analytics, or an audit log) be
 *  added or removed later without OrderService/PaymentService changing a line. */
final class EventBus {
    private final Map<String, List<EventObserver>> subscribers = new ConcurrentHashMap<String, List<EventObserver>>();

    void subscribe(String eventType, EventObserver obs) {
        List<EventObserver> list = subscribers.get(eventType);
        if (list == null) {
            list = new CopyOnWriteArrayList<EventObserver>();
            subscribers.put(eventType, list);
        }
        list.add(obs);
    }

    void publish(Event e) {
        List<EventObserver> list = subscribers.get(e.type);
        if (list == null) return;
        for (EventObserver obs : list) obs.onEvent(e);
    }
}

/** Decides WHO gets notified for an event and fans out one Notification per interested
 *  user, honoring each user's own channel preference order. Turns "an order shipped"
 *  into N independent per-user delivery jobs. */
final class NotificationFanOut implements EventObserver {
    private final Map<String, List<String>> subscriptions;      // eventType -> interested userIds
    private final Map<String, List<String>> userChannelPrefs;   // userId -> channel order
    private final Template template;
    private final Dispatcher dispatcher;
    private final AtomicInteger idGen = new AtomicInteger();

    NotificationFanOut(Map<String, List<String>> subscriptions,
                        Map<String, List<String>> userChannelPrefs,
                        Template template,
                        Dispatcher dispatcher) {
        this.subscriptions = subscriptions;
        this.userChannelPrefs = userChannelPrefs;
        this.template = template;
        this.dispatcher = dispatcher;
    }

    public void onEvent(Event e) {
        List<String> interested = subscriptions.get(e.type);
        if (interested == null) return;
        for (String userId : interested) {
            List<String> order = userChannelPrefs.get(userId);
            if (order == null) order = Arrays.asList("email");
            String text = template.render(e.data);
            String id = e.type + ":" + userId + ":" + idGen.incrementAndGet();
            dispatcher.notifyAsync(new Notification(id, userId, text, order));
        }
    }
}

Reference implementation — Go

Same shape, idiomatic Go: goroutines + channels instead of a thread pool + BlockingQueue, a mutex instead of ConcurrentHashMap. Save as notif.go, package notif (run go mod init notif once in the directory).

// Package notif is the reference implementation for the "Design a
// Notification Service" LLD kata: a Channel strategy with a fallback chain,
// an Observer/pub-sub fan-out for "who gets notified", and a queue-backed
// async dispatcher with retry-with-backoff and a dedup guard.
package notif

import (
	"fmt"
	"strings"
	"sync"
	"sync/atomic"
	"time"
)

// Channel is the delivery STRATEGY -- one implementation per medium
// (email/SMS/push). A real implementation wraps a provider SDK (SES,
// Twilio, FCM); FakeChannel below is a controllable fake for the kata.
type Channel interface {
	Name() string
	// Send attempts delivery. Returns (true, nil) on CONFIRMED success,
	// or (false, err) on failure.
	Send(n Notification) (bool, error)
}

// Template is a renderable message: a pattern with {{placeholders}}.
type Template struct{ Pattern string }

func (t Template) Render(vars map[string]string) string {
	out := t.Pattern
	for k, v := range vars {
		out = strings.ReplaceAll(out, "{{"+k+"}}", v)
	}
	return out
}

// Notification is one delivery job: an idempotency key, the rendered text,
// and the user's channel preference order (tried in sequence until one
// confirms).
type Notification struct {
	ID           string   // idempotency / dedup key
	UserID       string
	Text         string
	ChannelOrder []string // e.g. []string{"push","sms","email"}
}

// FakeChannel is a controllable fake: toggle healthy/unhealthy, optionally
// sleep to simulate a slow provider, and count real send attempts.
type FakeChannel struct {
	channelName       string
	healthy           int32 // atomic bool: 1 = healthy
	attempts          int32 // atomic counter
	simulatedLatency  time.Duration
}

func NewFakeChannel(name string) *FakeChannel { return NewSlowFakeChannel(name, 0) }

func NewSlowFakeChannel(name string, latency time.Duration) *FakeChannel {
	return &FakeChannel{channelName: name, healthy: 1, simulatedLatency: latency}
}

func (c *FakeChannel) SetHealthy(h bool) {
	if h {
		atomic.StoreInt32(&c.healthy, 1)
	} else {
		atomic.StoreInt32(&c.healthy, 0)
	}
}
func (c *FakeChannel) Attempts() int { return int(atomic.LoadInt32(&c.attempts)) }
func (c *FakeChannel) Name() string  { return c.channelName }

func (c *FakeChannel) Send(n Notification) (bool, error) {
	atomic.AddInt32(&c.attempts, 1)
	if c.simulatedLatency > 0 {
		time.Sleep(c.simulatedLatency)
	}
	if atomic.LoadInt32(&c.healthy) == 0 {
		return false, fmt.Errorf("%s provider unavailable", c.channelName)
	}
	return true, nil
}

// FlakyChannel fails its first N attempts, then succeeds forever after --
// simulates a transient provider blip, used to prove retry-with-backoff.
type FlakyChannel struct {
	channelName string
	failFirstN  int
	attempts    int32
}

func NewFlakyChannel(name string, failFirstN int) *FlakyChannel {
	return &FlakyChannel{channelName: name, failFirstN: failFirstN}
}
func (c *FlakyChannel) Attempts() int { return int(atomic.LoadInt32(&c.attempts)) }
func (c *FlakyChannel) Name() string  { return c.channelName }
func (c *FlakyChannel) Send(n Notification) (bool, error) {
	attempt := atomic.AddInt32(&c.attempts, 1)
	if int(attempt) <= c.failFirstN {
		return false, fmt.Errorf("%s transient failure #%d", c.channelName, attempt)
	}
	return true, nil
}

// NaiveNotifier is THE NAIVE V1 -- what most people ship first. Synchronous,
// single channel, no fallback. Kept here ONLY so the break-it tests can
// reproduce the two classic failure modes on real code; never model
// production notification delivery this way.
type NaiveNotifier struct{ channel Channel }

func NewNaiveNotifier(c Channel) *NaiveNotifier { return &NaiveNotifier{channel: c} }

// Notify blocks the CALLING goroutine until the channel call returns. No
// fallback: if this channel fails, the notification is gone.
func (nn *NaiveNotifier) Notify(n Notification) (bool, error) { return nn.channel.Send(n) }

// FallbackDispatcher tries each channel in Notification.ChannelOrder in turn
// until one confirms delivery -- the reliability crux: a failed channel
// falls through to the next instead of the notification silently dying.
type FallbackDispatcher struct {
	channels map[string]Channel
}

func NewFallbackDispatcher(chs []Channel) *FallbackDispatcher {
	m := make(map[string]Channel, len(chs))
	for _, c := range chs {
		m[c.Name()] = c
	}
	return &FallbackDispatcher{channels: m}
}

// Deliver returns the name of the channel that delivered, or "" if EVERY
// channel in the notification's preference order failed.
func (fb *FallbackDispatcher) Deliver(n Notification) string {
	for _, name := range n.ChannelOrder {
		c, ok := fb.channels[name]
		if !ok {
			continue
		}
		if ok, _ := c.Send(n); ok {
			return name
		}
		// fall through to the next channel in the chain
	}
	return ""
}

// Dispatcher is async, queue-backed dispatch with retry-with-backoff and a
// dedup guard. NotifyAsync returns as soon as the job is enqueued -- NOT
// once it is delivered -- so a slow or down provider never blocks the
// caller. A worker pool drains the queue, runs the fallback chain, and on
// total failure requeues with exponential backoff up to MaxRetries before
// giving up to a dead-letter list.
//
// DedupEnabled exists ONLY to let the break-it test demonstrate the
// double-send bug on this exact type by constructing it with
// dedupEnabled=false; production code must always use dedupEnabled=true.
type Dispatcher struct {
	fallback     *FallbackDispatcher
	dedupEnabled bool
	maxRetries   int

	mu            sync.Mutex
	delivered     map[string]bool // dedup store
	attemptCounts map[string]int
	deadLetters   []Notification

	queue chan Notification
	quit  chan struct{}
	wg    sync.WaitGroup
}

func NewDispatcher(fallback *FallbackDispatcher, workerCount, maxRetries int, dedupEnabled bool) *Dispatcher {
	d := &Dispatcher{
		fallback:      fallback,
		dedupEnabled:  dedupEnabled,
		maxRetries:    maxRetries,
		delivered:     make(map[string]bool),
		attemptCounts: make(map[string]int),
		queue:         make(chan Notification, 1024),
		quit:          make(chan struct{}),
	}
	for i := 0; i < workerCount; i++ {
		d.wg.Add(1)
		go d.workerLoop()
	}
	return d
}

// NotifyAsync enqueues and returns immediately -- the caller (e.g. a
// checkout request handler) never waits on a notification provider.
func (d *Dispatcher) NotifyAsync(n Notification) {
	select {
	case d.queue <- n:
	case <-d.quit:
	}
}

func (d *Dispatcher) workerLoop() {
	defer d.wg.Done()
	for {
		select {
		case n := <-d.queue:
			d.Process(n)
		case <-d.quit:
			return
		}
	}
}

// Process is exported so a test can re-submit the SAME Notification
// directly, simulating a message queue's at-least-once redelivery (e.g.
// the worker crashed, or its ack was lost, after the notification had
// already been successfully delivered).
func (d *Dispatcher) Process(n Notification) {
	d.mu.Lock()
	if d.dedupEnabled && d.delivered[n.ID] {
		d.mu.Unlock()
		return // DEDUP GUARD -- this check is the entire fix for the double-send bug below
	}
	d.mu.Unlock()

	deliveredVia := d.fallback.Deliver(n)
	if deliveredVia != "" {
		d.mu.Lock()
		d.delivered[n.ID] = true
		d.mu.Unlock()
		return
	}

	d.mu.Lock()
	d.attemptCounts[n.ID]++
	attempts := d.attemptCounts[n.ID]
	d.mu.Unlock()

	if attempts > d.maxRetries {
		d.mu.Lock()
		d.deadLetters = append(d.deadLetters, n)
		d.mu.Unlock()
		return
	}
	backoff := BackoffMillis(attempts)
	time.AfterFunc(time.Duration(backoff)*time.Millisecond, func() {
		d.NotifyAsync(n)
	})
}

// BackoffMillis is exponential: 50ms, 100ms, 200ms, 400ms, ...
func BackoffMillis(attempt int) int64 {
	return 50 << uint(attempt-1)
}

func (d *Dispatcher) DeadLetters() []Notification {
	d.mu.Lock()
	defer d.mu.Unlock()
	out := make([]Notification, len(d.deadLetters))
	copy(out, d.deadLetters)
	return out
}

func (d *Dispatcher) IsDelivered(id string) bool {
	d.mu.Lock()
	defer d.mu.Unlock()
	return d.delivered[id]
}

func (d *Dispatcher) Shutdown() {
	close(d.quit)
	d.wg.Wait()
}

// Event is a domain event some other part of the system publishes -- e.g.
// OrderService fires "ORDER_SHIPPED", PaymentService fires
// "PAYMENT_RECEIVED". The publisher has no idea who (if anyone) is
// listening.
type Event struct {
	Type string
	Data map[string]string
}

// Observer is anything that reacts to a published Event.
type Observer interface {
	OnEvent(e Event)
}

// EventBus is the pub-sub bus. Publishers call Publish(); they never
// reference a subscriber directly. This is what lets NotificationFanOut (or
// analytics, or an audit log) be added or removed later without
// OrderService/PaymentService changing a line.
type EventBus struct {
	mu          sync.Mutex
	subscribers map[string][]Observer
}

func NewEventBus() *EventBus { return &EventBus{subscribers: make(map[string][]Observer)} }

func (b *EventBus) Subscribe(eventType string, obs Observer) {
	b.mu.Lock()
	defer b.mu.Unlock()
	b.subscribers[eventType] = append(b.subscribers[eventType], obs)
}

func (b *EventBus) Publish(e Event) {
	b.mu.Lock()
	obs := append([]Observer(nil), b.subscribers[e.Type]...)
	b.mu.Unlock()
	for _, o := range obs {
		o.OnEvent(e)
	}
}

// NotificationFanOut decides WHO gets notified for an event and fans out
// one Notification per interested user, honoring each user's own channel
// preference order. Turns "an order shipped" into N independent per-user
// delivery jobs.
type NotificationFanOut struct {
	subscriptions   map[string][]string // eventType -> interested userIDs
	userChannelPref map[string][]string // userID -> channel order
	template        Template
	dispatcher      *Dispatcher
	idGen           int64
}

func NewNotificationFanOut(subs map[string][]string, prefs map[string][]string, t Template, d *Dispatcher) *NotificationFanOut {
	return &NotificationFanOut{subscriptions: subs, userChannelPref: prefs, template: t, dispatcher: d}
}

func (f *NotificationFanOut) OnEvent(e Event) {
	interested, ok := f.subscriptions[e.Type]
	if !ok {
		return
	}
	for _, userID := range interested {
		order, ok := f.userChannelPref[userID]
		if !ok {
			order = []string{"email"}
		}
		text := f.template.Render(e.Data)
		id := atomic.AddInt64(&f.idGen, 1)
		nid := fmt.Sprintf("%s:%s:%d", e.Type, userID, id)
		f.dispatcher.NotifyAsync(Notification{ID: nid, UserID: userID, Text: text, ChannelOrder: order})
	}
}

Break it

The reference implementation above ships two deliberately-unsafe seams on the exact same production classes, purely so this section can reproduce real bugs rather than describe them: NaiveNotifier (synchronous, single channel, no fallback — the Trap, verbatim) and Dispatcher constructed with dedupEnabled=false. Every assertion below runs against the reference code shown above; nothing here is pseudocode.

Java — save alongside Notif.java as NotifTests.java, then javac Notif.java NotifTests.java && java NotifTests:

import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;

/** A channel that fails its first N attempts, then succeeds forever after --
 *  simulates a transient provider blip, used to prove retry-with-backoff. */
final class FlakyChannel implements Channel {
    private final String channelName;
    private final int failFirstN;
    private final AtomicInteger attempts = new AtomicInteger(0);
    FlakyChannel(String channelName, int failFirstN) {
        this.channelName = channelName;
        this.failFirstN = failFirstN;
    }
    int attempts() { return attempts.get(); }
    public String name() { return channelName; }
    public boolean send(Notification n) throws ChannelException {
        int attempt = attempts.incrementAndGet();
        if (attempt <= failFirstN) throw new ChannelException(channelName + " transient failure #" + attempt);
        return true;
    }
}

public final class NotifTests {

    static int pass = 0, fail = 0;

    static void check(String name, boolean cond) {
        if (cond) { pass++; System.out.println("PASS: " + name); }
        else      { fail++; System.out.println("FAIL: " + name); }
    }

    /** Polls a condition until true or a deadline; avoids sleeping a fixed guessed duration. */
    static boolean waitUntil(long timeoutMs, Cond c) throws InterruptedException {
        long deadline = System.currentTimeMillis() + timeoutMs;
        while (System.currentTimeMillis() < deadline) {
            if (c.ok()) return true;
            Thread.sleep(10);
        }
        return c.ok();
    }
    interface Cond { boolean ok(); }

    public static void main(String[] args) throws Exception {

        // --- M1: single channel, synchronous lifecycle ---
        {
            FakeChannel email = new FakeChannel("email");
            NaiveNotifier notifier = new NaiveNotifier(email);
            Notification n = new Notification("n1", "alice",
                "Welcome!", Arrays.asList("email"));
            boolean sent = notifier.notify(n);
            check("M1: naive single-channel send succeeds", sent);
            check("M1: exactly one attempt on the channel", email.attempts() == 1);
        }

        // --- M2: multi-channel strategy, template rendering ---
        {
            Template t = new Template("Payment of {{amount}} received for order {{orderId}}");
            Map<String, String> vars = new HashMap<String, String>();
            vars.put("amount", "₹499"); vars.put("orderId", "ORD-42");
            String rendered = t.render(vars);
            check("M2: template renders both placeholders", rendered.equals("Payment of ₹499 received for order ORD-42"));

            FakeChannel push = new FakeChannel("push");
            FakeChannel sms = new FakeChannel("sms");
            FakeChannel email = new FakeChannel("email");
            FallbackDispatcher fb = new FallbackDispatcher(
                Arrays.<Channel>asList(push, sms, email));
            Notification n = new Notification("n2", "alice", rendered, Arrays.asList("push", "sms", "email"));
            String via = fb.deliver(n);
            check("M2: healthy top-preference channel (push) wins", "push".equals(via));
            check("M2: lower-preference channels never touched", sms.attempts() == 0 && email.attempts() == 0);
        }

        // --- M3: fallback chain -- push down, sms picks it up ---
        {
            FakeChannel push = new FakeChannel("push");
            FakeChannel sms = new FakeChannel("sms");
            FakeChannel email = new FakeChannel("email");
            push.setHealthy(false);
            FallbackDispatcher fb = new FallbackDispatcher(Arrays.<Channel>asList(push, sms, email));
            Notification n = new Notification("n3", "bob", "hi", Arrays.asList("push", "sms", "email"));
            String via = fb.deliver(n);
            check("M3: fallback delivers via sms after push fails", "sms".equals(via));
            check("M3: push was tried exactly once", push.attempts() == 1);
            check("M3: email never needed (chain stopped at sms)", email.attempts() == 0);
        }

        // --- M4: Observer/pub-sub fan-out -- one event, per-user channel prefs ---
        {
            FakeChannel push = new FakeChannel("push");
            FakeChannel sms = new FakeChannel("sms");
            FakeChannel email = new FakeChannel("email");
            FallbackDispatcher fb = new FallbackDispatcher(Arrays.<Channel>asList(push, sms, email));
            Dispatcher dispatcher = new Dispatcher(fb, 2, 3, true);

            Map<String, List<String>> subs = new HashMap<String, List<String>>();
            subs.put("ORDER_SHIPPED", Arrays.asList("alice", "bob"));
            Map<String, List<String>> prefs = new HashMap<String, List<String>>();
            prefs.put("alice", Arrays.asList("push"));
            prefs.put("bob", Arrays.asList("email"));
            Template t = new Template("Order {{orderId}} shipped!");
            NotificationFanOut fanOut = new NotificationFanOut(subs, prefs, t, dispatcher);

            EventBus bus = new EventBus();
            bus.subscribe("ORDER_SHIPPED", fanOut);
            Map<String, String> data = new HashMap<String, String>();
            data.put("orderId", "ORD-7");
            bus.publish(new Event("ORDER_SHIPPED", data));

            boolean bothDelivered = waitUntil(2000, new Cond() {
                public boolean ok() { return push.attempts() >= 1 && email.attempts() >= 1; }
            });
            check("M4: one event fanned out to 2 users on their OWN preferred channel", bothDelivered);
            check("M4: alice got it on push (her preference)", push.attempts() == 1);
            check("M4: bob got it on email (his preference), never push/sms", sms.attempts() == 0 && email.attempts() == 1);
            dispatcher.shutdown();
        }

        // --- M5: async dispatch never blocks the caller ---
        {
            FakeChannel slow = new FakeChannel("push", 300); // simulates a slow provider
            NaiveNotifier sync = new NaiveNotifier(slow);
            Notification n1 = new Notification("n5-sync", "carol", "hi", Arrays.asList("push"));
            long t0 = System.currentTimeMillis();
            sync.notify(n1);
            long syncElapsed = System.currentTimeMillis() - t0;
            check("M5: naive synchronous send BLOCKS the caller for the full provider latency",
                syncElapsed >= 280);

            FakeChannel slow2 = new FakeChannel("push", 300);
            FallbackDispatcher fb = new FallbackDispatcher(Arrays.<Channel>asList(slow2));
            Dispatcher dispatcher = new Dispatcher(fb, 2, 3, true);
            Notification n2 = new Notification("n5-async", "carol", "hi", Arrays.asList("push"));
            long t1 = System.currentTimeMillis();
            dispatcher.notifyAsync(n2);
            long asyncElapsed = System.currentTimeMillis() - t1;
            check("M5: async notifyAsync returns almost immediately regardless of channel latency",
                asyncElapsed < 100);
            boolean delivered = waitUntil(2000, new Cond() {
                public boolean ok() { return slow2.attempts() == 1; }
            });
            check("M5: the notification IS eventually delivered in the background", delivered);
            dispatcher.shutdown();
        }

        // --- M6: retry with backoff recovers a transient failure ---
        {
            final FlakyChannel flaky = new FlakyChannel("sms", 2); // fails twice, then succeeds
            FallbackDispatcher fb = new FallbackDispatcher(Arrays.<Channel>asList((Channel) flaky));
            Dispatcher dispatcher = new Dispatcher(fb, 2, 5, true);
            Notification n = new Notification("n6", "dave", "hi", Arrays.asList("sms"));
            dispatcher.notifyAsync(n);
            boolean recovered = waitUntil(2000, new Cond() {
                public boolean ok() { return dispatcher_delivered_contains(dispatcher, "n6"); }
            });
            check("M6: retry-with-backoff recovers after 2 transient failures", recovered);
            check("M6: exactly 3 real attempts were made (2 fail + 1 success)", flaky.attempts() == 3);
            check("backoffMillis is exponential: 50,100,200,400", Dispatcher.backoffMillis(1) == 50
                && Dispatcher.backoffMillis(2) == 100 && Dispatcher.backoffMillis(3) == 200
                && Dispatcher.backoffMillis(4) == 400);
            dispatcher.shutdown();
        }

        // ================= BREAK IT =================

        // --- Break-it #1: a channel with no fallback loses the notification, silently ---
        {
            FakeChannel push = new FakeChannel("push");
            push.setHealthy(false);
            FallbackDispatcher fbNoFallback = new FallbackDispatcher(Arrays.<Channel>asList((Channel) push));
            Notification n = new Notification("break1", "eve", "your OTP", Arrays.asList("push")); // ONLY push, no fallback configured
            String via = fbNoFallback.deliver(n);
            check("BREAK-IT #1: with no fallback configured, a down channel LOSES the notification (via == null)",
                via == null);

            // The fix: configure a fallback order. Same failure, same channel down, but now recovered.
            FakeChannel push2 = new FakeChannel("push"); push2.setHealthy(false);
            FakeChannel sms2 = new FakeChannel("sms");
            FallbackDispatcher fbFixed = new FallbackDispatcher(Arrays.<Channel>asList(push2, sms2));
            Notification n2 = new Notification("break1-fixed", "eve", "your OTP", Arrays.asList("push", "sms"));
            String via2 = fbFixed.deliver(n2);
            check("FIX: the same failure recovers once a fallback channel is configured",
                "sms".equals(via2));
        }

        // --- Break-it #2: synchronous send blocks the caller (already measured in M5, restated as the lesson) ---
        {
            FakeChannel slow = new FakeChannel("push", 250);
            NaiveNotifier sync = new NaiveNotifier(slow);
            long t0 = System.currentTimeMillis();
            sync.notify(new Notification("break2", "frank", "hi", Arrays.asList("push")));
            long elapsed = System.currentTimeMillis() - t0;
            check("BREAK-IT #2: a synchronous send stalls the caller thread for the provider's full latency ("
                + elapsed + "ms >= 230ms)", elapsed >= 230);
        }

        // --- Break-it #3: retry without dedup double-sends; the dedup guard fixes it ---
        {
            FakeChannel channel = new FakeChannel("sms");
            FallbackDispatcher fb = new FallbackDispatcher(Arrays.<Channel>asList((Channel) channel));
            // dedupEnabled = FALSE -- reproduces the bug
            Dispatcher unsafeDispatcher = new Dispatcher(fb, 1, 3, false);
            Notification n = new Notification("break3", "grace", "OTP: 4821", Arrays.asList("sms"));
            unsafeDispatcher.process(n);   // first delivery -- succeeds
            // Simulate the queue's at-least-once redelivery: the ack for this job was lost
            // (worker crashed, or the broker's visibility timeout expired) AFTER it already
            // sent successfully, so the SAME job is redelivered to a worker.
            unsafeDispatcher.process(n);
            check("BREAK-IT #3: without dedup, an at-least-once redelivery DOUBLE-SENDS the real channel ("
                + channel.attempts() + " sends for 1 logical notification)", channel.attempts() == 2);
            unsafeDispatcher.shutdown();

            FakeChannel channel2 = new FakeChannel("sms");
            FallbackDispatcher fb2 = new FallbackDispatcher(Arrays.<Channel>asList((Channel) channel2));
            // dedupEnabled = TRUE -- the fix
            Dispatcher safeDispatcher = new Dispatcher(fb2, 1, 3, true);
            Notification n2 = new Notification("break3-fixed", "grace", "OTP: 4821", Arrays.asList("sms"));
            safeDispatcher.process(n2);
            safeDispatcher.process(n2);   // identical redelivery
            check("FIX: with the dedup guard, the redelivered job is a no-op (still exactly 1 real send)",
                channel2.attempts() == 1);
            safeDispatcher.shutdown();
        }

        System.out.println();
        System.out.println(pass + " passed, " + fail + " failed");
        if (fail > 0) System.exit(1);
    }

    static boolean dispatcher_delivered_contains(Dispatcher d, String id) {
        return d.delivered().contains(id);
    }
}

Measured in this session: javac compiles clean (only harmless "auxiliary class" style warnings for multiple top-level classes in one file — zero errors), and java NotifTests passes 22/22 assertions, including all three break-it demonstrations: (1) a channel with no fallback configured loses the notification outright (via == null), and the identical failure recovers the moment a fallback channel is added; (2) the naive synchronous send measurably stalls the caller for the full simulated provider latency (250–300ms, every run); (3) replaying the exact same Notification through Dispatcher.process() twice — simulating an at-least-once queue redelivering a job whose ack was lost after it had already succeeded — produces 2 real channel sends for 1 logical notification with dedupEnabled=false, and exactly 1 with dedupEnabled=true. Same input, same replay, the single if (dedupEnabled && delivered.contains(n.id)) line is the entire difference between a duplicate SMS and a correct one.

Go — save alongside notif.go as notif_test.go:

package notif

import (
	"testing"
	"time"
)

func waitUntil(t *testing.T, timeout time.Duration, cond func() bool) bool {
	t.Helper()
	deadline := time.Now().Add(timeout)
	for time.Now().Before(deadline) {
		if cond() {
			return true
		}
		time.Sleep(10 * time.Millisecond)
	}
	return cond()
}

// --- M1: single channel, synchronous lifecycle ---
func TestM1_NaiveSingleChannel(t *testing.T) {
	email := NewFakeChannel("email")
	notifier := NewNaiveNotifier(email)
	n := Notification{ID: "n1", UserID: "alice", Text: "Welcome!", ChannelOrder: []string{"email"}}
	ok, err := notifier.Notify(n)
	if !ok || err != nil {
		t.Fatalf("expected send to succeed, got ok=%v err=%v", ok, err)
	}
	if email.Attempts() != 1 {
		t.Errorf("expected exactly 1 attempt, got %d", email.Attempts())
	}
}

// --- M2: multi-channel strategy + template rendering ---
func TestM2_ChannelStrategyAndTemplate(t *testing.T) {
	tmpl := Template{Pattern: "Payment of {{amount}} received for order {{orderId}}"}
	rendered := tmpl.Render(map[string]string{"amount": "₹499", "orderId": "ORD-42"})
	want := "Payment of ₹499 received for order ORD-42"
	if rendered != want {
		t.Errorf("template render = %q, want %q", rendered, want)
	}

	push := NewFakeChannel("push")
	sms := NewFakeChannel("sms")
	email := NewFakeChannel("email")
	fb := NewFallbackDispatcher([]Channel{push, sms, email})
	n := Notification{ID: "n2", UserID: "alice", Text: rendered, ChannelOrder: []string{"push", "sms", "email"}}
	via := fb.Deliver(n)
	if via != "push" {
		t.Errorf("expected healthy top-preference channel (push) to win, got %q", via)
	}
	if sms.Attempts() != 0 || email.Attempts() != 0 {
		t.Errorf("lower-preference channels should never be touched: sms=%d email=%d", sms.Attempts(), email.Attempts())
	}
}

// --- M3: fallback chain -- push down, sms picks it up ---
func TestM3_FallbackChain(t *testing.T) {
	push := NewFakeChannel("push")
	sms := NewFakeChannel("sms")
	email := NewFakeChannel("email")
	push.SetHealthy(false)
	fb := NewFallbackDispatcher([]Channel{push, sms, email})
	n := Notification{ID: "n3", UserID: "bob", Text: "hi", ChannelOrder: []string{"push", "sms", "email"}}
	via := fb.Deliver(n)
	if via != "sms" {
		t.Errorf("expected fallback to sms after push fails, got %q", via)
	}
	if push.Attempts() != 1 {
		t.Errorf("push should be tried exactly once, got %d", push.Attempts())
	}
	if email.Attempts() != 0 {
		t.Errorf("email should never be needed (chain stopped at sms), got %d attempts", email.Attempts())
	}
}

// --- M4: Observer/pub-sub fan-out -- one event, per-user channel prefs ---
func TestM4_ObserverFanOut(t *testing.T) {
	push := NewFakeChannel("push")
	sms := NewFakeChannel("sms")
	email := NewFakeChannel("email")
	fb := NewFallbackDispatcher([]Channel{push, sms, email})
	dispatcher := NewDispatcher(fb, 2, 3, true)
	defer dispatcher.Shutdown()

	subs := map[string][]string{"ORDER_SHIPPED": {"alice", "bob"}}
	prefs := map[string][]string{"alice": {"push"}, "bob": {"email"}}
	tmpl := Template{Pattern: "Order {{orderId}} shipped!"}
	fanOut := NewNotificationFanOut(subs, prefs, tmpl, dispatcher)

	bus := NewEventBus()
	bus.Subscribe("ORDER_SHIPPED", fanOut)
	bus.Publish(Event{Type: "ORDER_SHIPPED", Data: map[string]string{"orderId": "ORD-7"}})

	ok := waitUntil(t, 2*time.Second, func() bool {
		return push.Attempts() >= 1 && email.Attempts() >= 1
	})
	if !ok {
		t.Fatal("expected one event to fan out to 2 users on their own preferred channel")
	}
	if push.Attempts() != 1 {
		t.Errorf("alice should get it on push exactly once, got %d", push.Attempts())
	}
	if sms.Attempts() != 0 || email.Attempts() != 1 {
		t.Errorf("bob should get it on email exactly once, never push/sms: sms=%d email=%d", sms.Attempts(), email.Attempts())
	}
}

// --- M5: async dispatch never blocks the caller ---
func TestM5_AsyncDoesNotBlock(t *testing.T) {
	slow := NewSlowFakeChannel("push", 300*time.Millisecond)
	sync := NewNaiveNotifier(slow)
	n1 := Notification{ID: "n5-sync", UserID: "carol", Text: "hi", ChannelOrder: []string{"push"}}
	t0 := time.Now()
	sync.Notify(n1)
	syncElapsed := time.Since(t0)
	if syncElapsed < 280*time.Millisecond {
		t.Errorf("expected naive synchronous send to BLOCK for the full provider latency, only took %v", syncElapsed)
	}

	slow2 := NewSlowFakeChannel("push", 300*time.Millisecond)
	fb := NewFallbackDispatcher([]Channel{slow2})
	dispatcher := NewDispatcher(fb, 2, 3, true)
	defer dispatcher.Shutdown()
	n2 := Notification{ID: "n5-async", UserID: "carol", Text: "hi", ChannelOrder: []string{"push"}}
	t1 := time.Now()
	dispatcher.NotifyAsync(n2)
	asyncElapsed := time.Since(t1)
	if asyncElapsed > 100*time.Millisecond {
		t.Errorf("expected NotifyAsync to return almost immediately, took %v", asyncElapsed)
	}
	ok := waitUntil(t, 2*time.Second, func() bool { return slow2.Attempts() == 1 })
	if !ok {
		t.Error("expected the notification to be eventually delivered in the background")
	}
}

// --- M6: retry with backoff recovers a transient failure ---
func TestM6_RetryWithBackoffRecovers(t *testing.T) {
	flaky := NewFlakyChannel("sms", 2) // fails twice, then succeeds
	fb := NewFallbackDispatcher([]Channel{flaky})
	dispatcher := NewDispatcher(fb, 2, 5, true)
	defer dispatcher.Shutdown()

	n := Notification{ID: "n6", UserID: "dave", Text: "hi", ChannelOrder: []string{"sms"}}
	dispatcher.NotifyAsync(n)

	ok := waitUntil(t, 2*time.Second, func() bool { return dispatcher.IsDelivered("n6") })
	if !ok {
		t.Fatal("expected retry-with-backoff to recover after 2 transient failures")
	}
	if flaky.Attempts() != 3 {
		t.Errorf("expected exactly 3 real attempts (2 fail + 1 success), got %d", flaky.Attempts())
	}

	if BackoffMillis(1) != 50 || BackoffMillis(2) != 100 || BackoffMillis(3) != 200 || BackoffMillis(4) != 400 {
		t.Errorf("expected exponential backoff 50,100,200,400, got %d,%d,%d,%d",
			BackoffMillis(1), BackoffMillis(2), BackoffMillis(3), BackoffMillis(4))
	}
}

// ================= BREAK IT =================

// BREAK-IT #1: a channel with no fallback loses the notification, silently.
func TestBreakIt1_NoFallbackLosesNotification(t *testing.T) {
	push := NewFakeChannel("push")
	push.SetHealthy(false)
	fbNoFallback := NewFallbackDispatcher([]Channel{push})
	n := Notification{ID: "break1", UserID: "eve", Text: "your OTP", ChannelOrder: []string{"push"}} // ONLY push, no fallback configured
	via := fbNoFallback.Deliver(n)
	if via != "" {
		t.Fatalf("expected the down channel with no fallback to LOSE the notification, got via=%q", via)
	}

	// The fix: configure a fallback order. Same failure, same channel down, but now recovered.
	push2 := NewFakeChannel("push")
	push2.SetHealthy(false)
	sms2 := NewFakeChannel("sms")
	fbFixed := NewFallbackDispatcher([]Channel{push2, sms2})
	n2 := Notification{ID: "break1-fixed", UserID: "eve", Text: "your OTP", ChannelOrder: []string{"push", "sms"}}
	via2 := fbFixed.Deliver(n2)
	if via2 != "sms" {
		t.Errorf("expected the same failure to recover once a fallback channel is configured, got via=%q", via2)
	}
}

// BREAK-IT #2: a synchronous send stalls the caller for the provider's full latency.
func TestBreakIt2_SyncSendBlocksCaller(t *testing.T) {
	slow := NewSlowFakeChannel("push", 250*time.Millisecond)
	sync := NewNaiveNotifier(slow)
	t0 := time.Now()
	sync.Notify(Notification{ID: "break2", UserID: "frank", Text: "hi", ChannelOrder: []string{"push"}})
	elapsed := time.Since(t0)
	if elapsed < 230*time.Millisecond {
		t.Errorf("expected the synchronous send to stall the caller for the provider's full latency, only took %v", elapsed)
	}
}

// BREAK-IT #3: retry without dedup double-sends; the dedup guard fixes it.
func TestBreakIt3_RetryWithoutDedupDoubleSends(t *testing.T) {
	channel := NewFakeChannel("sms")
	fb := NewFallbackDispatcher([]Channel{channel})
	unsafeDispatcher := NewDispatcher(fb, 1, 3, false) // dedupEnabled = FALSE -- reproduces the bug
	defer unsafeDispatcher.Shutdown()
	n := Notification{ID: "break3", UserID: "grace", Text: "OTP: 4821", ChannelOrder: []string{"sms"}}
	unsafeDispatcher.Process(n) // first delivery -- succeeds
	// Simulate the queue's at-least-once redelivery: the ack for this job was
	// lost (worker crashed, or the broker's visibility timeout expired) AFTER
	// it already sent successfully, so the SAME job is redelivered.
	unsafeDispatcher.Process(n)
	if channel.Attempts() != 2 {
		t.Fatalf("expected an at-least-once redelivery to DOUBLE-SEND without dedup, got %d sends for 1 logical notification", channel.Attempts())
	}

	channel2 := NewFakeChannel("sms")
	fb2 := NewFallbackDispatcher([]Channel{channel2})
	safeDispatcher := NewDispatcher(fb2, 1, 3, true) // dedupEnabled = TRUE -- the fix
	defer safeDispatcher.Shutdown()
	n2 := Notification{ID: "break3-fixed", UserID: "grace", Text: "OTP: 4821", ChannelOrder: []string{"sms"}}
	safeDispatcher.Process(n2)
	safeDispatcher.Process(n2) // identical redelivery
	if channel2.Attempts() != 1 {
		t.Errorf("expected the dedup guard to make the redelivered job a no-op, got %d sends, want 1", channel2.Attempts())
	}
}

Measured in this session: go build ./... and go vet ./... are clean, and go test -race ./... passes 9/9 tests with zero data races reported, across three repeated runs. The Go break-it tests reproduce the identical three failures as the Java suite: TestBreakIt1_NoFallbackLosesNotification proves the down-channel-with-no-fallback loses the message and the fix (adding sms to the order) recovers it; TestBreakIt2_SyncSendBlocksCaller measures the naive synchronous Notify call blocking for ≥230ms against a 250ms-latency fake provider; TestBreakIt3_RetryWithoutDedupDoubleSends shows Process() replayed twice against a dedupEnabled=false dispatcher sends the real channel exactly twice, and exactly once with dedupEnabled=true. Unlike a shared-map data race, this bug is a pure logic error (a missing idempotency check) — the race detector correctly reports nothing wrong, because nothing about the mutex discipline is wrong; the design decision is wrong until the dedup guard is added.

Optimise — trade-offs

DecisionOption AOption BWhen A winsWhen B wins
Delivery timingSynchronous (inline call, caller waits for the send to finish)Async (enqueue, caller returns immediately)The very next step in the SAME request genuinely needs the send to have completed — e.g. a login-OTP flow whose next screen requires the code already be in the user's hands, or a compliance rule requiring proof-of-notify before committing the surrounding transactionVirtually everything else — any notification that is a side effect of an already-completed business action (a payment receipt, a shipping update) where the caller must not pay a third-party provider's tail latency for something it doesn't control
Fallback orderingCost/latency-first (push → sms → email, cheapest and fastest tried first)Criticality-first (e.g. sms → push, or a custom order per notification TYPE)Most notifications, most of the time — push is free and instant and most devices are reachable; only pay for SMS/email when push actually failsA notification the business cannot afford to silently miss (an OTP, a fraud alert) where push's dependency on a single provider (FCM/APNs uptime + device connectivity) is a worse risk than paying for SMS upfront — ordering should be configured per notification TYPE and per user preference, never one global constant
Delivery guaranteeAt-least-once + dedup (retry + fallback + idempotency guard)At-most-once (fire-and-forget, no retry)Almost always, for anything user-facing that matters — the cost of a customer never learning their payment succeeded usually dwarfs the cost of building the dedup guardHigh-volume, low-value notifications where a rare miss is cheap and a duplicate is actively annoying — "someone liked your photo" push spam, or telemetry-style pings, where retry infrastructure costs more than the occasional drop
Fan-out mechanismObserver/pub-sub (an EventBus; publishers never reference subscribers)Direct call (OrderService calls NotificationService.notify() explicitly)More than one thing reacts to the same event over the system's lifetime (notifications + analytics + an audit log + a partner webhook) and new subscribers should cost zero changes to the publisher — the standing-majority case in any real productExactly one caller, exactly one callee, and you're confident that will never change — the bus is pure indirection overhead for a one-producer, one-consumer kata-sized system
Dedup scopeBy notification id onlyBy (notification id + channel)The dispatcher always drives channel selection start-to-finish for one job — wherever it eventually lands, it's "the same logical notification" and should count onceYou allow re-triggering delivery on a SPECIFIC channel independently (e.g. an ops runbook lets support manually "resend via SMS" without touching the original push attempt) — then dedup must be scoped per channel, or the legitimate manual resend gets silently swallowed too

Defend under drilling

You can now defend

Re-authored/Deepened for this guide. Related theory: Strategy Pattern — UML, Code & When to Use, Observer Pattern. Distributed/HLD-scale sibling: Designing Notification Service (System Design). Reference implementations (Java + Go) compiled and tested in-session: javac clean, 22/22 assertions pass (including all three deliberately-adversarial break-it assertions, which measured a genuine 2x-vs-1x send count for the dedup bug and a genuine 230–300ms caller-blocking stall for the sync-vs-async bug, on every run); go build + go vet + go test -race clean, 9/9 tests pass across repeated runs with zero data races reported.

🤖 Don't fully get this? Learn it with Claude

Stuck on Design a Notification Service? Open Claude, copy a block below, and it'll teach you this exact concept — visually and interactively.

🎨 Explain it visually

Build the mental picture, not memorization.

I just read a lesson on **Design a Notification Service** (Hands-On Builds) and want to truly understand it. Explain Design a Notification Service from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🤔 Walk me through it (interactive)

Socratic — adapts to where you're stuck.

Teach me **Design a Notification Service** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧪 Quiz me & fix my gaps

Active recall exposes what you missed.

Quiz me on **Design a Notification Service** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧠 Make it stick

Intuition + hook + flashcards for long-term memory.

Help me remember **Design a Notification Service** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

📝 My notes

← Design a Logging FrameworkDesign an ATM →