medium Designing Notification Service

Let's design a multi-tenant notification service: one shared platform that many separate applications (tenants) — a social network, an e-commerce site, a productivity tool — use to deliver alerts to their users across email, SMS, mobile push, and in-app channels. Think Amazon SNS or Firebase Cloud Messaging: a single system that fans events out to the right channels, respects each user's preferences, and keeps every tenant's data isolated.

The design goal is a pipeline that accepts an event once and delivers it at least once per selected channel. "At least once" is a deliberate choice — it means duplicates are possible, so we suppress them at two distinct layers: (1) an idempotency key that collapses duplicate client submissions within a bounded time window, and (2) a delivery-time guard that stops a reprocessed queue message from calling a provider twice. These are different guarantees for different failure modes; the rest of the page keeps them separate on purpose.

Key terms

• Tenant — an independent client application, with its own users and events, isolated from other tenants.
• Notification — a message about an event ("Alice liked your photo", "Order #1234 shipped") destined for one or more users.
• Channel — the delivery medium: in-app, push, email, or SMS.
• User preferences — per-user opt-outs, quiet hours, and per-category channel choices.
• Idempotency key — a client-supplied unique id per logical send, used to deduplicate retried submissions.

Requirements

Functional

• Multi-channel delivery — email, SMS, push, in-app; one event may fan out to several channels, each delivered independently with no cross-channel prioritization.
• Real-time and batched — event-driven sends delivered in ~1–2s, plus scheduled sends and bulk campaigns (daily digests, marketing blasts) to millions of users.
• Scheduling — notifications can be queued for a future send time.
• Retry on failure — transient failures retried with exponential backoff, giving at-least-once delivery.
• User preferences and opt-out — honor per-user, per-category, per-channel choices and quiet hours.
• Templates — tenants define reusable templates with placeholders; the service fills in event data per channel.
• In-app inbox — persist notifications so users can fetch a recent list; push in real time to active sessions.
• Multi-tenant isolation — a tenant can only address its own users.

Non-functional

• Scale and throughput — web-scale volume with peaks of tens of thousands of events/sec (a celebrity post fanning out to millions).
• Low latency — sub-second for in-app/push; email/SMS may tolerate up to ~1 minute.
• High availability and durability — 99.99% target; once accepted, an event is never silently lost.
• Per-user ordering — a user's feed stays chronological per channel; global ordering is not required.
• Extensibility — adding a new channel or feature (scheduling, digests) must not require a redesign.

Capacity estimation

One consistent set of numbers, used everywhere below.

The storage figure now follows directly from the stated inputs: 25M active users × 100 retained notifications = 2.5 billion records, and at ~500 bytes each that is ~1.25 TB of primary data — roughly 3.75 TB once replicated three ways. (Retaining history for all 100M registered users instead of just actives would push this to ~10B records / ~5 TB; we scope storage to actives.) Preferences are one small row per registered user — 100M × a few hundred bytes ≈ tens of GB — and fit comfortably in a partitioned SQL or NoSQL store. Payload bandwidth at ~1 KB/notification is ~125 GB/day.

Takeaway: aggressive horizontal scaling, per-user data partitioning, and fully asynchronous processing.

API design

A small REST surface. The idempotency_key is the client's guard against submitting the same logical notification twice (e.g., its own retry after a network timeout).

1. Send notification

POST /v1/notifications

{
  "user_id": "u_12345",
  "template_id": "order_shipped_v1",
  "channels": ["push", "email"],
  "idempotency_key": "uuid-generated-by-client",
  "data": { "order_id": "999", "name": "Alice" }
}

Response 202 Accepted: { "notification_id": "notif_abc123" }. A retried submission with the same idempotency_key returns the same notification_id (see step 2). Error 429 on quota exceeded.

2. Get status

GET /v1/notifications/{notification_id} → { "status": "delivered", "channel": "sms", "updated_at": "..." }

3. Update preferences

PUT /v1/users/{user_id}/preferences → body { "categories": { "marketing": false, "shipping": true } }

High-level design and data flow

The system is an event-driven pipeline that decouples generating a notification from delivering it via a durable queue (Kafka). Each stage scales horizontally. Here is a single request's journey — the two dedup layers appear at steps 2 and 8.

1. Request ingestion. The producer calls POST /v1/notifications. The API Gateway authenticates the tenant, applies per-tenant rate limits, and forwards to a Notification Service instance.
2. Idempotency check (Layer 1 — client-submission dedup). The service mints a candidate notification_id and runs an atomic set-if-absent that stores the id as the value:

   SET  {tenant_id}:{idempotency_key}  {notification_id}  NX  EX 86400

   If the SET NX succeeds, this is a first-seen request — the service proceeds with that notification_id. If it fails (key already exists), this is a duplicate client submission — the service does a GET {tenant_id}:{idempotency_key} to read the previously assigned id and returns 202 with that same notification_id, doing no further work. A bare SETNX flag (a boolean sentinel) could not return the prior id; storing the id as the value is exactly what makes the dedup path able to return it. Limitation: this guard only holds within the TTL (24h here). A client retry after the key expires would be treated as new and mint a second notification — so the TTL must be set at least as long as the client's maximum retry horizon, and this bounded window is an accepted limitation, not global exactly-once. Second limitation — the crash window: winning the SET NX records the promise, not the work. If the service crashes after the SET NX but before the Kafka enqueue in step 5, every client retry now hits the dedup path and gets 202 with an orphaned notification_id — the event is accepted-and-lost, exactly the record-before-handle failure Layer 2 avoids. Closing it: the service writes a status row (state=accepted) as it enqueues, and the dedup path verifies instead of absorbing — on a duplicate hit it checks the status store, and if the returned id has no row, the first attempt died mid-flight, so it re-runs preference → render → enqueue under the same notification_id (safe, because a double enqueue is closed by Layer 2). A background reconciliation sweep that re-enqueues idempotency keys with no matching status row gives the same repair without client help. The rule is the same one Layer 2 lives by: a dedup key is only honest once the work behind it is durable.
3. Preference & routing. The service reads the user's preferences (cached in Redis) to drop opted-out channels/categories, apply quiet hours, and enforce per-user caps (e.g., ≤ N marketing/day).
4. Message preparation. For each surviving channel it renders the template into a channel-specific payload (email subject+HTML, short SMS text, push title/body).
5. Enqueue. One message per channel is produced to that channel's Kafka topic, partitioned by user_id so a user's messages on a channel stay ordered. Each message carries notification_id, channel, and an attempt count.
6. Scheduled path. If a future send_time is set, the request is stored by the Scheduler instead; when due, the Scheduler enqueues it into the same topics (rejoining at step 5).
7. Consumption. Stateless channel workers consume their topic's partitions; parallelism is bounded by partition count.
8. Worker delivery (Layer 2 — provider-call dedup). Kafka is at-least-once: if a worker sends to the provider and then crashes before committing its offset, the message is redelivered and — with no guard — the provider is called twice, producing a duplicate SMS/email. Layer 1 does not cover this: the idempotency key deduplicates client submissions, not repeated provider calls for the same already-accepted notification. So the worker runs a delivery-time guard keyed by (notification_id, channel) — and the order of guard versus call is the whole design. The tempting order — claim the marker first (SET sent:... NX), then call the provider — is wrong: if the worker crashes after winning the NX but before the provider accepts, or the call fails transiently, the marker now exists with nothing sent behind it. The redelivered copy sees the marker, skips, commits its offset — and the notification is permanently lost; every backoff retry from step 9 dies the same way, because the guard already says "sent". Record-before-send converts the duplicate problem into a silent-loss problem, violating the "never silently lost" NFR. The correct order is send, then record:

   GET  sent:{notification_id}:{channel}        # marker present ⇒ already delivered, skip
   → call the provider (passing a provider-side idempotency key where supported)
   SET  sent:{notification_id}:{channel}  1     # record ONLY after the provider accepts

   A redelivered copy now re-attempts the send whenever no success was ever recorded — exactly what at-least-once demands, and what makes step 9's retry-with-backoff actually able to deliver. The residual window (crash after provider-accept but before the marker write) can produce a rare duplicate; the provider-side idempotency key closes it where the provider supports one, and where it doesn't we knowingly accept the duplicate — for notifications, a rare double-send is strictly better than a silent loss. (Equivalently and more durably: attempt the send, then a unique-constraint insert of (notification_id, channel, "sent") in the status store records success.) This closes the double-send gap that Layer 1 alone leaves open — without ever trading it for loss.
9. Confirmation & logging. The worker records the outcome (sent/delivered/failed + provider response) in the status store; transient failures are re-queued with backoff, and terminal failures land in a DLQ.
10. Read path. The client fetches the in-app inbox from a separate query service / read replica, served from cache where possible, isolating reads from the write path.

No channel blocks another — each is delivered independently, so an email-provider lag never holds up push or SMS.

Data storage and schema

A hybrid store: SQL for structured, slowly-changing, relational data needing ACID (tenants, users, preferences, templates, scheduled jobs, audit logs); NoSQL (Cassandra/MongoDB) for the high-volume notification history and status, denormalized for join-free reads and horizontal scale.

SQL (selected)

• tenants(tenant_id PK, name, contact_email, created_at, status)
• users(user_id PK, tenant_id FK, name, email, phone, status)
• user_preferences(user_id PK/FK, email_notifications, sms_notifications, push_notifications, language, updated_at)
• notification_templates(template_id PK, tenant_id FK, name, subject, body, channel, updated_at)
• scheduled_notifications(schedule_id PK, tenant_id FK, user_id FK, template_id FK, scheduled_time, status)
• audit_logs(log_id PK, tenant_id FK, user_id FK, action, details, performed_by, timestamp)

NoSQL (high-scale)

• notifications(notification_id PK, tenant_id, user_id, template_id, message, channel, sent_at, metadata) — the ~2.5B-record inbox history; tenant_id/user_id denormalized for fetch-by-user.
• notification_status(status_id PK, notification_id, user_id, status, updated_at, details) — append-only status transitions (sent → delivered → read/failed), avoiding contention on the main record. A unique index on (notification_id, channel, "sent") can double as the durable backing for the Layer-2 delivery guard.

A dead-letter store — failed_notifications(notification_id, user_id, channel, error_code, last_attempt_time) or a queue DLQ — holds messages that exhausted retries for later inspection or reprocessing.

Detailed component design

Scheduler & batch

Scheduled sends live in scheduled_notifications; a fault-tolerant scheduler (leader-elected or partitioned by time bucket) scans for scheduled_time <= now() and enqueues due items, pre-staging large batches slightly early for on-time delivery. Big campaigns (millions of users) use a bulk-load path (upload a user list to object storage, expand into per-user messages) rather than millions of API calls, with worker throughput scaled up and provider rate limits respected.

Workers & queue semantics

Workers are stateless consumers, one thread per partition; add consumers up to the partition count to scale. Kafka commits offsets after processing, giving at-least-once. Partitioning by user_id preserves per-user ordering within a channel ("Order Placed" before "Order Shipped").

Two-layer deduplication (summary)

• Layer 1 — client submission (ingest, step 2): SET {tenant}:{idempotency_key} {notification_id} NX EX ttl. Collapses duplicate client requests and returns the original id. Bounded by TTL — set it ≥ the client's retry horizon; retries past the window are treated as new (accepted limitation) — and paired with a status-store check so a crash between the SET NX and the enqueue is re-driven rather than absorbed (step 2).
• Layer 2 — provider call (delivery, step 8): check sent:{notification_id}:{channel}, call the provider, and write the marker (or unique-constraint status row) only after the provider accepts, with a provider-side idempotency key closing the residual crash-after-accept window. Suppresses at-least-once Kafka reprocessing (post-send, pre-commit crash) double-sends — and because success is recorded after the fact, a failed or crashed attempt stays retryable instead of becoming a silent loss, which is exactly what marking before the call would cause.

Neither layer alone is sufficient: Layer 1 does nothing about queue reprocessing of an already-accepted notification, and Layer 2 does nothing about a client submitting the same event twice. Together they bound duplicates end-to-end without paying for full exactly-once.

Retry & failure handling

Workers classify errors: transient (timeouts, throttling) are retried with exponential backoff via a retry topic / delayed re-enqueue up to a max attempt count; permanent (invalid address, unregistered device token) are logged and dropped without retry. After max attempts a message goes to the DLQ. Critical channels can fail over to a secondary provider — and because the sent-marker is recorded per (notification_id, channel) only on success, the fallback fires exactly while no attempt has succeeded and stops the moment one has, so failover does not double-send.

Other

Per-user/channel rate limits to prevent spam; templating + localization; API auth (API keys/OAuth) so only authorized backends can send; content validation to prevent injection; graceful degradation (shed or defer load, buffer logs) under overload.

Scalability & performance

Designing for ~125M notifications/day with ~15–45k/sec peaks:

• Horizontal scaling — stateless Notification Service instances behind a load balancer; channel workers scaled per-channel by queue backlog; Kafka as a multi-broker cluster. No single funnel point.
• Queue parallelism — partition counts sized to target throughput (e.g., if a worker does ~1k msg/sec, ~50 partitions of email absorb the ~45k/sec peak with headroom). Batch produces and offset commits to cut overhead.
• Caching — Redis for preferences and templates keeps the critical-path lookup at microseconds; cache the user's recent inbox for the read path.
• Partitioning/sharding — shard the ~2.5B-record notification history by user_id; time-partition logs by day so hot writes hit one partition and old partitions age out to cheap storage and are dropped wholesale.
• External integrations — spread provider calls across multiple connections/accounts/vendors and honor per-provider rate limits; request quota increases ahead of known peaks.

Sources

Adapted and corrected from the original "Designing a Notification Service" problem in this guide, with the capacity math reconciled to the stated inputs (25M DAU × 100 retained ⇒ ~2.5B records / ~1.25 TB), the idempotency mechanism specified as SET key value NX EX (value = notification_id) rather than a bare SETNX flag, and delivery-time (provider-call) deduplication added at the worker to close the at-least-once double-send gap. Supporting references: Apache Kafka documentation on consumer offset commits and at-least-once delivery semantics; Redis SET command documentation (NX/EX options); Amazon SNS and Firebase Cloud Messaging service documentation for multi-tenant push/fan-out patterns.

Pitfalls

• Dedup only at enqueue. At-least-once queues redeliver; without a worker-side sent-marker (or unique-constraint status row), users get double SMS/push on retry.
• Recording "sent" before the provider call. Claim-then-send means a crash or transient failure inside the send leaves a marker that blocks every retry — the notification is silently lost and the backoff pipeline is dead code. Record success only after the provider accepts; the residual duplicate window is closed by the provider-side idempotency key, and a rare double beats a silent loss.
• One queue for all priorities. Marketing blasts head-of-line-block password resets. Separate topics/queues by urgency and give critical paths dedicated workers + provider quota.
• Ignoring provider rate limits. A fan-out storm that exceeds FCM/SNS/Twilio quotas fails open or gets banned. Token-bucket per provider, multi-vendor failover, and pre-raised quotas for known peaks.
• Preferences checked too late. Filtering unsubscribed channels after rendering wastes work and can leak content into logs; gate on preferences before template render.

When to use this shape — and when not

Use a scheduler → durable queue → multi-channel workers design when you need reliable, preference-aware, multi-provider delivery with retries and audit. Do not build this for pure in-app live updates (prefer WebSocket/SSE pub-sub) or for one-off admin emails (a single transactional mailer is enough).

Takeaways

• Two-layer dedup (enqueue idempotency + post-success sent-marker) bounds at-least-once doubles; never record "sent" before the provider accepts, or retries go dead and failures become silent losses.
• Partition hot paths by user; age logs by day; isolate provider quotas.
• Critical vs bulk traffic must not share a single contended queue.

🪜 Drill ladder: Designing Notification Service (medium)

1. Recompute the peak. 25M DAU × 5 notifications/day = 125M/day ≈ 1.5k/sec average. At the 30× burst multiplier, what peak must the email pipeline absorb, and how many partitions does that need at ~1k msg/sec per worker? (≈45k/sec → ~45–50 partitions.)
2. Trace the redelivery. A worker sends an SMS, then crashes before committing its Kafka offset. Walk what happens next, name which dedup layer stops the user getting two texts — and explain why Layer 1 (the client idempotency key) cannot.
3. Break the buggy order. Suppose the worker recorded sent:{id}:{channel} before calling the provider, and the provider times out. Show why step 9's retry-with-backoff can now never deliver this notification, then state the correct order and the residual window it leaves.
4. TTL arithmetic. The idempotency key has a 24h TTL; a client's retry policy backs off up to 36h after an outage. What happens on the 36h retry, and what is the rule relating TTL to retry horizon?
5. Head-of-line math. A 5M-notification marketing blast lands on the same topic as password resets flowing at 200/sec. At a 45k/sec drain rate, how long does a reset submitted right after the blast wait? (5M ÷ 45k/s ≈ 110s — versus a sub-second SLO; hence separate priority topics with dedicated workers.)