medium Designing Notification Service

Let's design a multi-tenant notification service: one shared platform that many separate applications (tenants) — a social network, an e-commerce site, a productivity tool — use to deliver alerts to their users across email, SMS, mobile push, and in-app channels. Think Amazon SNS or Firebase Cloud Messaging: a single system that fans events out to the right channels, respects each user's preferences, and keeps every tenant's data isolated.

The design goal is a pipeline that accepts an event once and delivers it at least once per selected channel. "At least once" is a deliberate choice — it means duplicates are possible, so we suppress them at two distinct layers: (1) an idempotency key that collapses duplicate client submissions within a bounded time window, and (2) a delivery-time guard that stops a reprocessed queue message from calling a provider twice. These are different guarantees for different failure modes; the rest of the page keeps them separate on purpose.

Key terms

• Tenant — an independent client application, with its own users and events, isolated from other tenants.
• Notification — a message about an event ("Alice liked your photo", "Order #1234 shipped") destined for one or more users.
• Channel — the delivery medium: in-app, push, email, or SMS.
• User preferences — per-user opt-outs, quiet hours, and per-category channel choices.
• Idempotency key — a client-supplied unique id per logical send, used to deduplicate retried submissions.

Requirements

Functional

• Multi-channel delivery — email, SMS, push, in-app; one event may fan out to several channels, each delivered independently with no cross-channel prioritization.
• Real-time and batched — event-driven sends delivered in ~1–2s, plus scheduled sends and bulk campaigns (daily digests, marketing blasts) to millions of users.
• Scheduling — notifications can be queued for a future send time.
• Retry on failure — transient failures retried with exponential backoff, giving at-least-once delivery.
• User preferences and opt-out — honor per-user, per-category, per-channel choices and quiet hours.
• Templates — tenants define reusable templates with placeholders; the service fills in event data per channel.
• In-app inbox — persist notifications so users can fetch a recent list; push in real time to active sessions.
• Multi-tenant isolation — a tenant can only address its own users.

Non-functional

• Scale and throughput — web-scale volume with peaks of tens of thousands of events/sec (a celebrity post fanning out to millions).
• Low latency — sub-second for in-app/push; email/SMS may tolerate up to ~1 minute.
• High availability and durability — 99.99% target; once accepted, an event is never silently lost.
• Per-user ordering — a user's feed stays chronological per channel; global ordering is not required.
• Extensibility — adding a new channel or feature (scheduling, digests) must not require a redesign.

Capacity estimation

One consistent set of numbers, used everywhere below.

The storage figure now follows directly from the stated inputs: 25M active users × 100 retained notifications = 2.5 billion records, and at ~500 bytes each that is ~1.25 TB of primary data — roughly 3.75 TB once replicated three ways. (Retaining history for all 100M registered users instead of just actives would push this to ~10B records / ~5 TB; we scope storage to actives.) Preferences are one small row per registered user — 100M × a few hundred bytes ≈ tens of GB — and fit comfortably in a partitioned SQL or NoSQL store. Payload bandwidth at ~1 KB/notification is ~125 GB/day.

Takeaway: aggressive horizontal scaling, per-user data partitioning, and fully asynchronous processing.

API design

A small REST surface. The idempotency_key is the client's guard against submitting the same logical notification twice (e.g., its own retry after a network timeout).

1. Send notification

POST /v1/notifications

{
  "user_id": "u_12345",
  "template_id": "order_shipped_v1",
  "channels": ["push", "email"],
  "idempotency_key": "uuid-generated-by-client",
  "data": { "order_id": "999", "name": "Alice" }
}

Response 202 Accepted: { "notification_id": "notif_abc123" }. A retried submission with the same idempotency_key returns the same notification_id (see step 2). Error 429 on quota exceeded.

2. Get status

GET /v1/notifications/{notification_id} → { "status": "delivered", "channel": "sms", "updated_at": "..." }

3. Update preferences

PUT /v1/users/{user_id}/preferences → body { "categories": { "marketing": false, "shipping": true } }

High-level design and data flow

The system is an event-driven pipeline that decouples generating a notification from delivering it via a durable queue (Kafka). Each stage scales horizontally. Here is a single request's journey — the two dedup layers appear at steps 2 and 8.

1. Request ingestion. The producer calls POST /v1/notifications. The API Gateway authenticates the tenant, applies per-tenant rate limits, and forwards to a Notification Service instance.
2. Idempotency check (Layer 1 — client-submission dedup). The service mints a candidate notification_id and runs an atomic set-if-absent that stores the id as the value:

   SET  {tenant_id}:{idempotency_key}  {notification_id}  NX  EX 86400

   If the SET NX succeeds, this is a first-seen request — the service proceeds with that notification_id. If it fails (key already exists), this is a duplicate client submission — the service does a GET {tenant_id}:{idempotency_key} to read the previously assigned id and returns 202 with that same notification_id, doing no further work. A bare SETNX flag (a boolean sentinel) could not return the prior id; storing the id as the value is exactly what makes the dedup path able to return it. Limitation: this guard only holds within the TTL (24h here). A client retry after the key expires would be treated as new and mint a second notification — so the TTL must be set at least as long as the client's maximum retry horizon, and this bounded window is an accepted limitation, not global exactly-once.
3. Preference & routing. The service reads the user's preferences (cached in Redis) to drop opted-out channels/categories, apply quiet hours, and enforce per-user caps (e.g., ≤ N marketing/day).
4. Message preparation. For each surviving channel it renders the template into a channel-specific payload (email subject+HTML, short SMS text, push title/body).
5. Enqueue. One message per channel is produced to that channel's Kafka topic, partitioned by user_id so a user's messages on a channel stay ordered. Each message carries notification_id, channel, and an attempt count.
6. Scheduled path. If a future send_time is set, the request is stored by the Scheduler instead; when due, the Scheduler enqueues it into the same topics (rejoining at step 5).
7. Consumption. Stateless channel workers consume their topic's partitions; parallelism is bounded by partition count.
8. Worker delivery (Layer 2 — provider-call dedup). Kafka is at-least-once: if a worker sends to the provider and then crashes before committing its offset, the message is redelivered and — with no guard — the provider is called twice, producing a duplicate SMS/email. Layer 1 does not cover this: the idempotency key deduplicates client submissions, not repeated provider calls for the same already-accepted notification. So before every provider call the worker runs a delivery-time guard keyed by (notification_id, channel):

   SET  sent:{notification_id}:{channel}  1  NX      # only the winner proceeds

   Only the worker that wins the NX calls the provider; a redelivered copy sees the marker and skips straight to committing its offset. (Equivalently, a unique-constraint insert or conditional write in the status store gives the same guarantee durably.) Where the provider supports it, the worker also passes a provider-side idempotency key as defense in depth. This closes the double-send gap that Layer 1 alone leaves open.
9. Confirmation & logging. The worker records the outcome (sent/delivered/failed + provider response) in the status store; transient failures are re-queued with backoff, and terminal failures land in a DLQ.
10. Read path. The client fetches the in-app inbox from a separate query service / read replica, served from cache where possible, isolating reads from the write path.

No channel blocks another — each is delivered independently, so an email-provider lag never holds up push or SMS.

Data storage and schema

A hybrid store: SQL for structured, slowly-changing, relational data needing ACID (tenants, users, preferences, templates, scheduled jobs, audit logs); NoSQL (Cassandra/MongoDB) for the high-volume notification history and status, denormalized for join-free reads and horizontal scale.

SQL (selected)

• tenants(tenant_id PK, name, contact_email, created_at, status)
• users(user_id PK, tenant_id FK, name, email, phone, status)
• user_preferences(user_id PK/FK, email_notifications, sms_notifications, push_notifications, language, updated_at)
• notification_templates(template_id PK, tenant_id FK, name, subject, body, channel, updated_at)
• scheduled_notifications(schedule_id PK, tenant_id FK, user_id FK, template_id FK, scheduled_time, status)
• audit_logs(log_id PK, tenant_id FK, user_id FK, action, details, performed_by, timestamp)

NoSQL (high-scale)

• notifications(notification_id PK, tenant_id, user_id, template_id, message, channel, sent_at, metadata) — the ~2.5B-record inbox history; tenant_id/user_id denormalized for fetch-by-user.
• notification_status(status_id PK, notification_id, user_id, status, updated_at, details) — append-only status transitions (sent → delivered → read/failed), avoiding contention on the main record. A unique index on (notification_id, channel, "sent") can double as the durable backing for the Layer-2 delivery guard.

A dead-letter store — failed_notifications(notification_id, user_id, channel, error_code, last_attempt_time) or a queue DLQ — holds messages that exhausted retries for later inspection or reprocessing.

Detailed component design

Scheduler & batch

Scheduled sends live in scheduled_notifications; a fault-tolerant scheduler (leader-elected or partitioned by time bucket) scans for scheduled_time <= now() and enqueues due items, pre-staging large batches slightly early for on-time delivery. Big campaigns (millions of users) use a bulk-load path (upload a user list to object storage, expand into per-user messages) rather than millions of API calls, with worker throughput scaled up and provider rate limits respected.

Workers & queue semantics

Workers are stateless consumers, one thread per partition; add consumers up to the partition count to scale. Kafka commits offsets after processing, giving at-least-once. Partitioning by user_id preserves per-user ordering within a channel ("Order Placed" before "Order Shipped").

Two-layer deduplication (summary)

• Layer 1 — client submission (ingest, step 2): SET {tenant}:{idempotency_key} {notification_id} NX EX ttl. Collapses duplicate client requests and returns the original id. Bounded by TTL — set it ≥ the client's retry horizon; retries past the window are treated as new (accepted limitation).
• Layer 2 — provider call (delivery, step 8): SET sent:{notification_id}:{channel} 1 NX (or a unique-constraint write), plus a provider-side idempotency key when available. Prevents at-least-once Kafka reprocessing (post-send, pre-commit crash) from double-sending to the provider.

Neither layer alone is sufficient: Layer 1 does nothing about queue reprocessing of an already-accepted notification, and Layer 2 does nothing about a client submitting the same event twice. Together they bound duplicates end-to-end without paying for full exactly-once.

Retry & failure handling

Workers classify errors: transient (timeouts, throttling) are retried with exponential backoff via a retry topic / delayed re-enqueue up to a max attempt count; permanent (invalid address, unregistered device token) are logged and dropped without retry. After max attempts a message goes to the DLQ. Critical channels can fail over to a secondary provider — and because the Layer-2 guard is keyed by (notification_id, channel), only one provider attempt can ultimately win, so fallback does not cause a duplicate.

Other

Per-user/channel rate limits to prevent spam; templating + localization; API auth (API keys/OAuth) so only authorized backends can send; content validation to prevent injection; graceful degradation (shed or defer load, buffer logs) under overload.

Scalability & performance

Designing for ~125M notifications/day with ~20–50k/sec peaks:

• Horizontal scaling — stateless Notification Service instances behind a load balancer; channel workers scaled per-channel by queue backlog; Kafka as a multi-broker cluster. No single funnel point.
• Queue parallelism — partition counts sized to target throughput (e.g., if a worker does ~1k msg/sec, ~50 partitions of email handle 50k/sec). Batch produces and offset commits to cut overhead.
• Caching — Redis for preferences and templates keeps the critical-path lookup at microseconds; cache the user's recent inbox for the read path.
• Partitioning/sharding — shard the ~2.5B-record notification history by user_id; time-partition logs by day so hot writes hit one partition and old partitions age out to cheap storage and are dropped wholesale.
• External integrations — spread provider calls across multiple connections/accounts/vendors and honor per-provider rate limits; request quota increases ahead of known peaks.

Sources

Adapted and corrected from the original "Designing a Notification Service" problem in this guide, with the capacity math reconciled to the stated inputs (25M DAU × 100 retained ⇒ ~2.5B records / ~1.25 TB), the idempotency mechanism specified as SET key value NX EX (value = notification_id) rather than a bare SETNX flag, and delivery-time (provider-call) deduplication added at the worker to close the at-least-once double-send gap. Supporting references: Apache Kafka documentation on consumer offset commits and at-least-once delivery semantics; Redis SET command documentation (NX/EX options); Amazon SNS and Firebase Cloud Messaging service documentation for multi-tenant push/fan-out patterns.