Notification System Reliability — Templates, Settings, Retry & Tracking, Traced

The delivery contract, stated exactly

A notification system's requirement is easy to state and easy to get subtly wrong: notifications can be delayed or re-ordered, but never lost.

Two mechanisms satisfy that: the system persists notification data in a database (a notification log) and implements a retry mechanism. Persistence means a crash cannot erase an unsent notification; retry means a third-party failure is a delay rather than a loss.

Will a recipient receive it exactly once?

No. This is worth saying plainly, because it is the question interviewers use to find out whether you understand distributed delivery. Notifications are delivered exactly once most of the time, but the distributed nature of the system will produce duplicates. You cannot have exactly-once delivery, so you reduce duplication rather than eliminate it.

The dedupe logic is simple: when an event arrives, check whether its event ID has been seen. If yes, discard; otherwise send. Combined with careful handling of each failure case, duplicates become rare rather than impossible.

The asymmetry that justifies the whole design: a user seeing one notification twice is an annoyance; a user never seeing it is a broken product. So when the two risks conflict, always choose the duplicate. That is why the notification is persisted before the send attempt — a crash after sending loses only a duplicate, while a crash before persisting loses the notification entirely. Ordering those two writes correctly is the single most important implementation detail on this page.

Notification templates

A large system sends millions of notifications a day, and many follow a similar format. A template is a preformatted notification whose parameters, styling and tracking links you customize, so you are not building each message from scratch. For example:

BODY:
You dreamed of it. We dared it. [ITEM NAME] is back — only until [DATE].

CTA:
Order Now. Or, Save My [ITEM NAME]

The benefits are a consistent format, a reduced margin of error, and time saved. The middle one matters most operationally: with templates, a copy or compliance fix is one change in one place rather than a search through code paths, and localization becomes a property of the template rather than a branch in the sending logic.

Notification settings: the opt-in check

Users receive too many notifications and easily feel overwhelmed, so give them fine-grained control. The setting table is deliberately small:

user_id  bigInt
channel  varchar   # push notification, email or SMS
opt_in   boolean   # opt-in to receive notification

Before any notification is sent, check that the user is opted in to that type. Note the shape: opt-in is per channel, not per user, because someone may welcome an email and refuse a push. Note also where the check sits — early in the pipeline, before templating and queueing, so you never spend work on a message you must not send. In many jurisdictions this check is also a legal requirement rather than a courtesy, which makes it a correctness concern: bypassing it on a "transactional" path is how compliance incidents happen.

Rate limiting, retry, and security

• Rate limiting — cap how many notifications a user can receive. The justification is a product one rather than a capacity one: if you send too often, recipients turn notifications off completely. Over-sending does not merely annoy, it permanently destroys the channel, which is why this limit protects revenue rather than servers.
• Retry mechanism — when a third-party service fails, the notification goes back onto the message queue to be retried. If the problem persists, alert the developers. Retry silently forever and a provider outage becomes an invisible backlog.
• Security — iOS and Android push APIs are secured with appKey and appSecret, so only authenticated or verified clients can send push notifications through your APIs. Without this, your notification endpoint is an open relay for spam sent under your app's name.

Monitoring and event tracking

The key metric to monitor is the total number of queued notifications. A large number means workers are not processing events fast enough, and the remedy is more workers. This is the health signal for the whole system, and it is a good one because it degrades visibly before users notice: the queue grows for a while before delivery is late enough to complain about.

Events tracking covers the product side — open rate, click rate and engagement — usually by integrating with an analytics service. These are what tell you whether notifications are worth sending at all, and they close the loop with rate limiting: falling open rates are the early warning that you are over-sending and about to lose the channel.

The assembled design

• Notification servers gain two critical features: authentication and rate limiting.
• A retry mechanism puts failures back on the queue for a predefined number of attempts.
• Templates make creation consistent and efficient.
• Monitoring and tracking cover system health and future improvement.

Which choice, when

Decision	Option	Choose when	Breaks when
Durability	Persist to log, then send	Always — loss is unacceptable	Adds a write to the hot path; risks duplicates on crash (the safe failure)
Durability	Send, then record	Never for real notifications	A crash between the two loses the notification silently
Duplicates	Dedupe on event ID	Always — cheap and effective	Requires a dedupe store with a bounded window; residual duplicates remain
Duplicates	Chase exactly-once	Never — not achievable	You spend effort and still cannot guarantee it
Content	Templates	Many notifications share a format	Highly bespoke one-off messages; template indirection adds friction
Volume control	Per-user rate limit + opt-in	Consumer products	Critical transactional alerts (security, payment) must bypass volume caps
Failure signal	Retry N times, then alert	Third-party provider failures	Infinite silent retry hides an outage; no retry turns a blip into loss

The sixth row hides a real design decision: not all notifications are equal. A marketing push should be rate-limited and opt-out-able; a two-factor code or a fraud alert must not be suppressed by a volume cap or a marketing opt-out. That means notifications need a class, and the opt-in and rate-limit checks must respect it. Systems that treat all notifications uniformly either spam users or fail to deliver security-critical messages — and the second failure mode is discovered by users locked out of their accounts.

Pitfalls

• Sending before persisting. Fails in the unsafe direction — the one direction the contract forbids.
• Promising exactly-once. It cannot be delivered; design the product to tolerate a rare duplicate.
• Checking opt-in late, after templating and queueing — wasted work and a real risk of sending anyway on a retry path that skips the check.
• Applying volume caps to transactional notifications, suppressing security codes and payment alerts.
• Retrying indefinitely with no alert. A provider outage becomes an invisible growing backlog.
• Retrying without backoff against a struggling provider, turning their brownout into an outage and getting your sender throttled or blocked.
• One shared queue for all channels. A slow SMS provider then blocks push and email delivery; queue per channel so failure domains stay separate.
• No queue-depth monitoring. The one metric that gives early warning, and it is easy to omit.
• Unbounded dedupe store. Bound the window, and know that duplicates arriving later are uncatchable.

Cost model — what dominates the bill

A notification system is the rare design where your own infrastructure is the cheap part: the dominant cost is what you pay third parties per message, so cost control is mostly about sending fewer, better-targeted notifications.

Rough BOTE at 10 million notifications/day. Push via APNs/FCM is effectively free, so channel mix decides everything. Email at roughly $0.10 per thousand is ~$100/day per million — still minor. SMS is the outlier: at even $0.005–0.01 per message, 1 million SMS/day is $5,000–10,000/day, i.e. $150,000–300,000/month. One channel choice dominates the entire budget by three orders of magnitude.

Your own side is modest: the notification log at ~1 KB per notification is 10 GB/day, ~300 GB/month — tens of dollars, and it compresses and tiers well. Workers and queues are sized by peak send rate, and since sends are bursty (a campaign, a breaking-news alert), the fleet is provisioned for burst and idle otherwise — the same provisioned-for-peak shape as a flash sale.

Dominant line items: SMS fees, by a wide margin; then email volume at very large scale; then worker capacity for bursts; then the log.

Levers, in order of leverage: (1) channel downgrade — prefer push (free) over SMS (expensive) whenever the user has the app installed, and fall back to SMS only when necessary; (2) respect opt-in and rate limits, which cut spend and improve engagement simultaneously — the rare lever with no trade-off; (3) batch or digest low-urgency notifications so ten events become one message; (4) use event tracking to stop sending notification types with near-zero open rates, which is the cheapest optimization available and the one least often done.

Operability: the fingerprints of a broken notification system

Queued-notification depth growing steadily is the headline signal — workers cannot keep up and delivery is slipping before anyone complains. Distinguish it from depth growing on one channel only, which is a third-party provider degrading rather than a capacity problem; if all channels share a queue you cannot tell these apart, which is a good argument for per-channel queues.

Duplicate notifications arriving in bursts means the dedupe store is missing or its window expired — often after a retry storm replayed events older than the window. Retry counts climbing with a flat failure alert means the retry ceiling is too high or the alert threshold too lax, so an outage is being absorbed silently into latency.

The most damaging fingerprint is a falling open rate with rising send volume. It is not an infrastructure failure at all, and it is the leading indicator of users disabling notifications permanently — the one failure this system cannot recover from, because a user who turns notifications off is not reachable by fixing a queue. Watch also for opt-out rate spiking after a release (a new notification type is unwelcome, or the opt-in check regressed) and transactional notifications appearing in rate-limit rejection logs, which means security-critical messages are being suppressed by a marketing cap.

Signals worth having: queue depth per channel, worker throughput versus arrival rate, per-provider error rate and latency, dedupe hit rate with eviction age, retry-attempt distribution, open/click rate per notification type, opt-out rate, and rate-limit rejections broken down by notification class.

---

Re-authored for this guide from the Alex Xu Vol. 1 notification-system chapter (reliability and the never-lost contract, dedupe by event ID, notification templates, the opt-in setting table, rate limiting, retry, appKey/appSecret security, queued-notification monitoring and events tracking); reliability-pipeline diagram hand-authored as SVG. Complements the existing "Designing Notification Service" page with the deep-dive reliability and operational layers.

I just read a lesson on **Notification System Reliability — Templates, Settings, Retry & Tracking, Traced** (System Design) and want to truly understand it. Explain Notification System Reliability — Templates, Settings, Retry & Tracking, Traced from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Notification System Reliability — Templates, Settings, Retry & Tracking, Traced** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Notification System Reliability — Templates, Settings, Retry & Tracking, Traced** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Notification System Reliability — Templates, Settings, Retry & Tracking, Traced** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.