Notification system design for SA interviews

Why interviewers love this question

Designing a notification system is one of those whiteboard prompts that looks innocent for ten seconds and then quietly tests every reliability concept a systems analyst should own — queues, retries, idempotency, fan-out, rate limits, opt-in semantics, multi-region failover, and templating. It is also a feature every product ships, which is why a hiring manager at Stripe, Notion, Linear or DoorDash can grade you against a system they actually run.

The trap is treating it as a CRUD exercise. "We POST to /notify and the server emails the user." That answer skips the part where Twilio is down, the user is asleep in Tokyo, the password-reset email must arrive in under three seconds, and the marketing blast must respect a 200-per-second per-mailbox provider cap. The interview is graded on how you separate critical from bulk, how you keep one slow channel from clogging the others, and how you avoid waking somebody up at 3 a.m. with a coupon.

This guide walks the answer the way a senior SA would deliver it in a 45-minute loop: scope, draw, defend, then dive into one corner the interviewer pokes at.

Scoping the problem

Before drawing a box, spend two minutes on functional and non-functional requirements. Skipping this is how strong candidates get rated mid because the interviewer thinks they "just started coding".

Three tiers, three SLAs, one system. Calling that out early signals you have seen this in production.

Also clarify scale: a mid-stage SaaS sends 5–20 notifications per DAU per day. At 10M DAU and 10 messages each, that is 100M events/day — roughly 1,200 events/sec average with a 10x peak. That envelope drives every sizing decision downstream.

Reference architecture

Sketch this on the board. Keep it boring.

Producer services
   |  (emit OrderShipped, PasswordReset, MarketingBlast)
   v
Notification API  ----> Audit log (append-only)
   |
   v
Routing service  --> preference store
   |                  template store
   |
   v
Per-channel queues:  email_q   sms_q   push_q   inapp_q
   |                    |        |       |        |
   v                    v        v       v        v
Worker pools  --> Providers: SES/SendGrid, Twilio, FCM/APNS, internal DB
   |
   v
Delivery receipts -> status store -> webhooks back to producers

The Notification API accepts a logical event ({user_id, event_type, payload, priority}) and returns 202 Accepted with a notification_id. The routing service expands one event into one-to-many channel messages based on user preferences and event policy, then writes to per-channel queues.

Three properties matter here. Channel isolation — if Twilio is throttling, only the SMS queue backs up, email keeps flowing. Backpressure — workers pull at a rate the provider tolerates, the queue absorbs spikes. Audit log — every accepted event is logged before fan-out so you can replay after an incident.

Channel-specific quirks

Each provider has personality. Knowing one specific fact per channel separates a senior answer from a junior one.

Email runs on SES, SendGrid, Mailgun, or Postmark. The hard part is deliverability: SPF, DKIM, DMARC, warm-up of new sending IPs, bounce-rate caps (Amazon SES suspends a sender at 5% hard-bounce rate). Templates carry both HTML and plain-text variants. Throughput is high (thousands/sec per provider tenant) but per-mailbox limits matter — Gmail will rate-limit a sender pushing > 100 to the same inbox in an hour.

SMS runs on Twilio, MessageBird, or Vonage. It is expensive (US long-code costs roughly $0.0075 per segment, international can be 10x), slow to provision (10DLC registration takes days), and carrier-rate-limited (1 message/sec per long-code by default). Reserve SMS for security and time-critical events. Carriers also drop messages with URL shorteners — surprise gotcha for marketing teams.

Push runs on FCM for Android and APNS for iOS. Tokens expire, so the system must handle Unregistered and InvalidToken responses by removing the token from the user record. Payloads are small (4KB on APNS) — long messages get truncated. Silent pushes (data-only) are throttled aggressively by iOS to save battery.

In-app is the easy one — write a row to a notifications table, surface it in UI via WebSocket or polling. The trick is read-state synchronization across devices and a sensible retention window (90 days is industry standard, longer eats storage).

User preferences and quiet hours

The preference model is a 2D matrix: (user_id, event_type) -> set of channels + frequency cap. Store it as a flat row per user-event combo, not nested JSON — flat rows let you index and bulk-update without rewriting blobs.

user 4297 / event=order_shipped   -> channels: [email, push]   freq: unlimited
user 4297 / event=marketing       -> channels: [email]         freq: 1/day
user 4297 / event=security_alert  -> channels: [email, sms]    freq: bypass caps

Quiet hours are stored as (user_id, start_local, end_local, tz). The worker checks the rule at delivery time, not enqueue time, because the user's timezone might change between events. If the message is non-critical and falls inside quiet hours, the worker reschedules to end_local of the same day, not to the next available slot — otherwise weekend nights pile up into Monday morning blasts.

Frequency caps live in a Redis counter keyed by (user_id, event_type, day) with TTL of 48h. Before sending, the worker does INCR + check < cap. This is also the place to enforce global per-user daily caps, which prevent a buggy producer from spamming one user 10,000 times.

Retries, dedup, and ordering

A delivery attempt either succeeds, fails-transient (5xx, timeout), or fails-permanent (4xx, invalid address). Retry only on transient failures.

A standard backoff schedule:

After 5 attempts or 24 hours, the message moves to a dead-letter queue with the last error code attached, and an alert fires if DLQ depth crosses 0.1% of daily volume.

Idempotency is enforced at two layers. The producer supplies an idempotency_key = hash(event_type + business_id + timestamp_bucket). The Notification API rejects duplicates within a 24-hour window using a Postgres UNIQUE(idempotency_key) constraint. On the provider side, each attempt carries a notification_id so a re-delivered event from a flaky webhook doesn't double-send.

Ordering is the trap nobody asks about until it bites. If "order placed" and "order cancelled" race through separate workers, the user gets the cancel email before the order email. The fix is per-user serialization: hash user_id to a queue partition so all events for one user go to one worker in order. Kafka makes this trivial with partition_key = user_id.

Templating and i18n

Templates live in a versioned store, not in code. A typical record:

template_id: order_shipped_v3
locale: en-US
channel: email
subject: "Your order #{{order_id}} is on its way"
body_html: "<p>Hi {{first_name}}, your package shipped via {{carrier}}...</p>"
body_text: "Hi {{first_name}}, your package shipped via {{carrier}}..."
variables: [first_name, order_id, carrier, tracking_url]

Use Liquid, Handlebars, or Mustache — all three are battle-tested, sandbox-safe by default, and have libraries in every language. Avoid raw Jinja2 in production unless you trust every template author with arbitrary code execution.

Validation runs at template upload time: missing required variables fail the upload, HTML is sanitized against an allowlist (no <script>, no javascript: URLs), and the rendered preview must fit channel limits (140 chars for SMS, 4KB for APNS).

Locale fallback chain: request pt-BR, fall back to pt, fall back to en-US, fall back to en. Never ship an empty notification because a translation is missing — log it, alert it, send the English version.

Common pitfalls

The most common mistake is using a single queue for all channels. The system runs fine until one provider has a bad afternoon — then SMS retries pile up, workers stall, and your password-reset emails are stuck behind 200,000 marketing pushes. The fix is one queue per channel, plus optionally one queue per priority tier inside each channel. Two extra queues per channel saves you a Sev1 a year.

A second trap is ignoring provider rate limits until the first 429. Twilio caps long-code throughput at 1 message/sec, SES throttles new accounts at 14/sec, FCM has a quota that sounds infinite until a campaign hits it. Configure the worker pool with a token-bucket rate limiter per provider, and back off exponentially on 429s. Reactive throttling adds hours to recovery.

A third issue is mixing transactional and marketing traffic on one sending domain. Marketing complaints tank reputation, which drops password-reset emails to spam. Split into two subdomains — mail.yourapp.com and notify.yourapp.com — each with its own SPF/DKIM/DMARC. Stripe, Notion, and Linear all do this.

Finally, teams under-invest in delivery receipts and observability. The provider's webhook telling you a message bounced, was delivered, or got marked spam is the only ground truth you have. Pipe receipts into a status table, expose per-channel success rates on a dashboard, and alert when any channel drops below its 30-day baseline by more than 2%. Without this, you find out about a deliverability cliff from an angry CEO, not from PagerDuty.

Related reading

• Webhook design for systems analyst interview
• API Gateway vs BFF for SA interview
• Idempotency keys for SA interview
• Kafka for systems analyst interview
• Event-driven architecture for SA interview

If you want to drill systems analyst design rounds like this every day, NAILDD ships with hundreds of system design prompts graded against senior-level rubrics.

FAQ

Should the Notification API be synchronous or asynchronous?

Asynchronous, always. The API accepts the event, persists it to the audit log, returns 202 with a notification_id, and hands off to the queue. A synchronous API couples producer latency to provider latency — if SES has a bad ten seconds, every checkout completion blocks for ten seconds. Async also makes retries, fan-out, and per-channel rate limiting tractable. Only the delivery-status lookup endpoint is synchronous, and it reads from a status table.

How do I prevent the same notification from being sent twice after a worker restart?

Two layers. At ingest, the producer's idempotency_key blocks duplicate events within a 24h window via a unique constraint on the audit table. At delivery, each provider call carries a notification_id, and the worker writes a delivered_at timestamp before acknowledging the queue message. If the worker crashes between send and ack, the redelivered message finds delivered_at already set and skips.

What's the right SLO for a password-reset email?

Industry standard is p95 under 3 seconds end-to-end from event emit to provider acceptance, and p95 under 30 seconds from emit to inbox delivery. Anything slower and users hit "resend" twice, generating duplicate tokens and support tickets. Track both the internal latency and the inbox-delivery latency separately.

How do I handle the unsubscribe link without breaking transactional messages?

CAN-SPAM and GDPR require unsubscribe on marketing email only. Transactional messages (receipts, password resets, security alerts) are exempt and must not show an unsubscribe footer — adding one trains users to mute critical mail. Keep the preference store at the (user_id, event_type, channel) granularity so a marketing unsubscribe doesn't suppress transactional traffic.

Should I build my own templating engine?

No. Use Handlebars or Liquid — both are open source, sandboxed, and have language bindings for every backend stack. A custom engine looks innocent for one quarter, then somebody asks for conditionals, then loops, then partials, and you've reinvented Liquid badly.