Rate limiting for systems analyst interview

Why rate limiting matters

You are in a systems analyst loop at Stripe, and the interviewer draws a public API on the whiteboard and asks: "A single client just sent 20,000 requests per second to your payment intent endpoint. What happens to the other tenants?" The answer the panel wants is not a generic "we add a limiter" — they want to hear token bucket vs sliding window, where the counter lives, and what HTTP status you return. This post walks through the four algorithms that come up in nearly every senior SA loop, plus the 429 response shape that backend teams will actually implement from your design doc.

Rate limiting protects four distinct things, and confusing them is the first mistake junior candidates make. It blocks abuse and DDoS at the edge, absorbs accidental traffic spikes from a misbehaving client (think a retry loop with no jitter), enforces fairness between tenants on shared capacity, and caps cost on expensive operations like AI inference or geocoding calls. Without a limiter, one badly written cron at a Notion customer can drain the connection pool for everyone else on the shard.

Token bucket

A token bucket holds N tokens and refills at a fixed rate R per second. Every incoming request consumes one token; if the bucket is empty, the request is rejected with 429 Too Many Requests. The bucket cap controls how big a burst you tolerate, and the refill rate controls the steady-state throughput.

capacity = 100 tokens
refill_rate = 10 tokens/sec

t=0s:   bucket = 100
t=0.5s: 100 requests in 500 ms -> bucket = 0
t=1s:   refill +10 -> bucket = 10
t=10s:  bucket back to capacity (100)

The trick reviewers test on is asymmetry: the bucket allows a burst up to its full capacity, but the long-run average is capped at the refill rate. This is why token bucket is the default for API gateways at AWS, GCP, and Cloudflare — real client traffic is bursty, and a pure rate cap rejects legitimate spikes. The implementation is two numbers per key in Redis: tokens_remaining and last_refill_ts. On each request you compute elapsed * refill_rate, clamp to capacity, then decrement.

Leaky bucket

Leaky bucket flips the model: requests enter a FIFO queue and drain at a constant rate. If the queue is full, the new request is rejected. The output is perfectly smooth — no bursts ever reach the downstream service — but you pay in two ways: requests sit in memory waiting for their drain slot, and the added queue latency can violate your p99 budget.

Use leaky bucket when the downstream is a fragile legacy system (an on-prem mainframe, a SOAP service with a hard CPS cap) that simply cannot absorb a spike. Use token bucket everywhere else. In a SA interview, mentioning that leaky bucket is essentially a single-server queue with deterministic service time scores points.

Fixed window

Fixed window keeps a counter per key per discrete time bucket (for example, per minute). When the bucket rolls over, the counter resets to zero. The whole thing fits in one Redis INCR with a TTL — it is the simplest production-grade limiter you can build.

window: 1 minute
limit: 100 requests

User A:
  12:00:00 - 12:00:59: 100 requests OK, request #101 -> 429
  12:01:00: counter resets to 0

The famous flaw is the boundary burst: a client can fire 100 requests at 12:00:59 and another 100 at 12:01:00, achieving 200 requests in 2 seconds while never violating the per-minute rule. For internal services where the limit is a soft fairness guardrail this is fine; for adversarial public APIs it is not.

Sliding window log and counter

The sliding window log stores a timestamp for every request in a sorted set, then counts entries in the last 60 seconds. It is exact — no boundary artefact — but memory scales linearly with traffic, which gets expensive at the Uber or DoorDash scale.

The sliding window counter is the compromise everyone actually ships. It keeps two fixed-window counters (current and previous) and estimates the rate by weighting the previous window:

elapsed_in_current = 20 sec       # we are 20s into the current minute
weight = (60 - 20) / 60 = 0.667
estimated_count = current + previous * weight

HTTP responses and headers

When the limiter rejects a request, return HTTP 429 Too Many Requests with three headers that real clients (and well-known SDKs) will respect:

HTTP/1.1 429 Too Many Requests
Retry-After: 60
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1748131200

Retry-After is the seconds (or HTTP-date) the client should wait before trying again — this is the only header in the RFC. The X-RateLimit-* family is de facto standard popularised by GitHub: Limit is the cap for the current window, Remaining is how many calls are left, Reset is the Unix timestamp when the bucket refills. Returning these on every response (not only on rejects) is the move that separates a thoughtful SA design from a junior one — well-behaved clients can self-throttle before they ever see a 429.

Algorithm comparison

Memorise this table — interviewers love asking "why not just use sliding window everywhere", and the answer is the memory column.

Common pitfalls

The first pitfall is picking the wrong key. Limiting by IP address feels safe until you discover that a single corporate NAT or a mobile carrier gateway sits behind one IP and represents thousands of legitimate users; you ban Verizon at lunch and the on-call gets paged. The fix is to layer keys — IP for unauthenticated traffic, API key or user ID for authenticated traffic, and a coarser tenant key on top for multi-tenant fairness. Each layer gets its own bucket, and a request must pass all of them.

A second pitfall is forgetting clock skew across the limiter fleet. If your Redis cluster uses node-local time and one node is 400 ms behind the others, the fixed-window reset becomes non-deterministic at the boundary and clients see phantom 429s right at minute rollover. The fix is to use a single authoritative clock — the Redis server time via TIME command, or a centralised NTP-synced source — and never the application server's Date.now().

The third pitfall, and the one that bites teams at Snowflake-scale, is not differentiating cost between endpoints. A GET /health and a POST /reports/generate both consume one token in a naive design, but the second one runs a 30-second query. Bake cost into the token consumption — health checks cost 1 token, reports cost 50 — so that the limiter actually protects the resource you care about, not just the request count.

A fourth pitfall is returning 503 instead of 429. A 503 means "the service itself is unavailable", which triggers aggressive retries in most client libraries and load balancer logic; a 429 means "you specifically should back off", and modern SDKs honour Retry-After. Mixing them up turns a rate-limit event into a cascading retry storm — exactly what you were trying to prevent.

Finally, candidates often forget that rate limiting is not security. A determined attacker rotates IPs and API keys; your limiter slows them down but doesn't stop them. Pair it with WAF rules, bot detection, and account lockout for credential stuffing. If the interviewer asks "is this enough to stop a botnet", the correct answer is "no, it buys time for the SOC to respond".

Related reading

• API Gateway vs BFF for systems analyst interview
• Circuit Breaker for systems analyst interview
• Capacity planning for systems analyst interview
• HTTP methods and status codes for SA interview
• Idempotency keys for systems analyst interview

If you want to drill systems-design questions like this every day, NAILDD is launching with hundreds of SA scenarios across exactly this pattern.

FAQ

Token bucket or sliding window — which one should I default to in an interview?

Default to token bucket for any public-facing API answer. It is what AWS API Gateway, Stripe, and GitHub actually ship, the implementation is two numbers in Redis, and it naturally allows the bursty traffic that real clients produce. Reach for sliding window counter only when the interviewer explicitly says "the boundary burst is unacceptable" — that signal points at the fixed-window flaw and the sliding counter is the textbook fix.

Where does the limiter state live in a multi-region deployment?

In an ideal world, each region has its own Redis-backed limiter and the per-region quota is sized so that regional caps sum to the global cap. Cross-region coordination is too slow — a Redis lookup in us-east-1 from eu-west-1 adds 80-100 ms to every request. If the business genuinely requires a global cap (say, a paid quota that customers buy once), accept the latency hit and use a single regional limiter as the source of truth, but make the typical case go through a local cache that pessimistically counts down.

How do I rate-limit per endpoint when one endpoint is 50x more expensive than another?

Assign a cost per route in your limiter config and consume that many tokens per request. A cheap GET costs 1 token, a heavy POST that triggers a downstream LLM call costs 50, a GET /health costs 0. This keeps the user-facing cap intuitive ("you have 1000 requests per minute") while actually protecting the resource. Document the cost table in your public API reference so customers can predict their consumption.

What is the right limit value? How do I pick a number?

Start from the downstream capacity, divide by expected concurrent tenants, and add a safety margin of 30-50%. If your database can sustain 5,000 QPS and you have roughly 100 active tenants, 30-50 RPS per tenant is a defensible starting point. Watch the 429 rate in production for two weeks and tune — a rate above 1% means the limit is biting legitimate users, a rate of zero means the limit is so loose it does nothing.

Should I rate-limit authenticated traffic differently from anonymous traffic?

Yes, almost always. Anonymous traffic should be limited tightly by IP because you have no other identifier and IP-based limits are the only defence against credential stuffing on the login endpoint. Authenticated traffic should be limited by user ID or API key with much looser caps, because you trust the identity. The login endpoint itself sits between the two — limit by IP and by attempted username, because the attacker rotates both.

How does rate limiting interact with retries and idempotency?

A well-behaved client receives a 429, reads Retry-After, waits, and retries the same idempotent request with the same idempotency key. If your retry storm is caused by clients not honouring Retry-After, the fix is client-side — exponential backoff with jitter and respect for the header. On the server, make sure your idempotency layer sits behind the rate limiter so that a rejected request never enters the dedup table and a successful retry is treated as the original attempt.