A system design answer is a set of decisions, each tied to a number and each with a named failure. Every box solves one problem and creates another: a cache absorbs reads and must then be invalidated; a quorum survives a lost node and slows writes; a queue lets a fast producer outrun a slow consumer and makes delivery semantics yours. One rule runs through all of it: never draw a box whose number is not on the page.
Not advice. General information, and every example is invented.
General information, not career advice, and no guarantee of any result. Every example, name and figure in an example box is invented, and so is every price. Numbers marked "assumption" are choices for the exercise, not measurements of any company. No company publishes its design rubric, so what is marked is the author's reading.
Start here: the priority spine
For: SDE-1 and up.
Context. Common designs come first, and each reuses blocks from the ones before.
Table: Order, Design, Where, Why here (author's estimate, not measured) (8 rows)
| Order | Design | Where | Why here (author's estimate, not measured) |
|---|---|---|---|
| 1 | Design 1: URL shortener | Appendix | The warm-up that uses every block |
| 2 | Rate limiter | The transcript | Asked alone, and part of every design |
| 3 | Design 6: notifications | This lesson | A vague prompt turned into numbers |
| 4 | Design 2: news feed | Appendix | Fan-out under skew |
| 5 | Design 4: chat | Appendix | Open sockets, one order per conversation |
| 6 | Design 3: payments ledger | This lesson | Correctness when a step half-completes |
| 7 | Design 7: flash-sale checkout | Appendix | One hot row |
| 8 | Design 5: ride matching | Appendix | A write rate and a claim race |
Know cold means you can say the skeleton from memory, defend it against the pushback bank and draw the boxes in three minutes. For any other prompt, use the unseen-prompt method near the end.
If you are short of time. Two hours: estimation and the block table, then say the eight skeletons aloud once. Lesson 5 sets the weeks beyond that; spend them on the pushbacks and on timed 45-minute runs scored with the rubric.
Eight spoken skeletons
Each follows the round's clock in minutes, from clarifying (0 to 5) to cost and paging (40 to 45). Say each aloud in two minutes. Cost lines sum nodes from "What one box handles", two thirds busy, at the Operate-it prices; a count marked "assumed" is a choice, not derived from the table.
Design 1: URL shortener
Code, 17 lines
0-5 Create, redirect, expiry, delete. A code is never reused.
Analytics, abuse scanning and login are out. p99 under
100 ms is my assumption.
5-10 100 million new links a month, 100 reads a write (assumed):
116 creates and 11,574 redirects a second at peak; 3.0 TB
in five years (500 B a link). So: reads win 100 to 1; cache first.
10-28 Load balancer, stateless app, cache by code, store sharded
by code on a hash ring, and an ID service handing out blocks.
28-40 Offer: the code. Base 62, 7 characters is 3.52 trillion.
Fail: a stale replica says 404 (check the primary first);
the cache dies and the store takes all 11,574 a second
at peak (the six shards are sized for it).
40-45 About Rs 8.96 lakh a month: 18 store nodes (six shards,
three copies) Rs 7.2 lakh, 8 app servers (assumed)
Rs 96,000, 3 cache nodes (one a zone) Rs 48,000, 5.0 TB
of egress Rs 32,000. Page on redirect errors above 1%
for 5 minutes.Rate limiter
Code, 16 lines
0-5 Limit per API key, at the gateway, per plan. Answer allow
or reject with retry-after. Billing-grade metering is out.
5-10 50,000 customers at 1,000 a minute; 40,000 checks a second
at peak (assumed); a 2 ms p99 budget inside a 100 ms call;
400 MB for a log against 0.8 MB for a counter.
So: the sliding counter, one atomic script per check.
10-28 Gateway calls the limiter; counters in a sharded store;
the limit table per plan shipped to gateways as config.
28-40 Offer: the race. Read, compare, write lets two gateways
pass the 100th; check and decrement in one script.
Fail: the store dies (local limit, limit over gateways);
one key past what its shard runs, say 100,000 a second
(lease 100 tokens at a time; 5,000 a second needs nothing).
40-45 About Rs 96,000 a month: six cache nodes at Rs 16,000,
three shards with a replica each, a third busy at peak.
Page on limiter p99 above 2 ms and the fail-open rate.Design 6: notifications
Code, 15 lines
0-5 Which channels, who sends, how many, what latency classes,
what must never happen. OTP in 5 s; no send after opt-out.
5-10 50 million users x 5 a day: 2,894 sends a second; a
campaign of 10 million in 10 minutes, 16,667 a second;
OTP within 5 seconds. So: separate lanes, not one queue.
10-28 One API with an idempotency key; per-channel queues per
priority; workers check preferences at send time, render,
call the provider; receipts update a status table.
28-40 Offer: the campaign against a 2,000-a-second provider.
83 minutes for the campaign; OTP has its own quota.
Fail: a provider dies (fail over; an OTP may repeat, a
payment alert checks status first); a send after opt-out
(check preferences at send time, not at enqueue).
40-45 About Rs 14.48 lakh a month, plus Rs 3 crore of SMS. Page
on OTP p99 above 5 seconds.Design 2: news feed
Code, 20 lines
0-5 Post, follow, read a feed newest first. Ranking is out
today; say it may come next quarter.
5-10 300 million daily users reading 5 times: 52,083 feed reads
a second at peak. 50 million posts a day at 200 followers:
pushing every post is 347,222 writes at peak (115,741 on
the mean); one star's post is 20 million. So: push for
most authors, pull for the large ones.
10-28 Post store; fan-out queue and workers; timeline cache per
user; a feed service that merges in large authors on read.
28-40 Offer: the threshold, 10,000 followers, assumed. If the top
1% hold 60% of follow edges (assumed), pushing the rest is
138,889 writes a second at peak, 46,296 on the mean.
Fail: an author just under the threshold posts and
floods the fan-out queue (lower the threshold);
my own post missing (write it to my timeline at once).
40-45 Rs 51.2 lakh to Rs 94.4 lakh a month: 90 to 225 memory
nodes of 64 GB for 5.8 to 14.4 TB of timelines, Rs 28.8
to 72 lakh; assumed: 100 app and fan-out servers Rs 12
lakh, 20 store nodes Rs 8 lakh, 6 brokers Rs 2.4 lakh.
Page on queue lag above 30 seconds.Design 4: chat
Code, 16 lines
0-5 One-to-one and groups up to 256; sent, delivered, read;
one order per conversation; offline sync. Media out.
5-10 500 million daily users x 40 messages: 694,444 a second at
peak; 2.6 deliveries each, 1.8 million a second; 100 million
sockets on 1,500 gateways, two thirds full.
So: the hard parts are sockets and the sequence number.
10-28 WebSocket gateways and a registry of who is where; a chat
service appends to a per-conversation log, then delivers.
28-40 Offer: ordering. One writer per conversation numbers it;
a client that sees 40 then 42 asks for 41.
Fail: a gateway dies (67,000 reconnect with jitter);
a crash after append, before delivery (sync on reconnect).
40-45 About Rs 2.93 crore a month: 1,500 gateways Rs 1.8 crore;
70 log shards x 3 = 210 nodes Rs 84 lakh; 30 days at 200 B
x 3 copies = 360 TB on block storage, Rs 28.8 lakh. Page on
send-to-delivered p99 above 2 seconds.Design 3: payments ledger
Code, 12 lines
0-5 Pay a person or a merchant; bank payouts in scope; money
never moves twice and never half-moves. Data stays in India.
5-10 2,400 transfers a second at peak; 10.1 TB a year; 69
million idempotency keys a day. So: not a scale problem.
10-28 Double-entry rows; the key, the entries and an outbox row
in one transaction; the bank leg as a saga from the outbox.
28-40 Offer: the bank times out. Pending, never a false failed;
ask the bank by my transfer id; compensate on a no.
Fail: a crash between the effect and the key (one commit);
two payments overdraw one account (conditional update).
40-45 About Rs 8.42 lakh a month at year end. Page on any
unbalanced transfer and on pending past the bank's window.Design 7: checkout and a flash sale
Code, 16 lines
0-5 1,000 units, 2 million people at noon, one per buyer.
Never oversell; a late payment is refunded.
5-10 200,000 attempts a second; one stock row clears about 500
holds a second; 20 rows clear 10,000. So: stop the crowd
at the edge, admit twice the stock, split the row.
10-28 CDN waiting page with a sold-out flag; a gate of 2,000
tokens; one transaction takes a unit and writes the hold,
the order and an outbox row; the ledger's saga pays.
28-40 Offer: the guard. avail >= 1 in the WHERE clause, checked
again after the lock wait. Fail: payment after expiry
(refund); a Redis failover losing a decrement (DB decides).
40-45 About Rs 5.84 lakh a month: 30 app servers Rs 3.6 lakh, 3 SSD
database nodes Rs 1.92 lakh, 2 cache nodes Rs 32,000; each
sale's 10 minutes add 120 million CDN requests, Rs 9,600.
Page on sold above stock and on an expired hold older
than 1 minute.Design 5: ride matching
Code, 18 lines
0-5 Drivers report every 4 seconds; a rider asks; offer to a
good nearby driver for 10 seconds. Pricing and ETA out.
5-10 1 million drivers online, one update every 4 seconds:
250,000 updates a second; 100 MB of latest positions;
about 5% of updates cross a cell. So: write rate is the
problem; keep positions in memory, sample a trail.
10-28 Location service keyed by driver; an in-memory index from
cell (a map square) to drivers; a matcher that reads about nine cells.
28-40 Offer: one driver, two riders. Compare-and-set the driver
from available to offered, with a lease.
Fail: a driver goes silent (dropped after 12 seconds
without an update); two matchers race (one CAS wins).
40-45 About Rs 10.45 lakh a month: 38 gateways Rs 4.56 lakh, 8
position shards x 2 = 16 cache nodes Rs 2.56 lakh; 6 index
nodes Rs 1.92 lakh and 10 matchers Rs 1.2 lakh, assumed;
the trail kept 30 days (assumed), 13 TB, Rs 20,800. Page
on match-found rate below its band and location lag
above 10 seconds.API and tables for the five card designs
What goes on the board at minutes 10 to 15.
Table: Design, Calls, Tables, and the key (5 rows)
| Design | Calls | Tables, and the key |
|---|---|---|
| 1 URL shortener | POST /v1/urls with an Idempotency-Key: 201 with the code, or 409 alias taken; GET /{code}: 302, 404 or 410 expired; DELETE /v1/urls/{code} | urls(code PK, long_url, owner_id, expires_at), sharded by code. Key: the code, a counter block per app server put through a keyed permutation, then base 62, so codes cannot be walked. |
| 2 News feed | POST /v1/posts; GET /v1/feed?cursor=&limit=20: items and the next cursor; PUT and DELETE /v1/follows/{user_id} | posts(post_id PK, author_id, created_at); follows(follower_id, followee_id), indexed both ways; 800 post ids per user's timeline in memory. Key: an opaque cursor (time plus post id), never an offset. |
| 4 Chat | On a WebSocket: send(conversation_id, client_msg_id, body) returns the seq; ack(up_to_seq) for delivered or read; sync(last_seq) | messages(conversation_id, seq, sender, client_msg_id, body), partitioned by conversation, unique on the client's id; cursors(conversation_id, user_id, delivered_seq, read_seq); a registry of user to gateway with an expiry. Key: (conversation, seq). |
| 5 Ride matching | PUT /v1/drivers/{id}/location every 4 s; POST /v1/rides: 202 matching; POST /v1/rides/{id}/accept: 200, or 409 offer gone | drivers(driver_id PK, status, offer_ride_id, offer_expires_at); rides(ride_id PK, rider_id, driver_id, status, version); positions and the cell index in memory. Key: the driver, claimed by UPDATE ... WHERE status = 'AVAILABLE'. |
| 7 Flash sale | POST /v1/sales/{id}/buy with an Idempotency-Key: 201 with a 10-minute hold, or 409 sold out; POST /v1/orders/{id}/confirm from the payment saga: 200 paid, or 410 hold expired | stock(sku, bucket, avail) with CHECK (avail >= 0), 20 buckets; holds; orders; sale_buyers(sale_id, buyer_id) as the primary key for one per buyer; an outbox. Key: UPDATE stock SET avail = avail - 1 WHERE ... AND avail >= 1. |
Next stop: Capacity estimation: the arithmetic you do out loud, 3 min
Capacity estimation: the arithmetic you do out loud
For: SDE-1 and up; interns facing a light design question.
Context. An estimate exists to find the number that decides the design: at 39 writes and 3,858 reads a second, the reads are what you design. A day has 86,400 seconds; rounding to 100,000 makes every rate about 14% too low, so round only if you say which way the error runs. A peak factor, usually three, is your assumption. Three formulas cover most rounds: rate = count / seconds in the window x peak factor; bytes = count x size x copies x days kept; nodes = load / per-node budget.
Bandwidth is rate x bytes x 8 bits; take a 10 Gbit/s card as 7 usable (assumption). Memory is items x (payload + overhead) x copies.
Table: Quantity, Arithmetic, So (4 rows)
| Quantity | Arithmetic | So |
|---|---|---|
| Shortener redirects | 11,574 a second x 500 B = 46 Mbit/s | Bandwidth is not the issue. |
| Feed responses | 52,083 a second x 20 KB = 8.3 Gbit/s | More than one card. |
| Feed timelines | 300 million x 800 ids x 8 B = 1.92 TB raw; x 3 copies, x 1 to 2.5 for overhead (assumed): 5.8 to 14.4 TB | Memory is the bill. |
| Chat sockets | 100,000 a gateway x 10 KB = 1 GB | Socket memory sets the gateway count. |
What one box handles
Every design here takes its per-node budget from this table, the author's estimate (not measured): round numbers for small requests, to be replaced by a load test. Each tier gets 1.5 x peak / budget nodes, so it runs two thirds busy.
Table: Box or hop, Handles or takes, roughly, Where this lesson uses it (12 rows)
| Box or hop | Handles or takes, roughly | Where this lesson uses it |
|---|---|---|
| App server, 8 cores | 1,000 requests a second that do real work; 10,000 that only read a cache or forward | 17 notification workers; 38 ride gateways for 250,000 updates |
| WebSocket gateway | 100,000 open sockets at 10 KB each | 1,500 chat gateways |
| Cache node, in memory | 50,000 simple operations a second; 40,000 short scripts | The limiter's 40,000 checks on 3 shards; ride positions on 8 |
| Database node, relational or key-value | 1,000 short write transactions a second; 2,000 reads that miss memory | The ledger's 4 shards; the shortener's 6 |
| Log or wide-column node | 10,000 small writes a second; 15,000 batched | The status store's 5 shards; the chat log's 70 |
| Queue partition | About 10,000 messages of 1 KB a second; one consumer in a group | Partitions: rate / 10,000 or the consumers you need, whichever is more |
| One hot row | 500 locked updates a second at 2 ms each; 16 with a 62 ms commit (2 + 60) to a second region | The flash sale's 20 rows for 10,000 holds; a ledger account's debits |
| Throttled backfill | 20,000 rows a second | 50.5 billion ledger rows in 29 days |
| Network card | 10 Gbit/s, 7 usable | The feed's 8.3 Gbit/s needs two |
| Memory read; SSD read; round trip in one zone; a cache get | 100 ns; 0.1 ms; 0.5 ms; 1 ms | Why a cache hit is cheap; the limiter's 2 ms budget |
| Commit on SSD | 1 to 5 ms; 2 ms used here | 500 a second on one hot row |
| Round trip between two Indian regions | Tens of ms; 60 ms assumed | The ledger's synchronous second site |
Next stop: Building blocks: one trade-off table, 2 min
Building blocks: one trade-off table
For: SDE-1 and up; interns facing a light design question.
The rows are the author's reading of standard practice.
Table: Block, Use it when, The number to say, Cost and failure (14 rows)
| Block | Use it when | The number to say | Cost and failure |
|---|---|---|---|
| Load balancer | Stateless servers behind one address | Least-loaded beats round robin when request costs vary | Sees a slow server late; pair it, or it is the single point |
| Cache (look-aside) | Reads dominate, 100 to 1 in the shortener | At a 90% hit ratio the store sees a tenth of the reads | Staleness; a lost cache sends every read to the store; a hot expired key stampedes (use a lease) |
| Consistent hashing | Shards are added and removed | Adding one of N nodes moves about 1/N of the keys; hash mod N moves nearly all | A hot key stays on one node; virtual nodes even out the rest |
| Replication and quorums | Survive a node; spread reads | N = 3; R + W > N makes read and write sets overlap | Overlap is not linearizability (see consistency); async replicas lag, so a user's own write can vanish |
| Partitioning | One machine cannot hold the data or the writes | 11,574 reads / 2,000 a shard = 5.8, so 6 | An off-key query hits every shard; a skewed key makes a hot shard |
| Queue or log | A fast producer, a slow consumer; fan-out | Drain = backlog / rate: 10 million at 2,000 a second is 83 minutes | At least once, so consumers drop repeats by id; order only within a partition; a poison message needs a dead-letter queue |
| CDN | Cacheable bytes far from the origin | At a 95% hit ratio (assumed) the origin sends a twentieth of the bytes | Purge delay; a per-request bill |
| Rate limiter | One caller must not starve the rest | 2 ms p99 inside a 100 ms call | Fail open or closed is a choice; the check must be atomic |
| Idempotency key | A retry must not repeat an effect | Kept for the longest retry window: 24 hours in Design 3 | Key and effect commit together, or a crash splits them |
| Search index | Text relevance, filters, facets | Searchable after a refresh, about a second by default (Elastic) | Never the only copy; rebuilt from the source of truth |
| Blob store | Files, images, video | Object storage at a fifth of block storage's price per GB | Upload direct with a signed URL; metadata lives in a database |
| Database family | Choose by the query, not the brand | Relational for constraints and joins; key-value for exact keys; wide-column for a log per entity; document for whole records | An LSM tree writes fast and may read several files; a B-tree reads steadily, costs more per random write |
| Microservices or a modular monolith | Split a service when a team or a scaling need demands its own deploy | A module call is nanoseconds; a network hop is a millisecond and a new failure | Microservices bring partial failure, sagas and many deploys; a modular monolith keeps one deploy and splits later along its modules, so start there |
| API gateway | One front door for many services | One hop of a few milliseconds (assumption) | Does TLS, authentication, rate limits, routing and request ids; keep business logic out of it |
Next stop: The API layer and security at each hop, 2 min
The API layer and security at each hop
For: SDE-1 and up.
The rows are the author's reading of common practice; codes and header names are a choice, so say "for example".
Table: Topic, Do this, Why, and the cost (5 rows)
| Topic | Do this | Why, and the cost |
|---|---|---|
| REST resources | Nouns in the path, HTTP verbs, a version: POST /v1/transfers, GET /v1/transfers/{id} | GET is safe to repeat, PUT and DELETE are idempotent; POST is neither, so it needs a key |
| Status codes | 201, 202 accepted, 401, 403, 409 conflict, 429 with Retry-After | Retries read them; 202 says pending |
| Pagination | An opaque cursor (time plus id), not an offset | An offset shifts as rows arrive and rescans skipped rows; a cursor is one index seek |
| Idempotency | An Idempotency-Key header on every POST with an effect, stored with the request hash and response | A retry gets the first answer; the same key with a different body is a 409 |
| Sessions or JWT | A session id in a cookie points at a server-side row; a JWT is a signed token checked without a lookup | A session is revoked by deleting its row; a JWT lives until expiry, so give access tokens minutes and refresh with a revocable token |
OAuth. OAuth 2.0 grants an app scoped access to a user's data without their password: after consent the app swaps a short-lived code, server to server, for an access token; PKCE makes a stolen code useless. OpenID Connect adds an ID token saying who the user is (the author's reading of the standards).
Table: Hop, Threat, Control (9 rows)
| Hop | Threat | Control |
|---|---|---|
| Client to edge | Eavesdropping, tampering | TLS and HSTS; TLS inside the network too if the threat model says so |
| Edge | Floods, scraping, password guessing | Rate limits per identity, not only per IP (many users share one NAT address) |
| Browser | XSS: injected script runs as the user | Escape output for its context, a Content-Security-Policy, HttpOnly cookies |
| Browser | CSRF: another site makes the browser send your cookie | SameSite cookies and a CSRF token on state changes |
| Browser | CORS taken for protection | CORS only says which origins a browser lets read a response; allow named origins only |
| App | SQL or command injection | Parameterised queries; validate input at the boundary |
| App | Reading another user's object (OWASP A01:2025, broken access control) | An ownership check on every read and write, with the user taken from the verified token |
| App to network | SSRF: a user-supplied URL makes the server fetch an internal address | Allow-list destinations; block private ranges after resolving |
| Data | Theft at rest, leaked secrets | Encryption at rest, secrets in a vault, an audit log of reads |
Next stop: Consistency: CAP, PACELC and the models, 3 min
Consistency: CAP, PACELC and the models
CAP. Seth Gilbert and Nancy Lynch proved a formal version of Eric Brewer's conjecture in 2002 (the author's reading). C is linearizability, A is a non-error answer from every live node, P is a network that may lose messages. During a partition a system cannot keep both C and A. It is not "pick two": partitions are not optional, and it says nothing about latency.
PACELC. Daniel Abadi (IEEE Computer, February 2012) adds the other half: if Partitioned, trade Availability against Consistency; Else, trade Latency against Consistency, a trade paid on every request. His 2012 classes under default settings: Dynamo and Cassandra PA/EL, VoltDB PC/EC, MongoDB PA/EC. Defaults change, so date them.
Table: Model, strongest first, What a reader may see, Price, and where it fits (5 rows)
| Model, strongest first | What a reader may see | Price, and where it fits |
|---|---|---|
| Linearizable | The latest completed write, as if one copy existed | A leader or majority round trip; usernames, locks, a balance |
| Causal | A write before whatever it caused (a reply after its post) | Version vectors; threads, chat replies |
| Read-your-writes | Your own writes, always | Read the leader briefly after a write; the "my post is missing" fix |
| Monotonic reads | Never an older value after a newer one | Pin a user to one replica |
| Eventual | Anything stale, until replicas converge | Cheapest; counters, likes, feeds |
A quorum with R + W > N is not linearizable by itself: concurrent writes, a write that failed after reaching some replicas, or a sloppy quorum can let one read see a value a later read does not (the author's reading of the Dynamo paper).
Isolation concerns concurrent transactions on one database. PostgreSQL 18 (documentation read 1 Nov 2026) defaults to Read Committed; its Repeatable Read is snapshot isolation. Write skew is two transactions reading the same facts, writing different rows and together breaking a rule.
Table: Level, Non-repeatable read, Phantom, Lost update, Write skew (3 rows)
| Level | Non-repeatable read | Phantom | Lost update | Write skew |
|---|---|---|---|---|
| Read committed (default) | Possible | Possible | Possible if the app reads, then writes | Possible |
| Repeatable read | No | No in PostgreSQL | Second writer rolled back | Possible |
| Serializable | No | No | No | No |
None allows a dirty read. The stronger levels fail with SQLSTATE 40001, and the documentation says to retry the whole transaction.
INVENTED example 1 min
A joint account holds Rs 1,000. Two withdrawals of Rs 800 run together at Read Committed; each sums the entries, sees Rs 1,000 and inserts its own debit, so both commit and the balance is minus Rs 600. Lock the account row with SELECT ... FOR UPDATE before summing, or run Serializable and retry on 40001.
One writer: fencing and leader election. A lease is a lock that expires, so a holder paused by garbage collection can wake after expiry and write anyway. Martin Kleppmann's essay (8 Feb 2016, read 1 Nov 2026) gives the fix, a fencing token: the lock service hands out a number that rises with every grant, and the storage rejects a write carrying a lower number than it has seen. Raft (read 31 Oct 2026) does the same with terms: a leader needs a majority, a node votes once per term, and an older term is rejected, so a cut-off old leader commits nothing. Put an odd number of voters in three failure domains.
Next stop: Two-phase commit versus a saga, 1 min
Two-phase commit versus a saga
Context. Two-phase commit (2PC) asks every participant to make its change durable, hold its locks and vote, then tells all to commit or all to abort. Jim Gray and Leslie Lamport (2004, read 2 Nov 2026) name the weakness: if the coordinator fails, a participant that voted yes must wait, holding locks. Their Paxos Commit uses 2F + 1 coordinators to survive F failures.
A saga gives up isolation instead. Hector Garcia-Molina and Kenneth Salem (SIGMOD 1987) define it as a sequence of local transactions, each with a compensating transaction; on a failure the compensations for completed steps run in reverse, and other transactions may see a partial saga.
Choose. One database: a local transaction. Shards of a store with distributed transactions: its 2PC, naming the blocking risk. Services or companies: a saga, with idempotent compensations, the irreversible step last and the in-between state shown as "pending".
The outbox. A service that commits and then publishes can crash between the two. The outbox writes the event as a row in the same transaction; a relay publishes it and marks it sent, at least once, so consumers drop repeats by event id. Never promise exactly-once end to end.
Next stop: The Operate-it block that closes every design, 1 min
The Operate-it block that closes every design
Context. The Google SRE Workbook (re-read 2 Nov 2026) defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation", kept small because harm scales with traffic: a 20% error rate on a 5% canary is about 1% overall. So every design ends with four lines, in two minutes.
- Roll out and roll back. A canary or a flag at 1%, compared on the two or three metrics that mean user harm; what flips it back, and how fast.
- Blast radius. What a bad deploy, config or dependency can break, and the wall that stops it (a shard, a tenant, a region, a bulkhead).
- Watch. Three signals tied to user harm: not "CPU" but "messages older than 5 seconds". Name the one that pages.
- Monthly cost line. An order of magnitude and the item that dominates. Every design prices from one invented list (assumption): Rs 12,000 a month for an app server; Rs 16,000 for a 32 GB cache node, Rs 32,000 for a 64 GB memory node; Rs 40,000 for a store node on network disk, Rs 64,000 on local SSD; Rs 8 per GB-month of block storage, Rs 1.6 of object storage; Rs 6.4 per GB of egress; Rs 80 per million CDN requests. A node's price includes a 1 TB disk; a store holding more than 1 TB a node keeps all its data on block storage instead, on network-disk nodes. In a real round, use the day's rate card.
Next stop: Design 3: a payments ledger, 5 min
Design 3: a payments ledger
Requirements. Accept a payment request that debits one account and credits another, with a status of pending, posted or failed. Every movement balances. A retry of the same request moves money once. A balance can be read at any time; a mistake is corrected by a reversal, never an edit. Out of scope: card data handling, tax and currency conversion, each owned by another system and said aloud.
Estimate. Assumption: 800 transfers a second on average, three times that at peak. Computed: 2,400 a second at peak; 69.1 million transfers a day; 25.2 billion a year. Two entries per transfer gives 4,800 entry inserts a second at peak, 50.5 billion rows a year; at an assumed 200 bytes a row that is 10.1 TB a year and 70.6 TB over seven years. Idempotency keys kept for 24 hours at 800 a second is 69 million keys a day, so the idempotency store is a sized table of its own. So: throughput is modest; the difficulty is correctness when a step half-completes, and when a bank's answer comes late or never.
Interface and data. POST /v1/transfers with an Idempotency-Key header returns 202 and status pending (the bank leg is slow) or 201 posted. GET /v1/transfers/{id} returns status and entries. GET /v1/accounts/{id}/balance returns posted and pending amounts. POST /v1/transfers/{id}/reversals makes the reversing transfer. The tables:
Code, 7 lines
accounts(account_id, currency, status)
transfers(transfer_id, idempotency_key UNIQUE, status, request_hash,
created_at, updated_at)
entries(entry_id, transfer_id, account_id, direction D|C, amount,
currency, posted_at) -- sum(D) = sum(C) per transfer
outbox(event_id, transfer_id, payload, sent_at NULL)
idempotency(key, response, created_at) -- kept 24 h or moreThe transfer row, its two entries and its outbox row commit in one database transaction, so a half-written transfer is impossible on one shard.
The relay delivers to the bank at least once, so the transfer id is the bank's idempotency key (an assumption to confirm with the bank).
Failures (SDE-2). Double posting is blocked by the unique key on the transfer and the idempotency key. Across shards, a saga with a compensating reversal prevents a partial posting. An overdraft: check the balance and write in one transaction, with a row lock on the account or a conditional update; say which (the write-skew example under consistency shows why). A reconciliation job compares the bank's statement file against the entries and sends every difference to a break queue for a person; it never edits posted entries.
INVENTED example 1 min
An invented candidate is asked what happens when the bank never answers. They say: "The transfer stays pending. The worker keeps asking by transfer ID on a schedule. After a limit, say a day, it goes to the break queue and a person decides, because I will not guess whether the money moved. The user sees pending the whole time, never a false failed." They have named the state, the retry, the limit and the human, and the user-visible effect.
Senior depth (adds). Three deep dives. Stored versus derived balance: a stored balance is fast to read and must be updated in the same transaction as the entries; a derived balance, summed from entries, is always right and slower; a snapshot plus the sum since the snapshot is the usual middle (author's reading). Sharding by account versus by transfer: by account makes a balance read one shard but puts a transfer between two accounts on two shards; by transfer keeps one transfer's entries together but makes balance reads scatter. Pick one and state its cost. The hot row: a synchronous commit to a second region holds an account's row lock about 62 ms (2 + 60), so one account takes about 16 locked debits a second, not 500; the node still runs 1,000, because group commit shares the round trip (Little's law: 1,000 x 0.062 s is 62 in flight). So credits append without the lock, only debits lock, and a payout pool debited 100 times a second splits into 1.5 x 100 / 16, so 10 sub-accounts. At ten times the load, 24,000 transfers a second at peak, the outbox relay and the bank's own rate limit become the bottleneck before the database; say which limit you would ask the bank for.
- Roll out and roll back. Money code goes to a shadow path first: the new code runs on copies of real requests and its entries are compared with the old code's, writing nothing. Then 1% of real transfers, then 10%. Roll back by flag; a transfer in flight finishes on the code that started it.
- Blast radius. A bug that writes wrong entries is the worst case, so the wall is the reversal rule: entries are never edited, only reversed, and a nightly job proves that every transfer sums to zero and that stored balances equal summed entries. The blast radius is then "transfers since the last clean check", a number you can name. A dead bank adapter holds only its own transfers pending.
- Watch. Page on any transfer whose entries do not balance (target zero), on pending older than the bank's promised window, and on the break queue growing. Ticket on idempotency-key conflicts, which can be a client bug.
- Monthly cost line. 2,400 transfers a second at peak x 1.5 over 1,000 transactions a primary is 3.6, so 4 shards with three copies. Storage at the year-end size, 10.1 TB x 3 copies, is 2.5 TB a node, over the 1 TB a node includes, so all of it sits on block storage at Rs 8 per GB (Rs 2.42 lakh), and the 12 nodes are network-disk ones at Rs 40,000 (Rs 4.8 lakh; load-test the 2 ms commit on network disk). 10 app servers (Rs 1.2 lakh, assumed). That is Rs 8.42 lakh a month at the end of year one. The ledger grows from zero, so the year's average storage is half, about Rs 1.21 lakh, and the year averages about Rs 7.21 lakh, rising about Rs 20,200 a month as rows accumulate. Database nodes dominate.
Next stop: Design 6: a notification system, 4 min
Design 6: a notification system
Context. The prompt is vague on purpose, so the first move is five questions that become numbers.
INVENTED example · a vague prompt turned into numbers 1 min
Interviewer. Design a notification system.
Candidate. Five questions, then I state assumptions. Which channels: push, email, SMS, in-app? Who sends: other services through an API, or marketing campaigns too? How many a day, and how bursty? What latency classes: a one-time password in seconds versus a newsletter in hours? And what must never happen: a duplicate, a send to someone who opted out, or a lost message? Assumptions, said aloud: all four channels; 50 million users, 5 notifications a user a day, so 250 million a day, 2,894 a second on average; a marketing campaign of 10 million pushes inside 10 minutes, 16,667 a second; OTP within 5 seconds; opt-outs honoured at send time; at least once with a dedupe key, so a duplicate is rare and a loss is not allowed. If you want a different number I change the design only where it matters.
Signal: five questions with a reason each, and every answer becomes a number or a rule the design uses.
The design. Producers call one API with a template id, a recipient, a channel list, a priority and an idempotency key. The service writes the request, then fans it into per-channel queues, one set per priority: a one-time password must never wait behind a campaign, so the priority lanes are separate queues with separate workers, not one queue sorted. A worker checks the user's preferences and quiet hours at send time (not at enqueue time, since a campaign may sit for an hour), applies a per-user cap (for example 3 marketing messages a day), renders the template, calls the provider, and records the result. Providers answer with a message id and later a delivery receipt on a webhook; the status table moves queued, sent, delivered or failed. Retries back off with jitter up to a limit, then go to a dead-letter queue. The dedupe key is checked before the provider call and passed to a provider that takes one, since a timeout may have sent. Two SMS providers with a health check and a failover.
Deep dives. The campaign burst of 16,667 a second against a provider that accepts 2,000 a second (assumption): the queue absorbs it, the campaign takes 10,000,000 / 2,000 = 5,000 seconds, 83 minutes, and the OTP lane is unaffected because it has its own workers and its own provider quota. Fan-out of one request to a million users is a job that walks a segment in pages and writes per-user rows, with a cursor so a crash resumes. Cost is the surprise. If 1% of the 250 million a day goes by SMS, that is 2.5 million at an invented Rs 0.4, Rs 10 lakh a day, Rs 3 crore a month, far above the infrastructure, so the product rule "push first, SMS only for OTP and fraud alerts" is a design decision worth Rs 3 crore.
- Roll out and roll back. A new template or provider route goes to 1% of recipients by hash of the user id, with the old route as control; compare delivery rate and complaint rate. A provider switch is a flag per channel, back in seconds. A kill switch stops a campaign's fan-out job; a sent message cannot be recalled, so the review step is the real control.
- Blast radius. A wrong campaign reaches its whole segment, which can be millions of people: the walls are a per-user daily cap, a send-rate ceiling per campaign, and a required second approver above a size limit. A dead provider affects one channel and one lane.
- Watch. Page on OTP end-to-end p99 above 5 seconds (the SLO), on dead-letter queue growth, and on a send to a user who opted out (target zero). Ticket on delivery-receipt rate per provider, complaint or unsubscribe spikes after a campaign, and queue age per lane.
- Monthly cost line. Peak sends are 3 x 2,894 = 8,682 plus a campaign paced at its provider's 2,000, about 10,700 a second. At an assumed 1,000 sends a worker and 1.5 times the peak, that is 17 workers, plus 5 servers for the API, the fan-out job and receipt webhooks: 22 app servers (Rs 2.64 lakh). Three status writes a message (queued, sent, receipt) make about 32,000 a second; at an assumed 10,000 writes a node, 5 shards with three copies, 15 nodes on local SSD (Rs 9.6 lakh), and 30 days of rows at 200 bytes, 4.5 TB with copies, is 0.3 TB a node. 3 queue brokers on local SSD, for the write rate (Rs 1.92 lakh); 2 cache nodes for preferences (Rs 32,000). Infrastructure about Rs 14.48 lakh a month; SMS, Rs 3 crore, is about 21 times that.
Next stop: Recovery, SLOs and migrations: one table, 1 min
Recovery, SLOs and migrations: one table
Table: Question, The rule, A number (6 rows)
| Question | The rule | A number |
|---|---|---|
| RPO: data you may lose | With an async replica, the lag at the moment of loss; at risk = lag x write rate | The ledger at 2,400 writes a second risks 2,400 per second of lag, so it commits synchronously to a second site (60 ms, assumed) |
| RTO: time to recover | Detect from outside, fence the old primary, promote, repoint, verify | Ledger 15 minutes, chat 5 (assumed); rehearse it |
| Active-active | A conflict rule: last writer wins (skew drops writes), a home region per entity, or CRDTs | Chat homes each conversation in one region |
| SLO and burn rate | Budget = 100% minus the SLO; page on fast burn over a short window, ticket on slow burn | 99.9% over 30 days is 43.2 minutes; a burn rate of 14.4 spends 2% of it in an hour |
| Headroom | Lose one of three zones and two thirds carry the peak | At most two thirds busy at peak: nodes = 1.5 x peak / budget, rounded up; except the shortener's six shards, sized for a dead cache, and the feed's memory, a cache that refills |
| Expand, then contract | Add the new shape, dual write, backfill in throttled batches that never overwrite newer rows, verify, switch reads behind a flag, then drop the old | Every step reversible but the last; 50.5 billion ledger rows at an assumed 20,000 a second take 29 days |
Next stop: Overload: retry budgets, deadlines, shedding, breakers, 1 min
Overload: retry budgets, deadlines, shedding, breakers
For: SDE-3, Staff, Principal.
Context. Overload gets worse when every caller retries. Rows one to three follow the Google SRE book (read 4 Nov 2026); the breaker row is the author's reading.
Table: Tool, The rule, A number (4 rows)
| Tool | The rule | A number |
|---|---|---|
| Retry budget | Retries multiply down the stack: retry only at the layer just above the one rejecting, and only while retries are under 10% of requests | 3 tries in each of 5 layers is 3^5 = 243 calls at the bottom for one request; the 10% budget holds it near 1.1 times |
| Deadlines | Pass the caller's remaining time down every hop; drop work whose deadline has passed | A 100 ms call that has spent 70 ms gives the next hop 30 ms |
| Load shedding | Near overload, reject the lowest priority first, cheaply, with 503 or 429 and Retry-After | Design 6 sheds the campaign lane long before an OTP |
| Circuit breaker | Stop calling a dependency that keeps failing: answer at once with a fallback, then let one probe through | Open at 50% errors over 10 s, probe after 5 s (assumed) |
Page on the shed rate and the retry ratio, not only on errors.
Next stop: A senior-bar model answer: a rate-limiting service, start to finish, 8 min
A senior-bar model answer: a rate-limiting service, start to finish
Context. An invented interview, condensed to the exchanges that matter. Lines marked Signal say why a Senior reader would credit the sentence. Cover the candidate's lines and answer each interviewer line aloud first.
INVENTED example · a Senior transcript 5 min
Interviewer. Design a rate-limiting service for a public API.
Candidate. Two questions before I draw. Who is limited: an API key, a user, an IP? And is this a library each service calls or a shared service? I will assume per API key, called by the gateway on every request. In scope: allow or reject with a reason and a retry-after, limits configured per plan, a decision in a few milliseconds. Out: billing-grade metering, and abuse detection beyond the limit. Is that right?
Interviewer. Fine. Say 50,000 customers.
Signal: the in-list and out-list are agreed first, and "per key, at the gateway" is an assumption said aloud, not silently chosen.
Candidate. Numbers. I will assume 40,000 requests a second at peak across the fleet, so 40,000 limiter checks a second. The check sits on every request, so I budget 2 ms at p99 of a 100 ms call. That is one cache node's whole budget of 40,000 short scripts, so I shard by key over three (two meet 1.5 x peak / budget; the third buys headroom), each a third busy with a replica, and load-test it. Memory depends on the algorithm, which is my first decision.
Candidate. Four options: fixed window, sliding log, sliding counter, token bucket. Fixed window lets a client send twice the limit across a boundary, 10 at 999 ms and 10 at 1,000 ms. The log is exact but stores a timestamp per allowed request: at 1,000 a minute and 50,000 customers that is up to 400 MB at 8 bytes each, more in a real store. The counter stores two 8-byte counts per customer, with the minute in the key name, about 0.8 MB, and is accurate enough if traffic inside a minute is roughly even. The token bucket stores two numbers and permits a burst, which customers like. I pick the sliding counter for the default limit and a token bucket for plans that sell bursts.
Interviewer. Two gateways read the count at the same time. What happens?
Candidate. If the check is a read, a compare and a write, both read 99 and both allow the 100th, so the limit leaks under concurrency. The check, the refill and the decrement have to be one atomic operation in the store, a Lua script on Redis for example. I tested it in Java: eight threads, a limit of 100 and a store round trip between the read and the write. The atomic version let exactly 100 through, get-then-set about 800 (each SET overwrites the others' increments) and get-then-INCR up to 107.
Signal: the race is named, the fix is the mechanism, and the claim is backed by something run, not asserted.
Interviewer. Whose clock?
Candidate. Not the gateways'. 200 ms of skew is only 0.3% of my one-minute window, but near each boundary two gateways would count one key's requests into different minutes. The script reads the store's own time, so one clock decides every window for that key.
Interviewer. The Redis node dies. What do you do?
Candidate. It is a choice between two failures. Fail closed rejects everyone, which turns a limiter outage into a full outage, so I would not. Fail open allows everything, which risks overload but only until the node returns. I take a middle path: on a timeout the gateway applies a local limit of the customer's limit divided by the number of gateways, in memory. It is inaccurate but bounded. A replica takes over; I accept losing the last second of counters, which means a brief over-allowance, not a wrong answer for a minute.
Signal: both extremes named with their cost, then a bounded third option; the lost state is quantified as one second.
Interviewer. One customer grows to 100,000 requests a second on one key.
Candidate. At 5,000 a second I would change nothing: that is an eighth of one node's 40,000 budget. At 100,000, that key alone outruns its shard, and sharding cannot split one key. The gateway can keep a local token lease: it asks the store for 100 tokens at a time for that key and serves from memory, so the store sees 1,000 calls a second, not 100,000. The cost is accuracy: unused leased tokens are held by a gateway, so the customer can briefly exceed the limit by up to the leased amount across the fleet.
Interviewer. Now two regions, and the limit is global.
Candidate. A global count synced on every request puts a cross-region round trip, tens of milliseconds, inside every call, which breaks my 2 ms budget. So the regions exchange usage every 10 seconds, and each admits against the global count as of the last exchange plus its own requests since. For a customer at a steady 1,000 a minute, 16.7 a second, a region misses the other's last 10 seconds, so the overshoot is rate times the interval, about 167 requests, 17% of the minute. A flood is worse: both regions see the same remaining quota and each can spend all of it, up to 100% over. So between exchanges each region spends at most half the remaining quota; the total cannot pass the limit, at the cost of rejecting a one-region burst early until the next exchange. If a paid quota must be exact, I route that customer's checks to one home region and accept the added latency for them only.
Signal: the bound is computed for steady and for hostile traffic, the fix names its cost, and the exact path (home region) is offered where it matters.
Interviewer. What would you alert on, and what breaks first at ten times the load?
Candidate. Alert on limiter latency p99 above 2 ms, the fail-open rate, and the rejection rate per plan; a sudden jump in rejections is either an attack or my bug, and I want to know which. At 400,000 checks a second the store's single-thread throughput is the first limit; I shard by key hash, 15 shards at two thirds busy, and the local token leases cut calls by about a hundred for heavy keys.
- What I gave up. Exactness for speed: the sliding counter misjudges uneven traffic and leased tokens sit unused on gateways, so a limit holds to within a few percent, not to the request.
- Revisit when. A contract makes a limit a hard cap (the home-region path), burst complaints reach support (the token bucket), or checks pass 400,000 a second (re-shard and re-measure).
- Signal that tells me. Disputed rejections at the minute boundary, the fail-open share above its baseline, limiter p99 near 2 ms.
The four algorithms, and the race run
Table: Algorithm, State per key, Rule, Fails when (4 rows)
| Algorithm | State per key | Rule | Fails when |
|---|---|---|---|
| Fixed window | One count, the window in the key name | Allow while count < limit; each window starts at zero | A burst at a boundary passes twice the limit (Figure 8.4) |
| Sliding window log | A timestamp per allowed request: 400 MB at 1,000 a minute for 50,000 keys | Drop stamps older than the window; allow while fewer than the limit remain | Memory and work grow with the limit |
| Sliding window counter | Two 8-byte counts, this window's and the last: 0.8 MB for 50,000 keys | Estimate = last x (1 - elapsed fraction) + current. Limit 100, last minute 80, 50 so far, halfway: 80 x 0.5 + 50 = 90, allow | Traffic in the last window was uneven |
| Token bucket | Tokens and the last refill time | tokens = min(capacity, tokens + rate x elapsed); take one if tokens >= 1. Capacity 10 at 5 a second: a burst of 10, then 5 a second | A full bucket lets a burst through, by design |
The race needs a gap between the read and the write, a store round trip; on one machine with no gap, get-then-set leaks almost nothing. The program gives eight gateways a limit of 100, puts a 50-microsecond pause where the round trip would be, and runs three checks.
Code, 55 lines
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.locks.LockSupport;
class RaceTest {
static final int LIMIT = 100, GATEWAYS = 8, TRIES = 100;
static final AtomicInteger count = new AtomicInteger();
static final AtomicInteger allowed = new AtomicInteger();
// Atomic: the store checks and adds in one step (a Lua script).
static synchronized void checkAtomic() {
if (count.get() < LIMIT) {
count.incrementAndGet();
allowed.incrementAndGet();
}
}
// GET, a store round trip (50 microseconds), then SET or INCR.
static void checkGap(boolean incr) {
int seen = count.get(); // GET
LockSupport.parkNanos(50_000); // the network gap
if (seen < LIMIT) {
if (incr) count.incrementAndGet(); // INCR: no update lost
else count.set(seen + 1); // SET: overwrites others
allowed.incrementAndGet();
}
}
static int run(Runnable check) throws InterruptedException {
count.set(0); allowed.set(0);
var start = new CountDownLatch(1);
var done = new CountDownLatch(GATEWAYS);
for (int g = 0; g < GATEWAYS; g++) {
Thread.ofPlatform().start(() -> {
try { start.await(); } catch (InterruptedException e) { return; }
for (int i = 0; i < TRIES; i++) check.run();
done.countDown();
});
}
start.countDown();
done.await();
return allowed.get();
}
public static void main(String[] args) throws Exception {
int atomic = run(RaceTest::checkAtomic);
int set = run(() -> checkGap(false));
int incr = run(() -> checkGap(true));
if (atomic != LIMIT) throw new AssertionError("atomic " + atomic);
if (set <= 2 * LIMIT) throw new AssertionError("set " + set);
if (incr <= LIMIT || incr > LIMIT + GATEWAYS - 1)
throw new AssertionError("incr " + incr);
System.out.println("atomic=" + atomic + " set=" + set + " incr=" + incr);
}
}Run with java --source 21 RaceTest.java on 4 Nov 2026, ten runs: atomic=100 every time; set 788 to 800 and incr 106 or 107, varying with thread timing. SET writes the count it read plus one over the others' increments, so the count stays under 100 and nearly every try passes. INCR loses no update, so only the reads in flight as the count nears 100 leak: at most 99 + 8 = 107.
Next stop: SDE-2 against SDE-3: where each answer can stop, 1 min
SDE-2 against SDE-3: where each answer can stop
For: SDE-1 (one line below the table), SDE-2 and SDE-3.
Table: Area, SDE-2: enough, SDE-3: adds (9 rows)
| Area | SDE-2: enough | SDE-3: adds |
|---|---|---|
| Scope | Core actions and an out-list agreed before any box | Targets as numbers and the never-list, said first |
| Numbers | Peak rates from the exact day, each ending in "so" | Nodes from a per-node budget; bytes, bandwidth and memory; the one number that decides the design |
| Design | A working design with one read and one write traced | Alternatives named and one chosen with its cost |
| Consistency | Says where a stale read can happen and fixes it (read the primary before a 404) | States the promise as a model and its price: read-your-writes for creates, linearizable for this key |
| Failure | Two component failures, each with what the user sees | Also a half-completed step and overload, with a number on each; region loss as RTO and RPO |
| Depth | One trade-off defended, such as 301 against 302 | Two deep dives chosen by risk |
| Abuse | Rate-limit creates | The economics: unguessable 7-character codes and 6.0 billion links mean 0.17% of guesses hit, so rate-limit misses |
| Operate | Two metrics to watch | The full Operate-it block with a cost line |
| Asked about something left out | A sentence of each is enough | Raised before being asked |
SDE-1: enough. The SDE-2 column for scope, numbers and design, plus one failure with what the user sees; no deep dive is expected in a light design question (author's estimate, not measured).
Next stop: What changes at Staff, 3 min
What changes at Staff
For: Staff and Principal; SDE-3 readers who want the next bar. Lesson 12 covers the rest of a Staff loop.
Context. In the author's reading, Staff evidence is what the candidate raises before being asked: the two hard parts, the failure classes with numbers, the cost, the migration and the owner.
Table: Design, Staff: what is raised unprompted (8 rows)
| Design | Staff: what is raised unprompted |
|---|---|
| URL shortener | Scopes the round to two hard parts, the redirect contract and the delete race; puts a cost on cacheability; plans link lifecycle (expiry sweep, cached-but-deleted links, legal takedown); says who owns the ID service. |
| News feed | Names the two hard parts, fan-out cost under skew and read-your-writes across regions; plans moving a user between push and pull without loss; falls back to newest-first when ranking is down; puts the fan-out queue on a cost line and names its owner. |
| Payments ledger | Raises data residency (RBI/2017-18/153 requires that "the entire data relating to payment systems operated by them are stored in a system only in India"; ask your own counsel how it applies to you); names what is out (card data, FX, tax) and who owns it; treats reconciliation as a product with break rates and an owner for manual fixes; plans a stored-to-derived balance migration with dual writes and a verifier, not a big-bang cut; puts cold storage for 70.6 TB over seven years on a cost line. |
| Chat | Names the connection tier and ordering through an owner failover as the hard parts; end-to-end encryption (author's reading: the server still sequences and stores encrypted bodies but cannot search them, and group fan-out is per device key); a home region per conversation so one writer stays one writer; the 360 TB retention line, push cost and spam abuse, each with an owner. |
| Ride matching | Greedy versus batch matching and its waiting cost; a degradation plan (wider radius, longer offers, a queue for riders); the location trail, 2.16 TB a day raw or 432 GB keeping every fifth point, on a cost line; location history as personal data: the DPDP Act, 2023 (section 8(7)) says to erase it once "the specified purpose is no longer being served", so the trail's 30 days is a decision with an owner, checked with counsel; regional shards and who is paged. |
| Rate limiter | Limits as owned, versioned config rolled out like code; a published fail-open policy, so product teams know what a limiter outage means; the exact home-region path only for paid hard caps, on a cost line; a new algorithm on 1% of gateways with the old one counting in shadow. |
| Notifications | Worked below as the Staff opening. |
| Flash sale | Fixes the words, not the promise (the cart says a unit is held only at checkout); levers as config during the sale, never a first deploy that day; names who owns go or no-go and the refund queue; puts the CDN requests on the cost line. |
Under every cell: the three failure classes (a component dies, a step half-completes, overload), each with a number; a cost on each decision; fewer boxes, more depth in the risky ones. So a Staff answer is shorter than a Senior one.
INVENTED example · a Staff opening on a product prompt, condensed 1 min
Interviewer. Design a notification system for our company.
Candidate. One question first: do product teams send their own notifications today?
Interviewer. Yes. Four teams, four senders.
Candidate. Then this is a platform and a migration, not a queue. I will spend the round on the two parts that hurt users, the OTP lane inside 5 seconds and opt-out checked at send time, and give templates a sentence. The bill: Rs 3 crore of SMS against Rs 14.48 lakh of servers makes "push first" a product rule, so I would ask the product head to own it. Migration: one team at a time behind a flag for 30 days, compared with its old sender, the OTP team last because a regression there locks users out. The platform team owns the lanes, the provider contracts and the pager; each product team owns its templates and its daily cap. The contract between them is an SLO, OTP p99 under 5 seconds at 99.9%, and a send quota per team; a team over quota is shed in its own lane, so its campaign never delays another team's sends.
Signal: the old senders, the bill, the migration order and the owners come before any box, and the round is scoped to two hard parts.
Next stop: A Staff deep dive: the ledger in two regions, 1 min
A Staff deep dive: the ledger in two regions
For: Staff, Principal.
INVENTED example · a Staff deep dive on Design 3, condensed 1 min
Interviewer. The ledger must survive losing a region. Do both regions take writes?
Candidate. Both serve traffic, but each account has one home region that writes it; both are in India (RBI/2017-18/153). Not last-writer-wins or CRDTs: two regions each approving Rs 800 on a Rs 1,000 balance is a double spend, so a balance that must not go negative needs one writer. A versioned directory maps account to home; each home commits synchronously to a standby in the other region, so RPO is zero, at about 16 debits a second for a hot account. Failover takes 15 minutes, fenced by an epoch the storage checks, so a cut-off old home commits nothing.
Interviewer. And a transfer between accounts homed apart?
Candidate. A saga through a clearing account per region pair: debit at A's home into clearing, then credit at B's home out of it, each step idempotent by transfer id. At an assumed 30% cross-region, 720 a second at peak, I alert on the clearing account's oldest unmatched entry: a stuck one is money in limbo. Moving an account's home: freeze its debits, drain its outbox, bump the directory version, unfreeze, a few accounts at a time.
Candidate. Cost: the same 12 nodes across two regions, plus about 2,000 GB a month of replication egress (1 KB a transfer), about Rs 13,000. The payments platform team owns the directory and the clearing alert; finance owns the breaks.
Signal: the conflict rule follows from the invariant; failover, the cross-region path and the cost each have a number, and owners are named unasked.
Next stop: A Staff cost prompt: the bill grew 60%, 2 min
A Staff cost prompt: the bill grew 60%
For: Staff, SDE-3.
Context. Lesson 12 names four shapes of Staff situation prompt. This is the cost shape, priced from the Operate-it list.
INVENTED example · a Staff cost prompt, condensed 2 min
Interviewer. Our cloud bill grew 60% in a year. You are the Staff engineer. What do you do?
Candidate. Three questions before I cut anything. Did the business grow, is the bill tagged by service, and is there a target?
Interviewer. Orders went from 40 lakh to 50 lakh a month, the bill from Rs 50 lakh to Rs 80 lakh. Most of it is tagged.
Candidate. Unit cost first: Rs 1.25 an order became Rs 1.60, up 28%, so growth is not the whole story. Sorted by change, three lines stand out. "Out for delivery" now goes by SMS on 60% of orders: 30 lakh messages at Rs 0.4, Rs 12 lakh. A release put a per-user query string in image URLs, so the CDN hit ratio fell from 95% to 85%; on 1,250 TB a month the origin sends 187.5 TB, not 62.5, so 125 TB more at Rs 6.4 a GB, Rs 8 lakh. Debug logs, 1 TB a day, are kept 90 days instead of 14: 76 TB more on block storage at Rs 8 a GB, about Rs 6 lakh.
Signal: a unit cost before any cut, and each driver priced from a volume and a rate.
Candidate. Fixes, ranked by saving over effort. SMS first: push, with SMS only for the one user in six with push off (assumed), saves Rs 10 lakh for a day of config. That rule is the product head's, so I take them the number. Logs second: 14 days again saves about Rs 6 lakh for two days' work. Egress third: the query string out of the cache key saves Rs 8 lakh for a week and an app release. Together Rs 24 lakh: Rs 56 lakh a month, Rs 1.12 an order, below last year's.
Interviewer. Finance wants 30% off this quarter.
Candidate. Rs 24 lakh is 30% of Rs 80 lakh, with no cut to capacity. Two things I would not do: buy a one-year commitment before the fixes land, which locks in today's waste; or drop the ledger's synchronous standby, which buys an RPO of zero, so I show finance its price and keep it.
Interviewer. How does it stay fixed?
Candidate. Guardrails with owners. Cost per order by service on one dashboard, with a ticket to the owning team when its line grows 10% faster than orders in a month; the platform team owns the dashboard, each team its line. Defaults in shared code: 14-day retention in the logging library, a ticket when the CDN hit ratio drops below 92%, and a channel list per template that only the product head can widen. And a cost line in every design review.
Signal: the biggest lever goes to the person who owns the decision, what is kept is priced, and each guardrail has an owner, so the fix outlives the project.
Next stop: How a system design round is examined, 1 min
How a system design round is examined
For: SDE-1 and up.
Context. Interviewers commonly keep timestamped notes of what you asked, the numbers, what you drew and each decision with its reason, and a hiring committee reads those notes (the author's reading). In the author's estimate (not measured), at SDE-2 the design weight often falls on a machine-coding round (lesson 7), and from SDE-3 upward the high-level round weighs heavily on the level offered. Ask the recruiter which design rounds you will have.
Table: Minutes, Step, What goes in the notes (6 rows)
| Minutes | Step | What goes in the notes |
|---|---|---|
| 0 to 5 | Clarify | Core actions as verbs, an out-list the interviewer agreed, targets as numbers. |
| 5 to 10 | Estimate | Peak rates and bytes, each ending in "so". |
| 10 to 15 | API and data | Two to four calls; entities, the partition key. |
| 15 to 28 | Design | The simplest design; one write and one read traced; the bottleneck your numbers found. |
| 28 to 40 | Deep dives | Two areas chosen by risk: options, a decision, its cost, how it fails. |
| 40 to 45 | Wrap up | Decisions you would defend, the main risk, what breaks first at ten times. |
A real round loses five to nine minutes to introductions and your questions (the author's estimate, not measured, like the minute marks). When behind, cut the API detail first, then fold the data model into labels on the diagram, then shorten the estimate. Never cut the deep dives to zero; they carry the depth evidence.
Next stop: A rubric to score your own answer, 1 min
A rubric to score your own answer
After a practice run, score each row yourself from a recording.
Table: Area, A pass looks like, Met (8 rows)
| Area | A pass looks like | Met |
|---|---|---|
| Scope | Agreed in-list and out-list before any box; targets stated as numbers. | |
| Numbers | Rates computed from stated assumptions with the exact day; every estimate ends in "so". | |
| Blocks | Every box justified by a number or a requirement; none drawn "just in case". | |
| Mechanism | You can say why this key, this cache pattern, this quorum, in one sentence each. | |
| Failure | Component death, half-completed step and overload each named, with the user-visible effect. | |
| Depth | Two deep dives on the riskiest parts, with a decision and its cost. | |
| Staff signals | Cost, migration, owner and the out-list raised before prompted. | |
| Delivery | Decisions summarised aloud; recovered cleanly when redirected. |
Next stop: Unseen prompts: a one-page method, 3 min
Unseen prompts: a one-page method
Context. Your prompt may be on no list. The method, the author's, starts from the eight you know.
- Say "it is X, except Y" in the first five minutes. Name the nearest design aloud and the one difference that matters. "An exchange is chat's ordered log, except that the order of two messages decides who gets money."
- Write the never-list. One or two things that must never happen. The hard part is almost always one of four hot things: a hot row (one counter many people want, Design 7), a hot key or partition (one celebrity, one symbol, Design 2), a hot connection tier (millions of open sockets, Design 4), or a hot bill (egress, SMS, model calls; Design 6's SMS line).
- Derive the numbers; do not recall them. Put the interviewer's figures into the three formulas from the estimation section, and say the window: 86,400 seconds for a day, but a trading session or a sale is shorter.
- Reuse boxes aloud, and spend the time on the new part. "The edge, a gateway, a queue and an outbox, as in Design 3" takes ten seconds and earns the minutes for the one part that is new.
- When you stall, say so and go back to the never-list. "Let me step back. What must never happen is X. Design N keeps the same promise with Y. Is Y fast enough? Here is the number." The simple correct version first, then the improvement.
- When a requirement changes, restate it and change one thing. Name the one earlier decision it breaks, change only that, recompute the one number it moves, and say what the change costs. Do not redraw the board.
Six prompts outside the eight, each taken through steps 1 and 2 (the author's selection; numbers invented):
Table: Prompt: it is X, except Y, Two hard parts, The key choice (6 rows)
| Prompt: it is X, except Y | Two hard parts | The key choice |
|---|---|---|
| Food delivery: ride matching, except three parties and a food-ready time | One partner claimed by two orders; one order's state across restaurant, partner and customer | Claim the partner by compare-and-set with a lease, offered at food-ready time minus travel time; the order moves state only by a conditional update on its version |
| Seat booking: the flash sale, except every unit is a named seat | Two buyers on one seat; a group that needs all its seats or none | One row per seat per show; hold them all with UPDATE ... WHERE state = 'FREE' in one transaction and check the count; an expiring hold the payment saga confirms |
| Typeahead: a cache read, except the key is a prefix and each keystroke is a request | A p99 inside a keystroke, 50 ms assumed; trending queries that are new | The top 10 per prefix precomputed from query logs into an in-memory table rebuilt hourly; a client debounce; new counts merged at the next build |
| Live streaming of a cricket final: a CDN, except the bytes are live | 10 million viewers at 3 Mbit/s is 30 Tbit/s; delay against stalls | Ingest, transcode a ladder of bitrates, 2 to 6 s segments through a CDN that coalesces requests; shorter segments cut delay and add stalls; egress is the hot bill |
| Product search: typeahead, except every word is indexed and stock changes | Stale stock in results; filters inside 200 ms p99 (assumed) | An inverted index, word to product ids, fed by change capture from the database and never written directly; rebuilt and swapped by alias; stock rechecked at add-to-cart, so staleness costs a click, not an oversell |
| A key-value store: Design 1's shards, except you build replication and repair | Writes while a replica is down; replicas that drift | Three copies on a hash ring, R = W = 2; a down node's writes go to the next node with a hint and are handed back (hinted handoff); background Merkle-tree comparison repairs drift; concurrent writes kept by vector clock, or last-writer-wins said aloud (the Dynamo paper) |
Next stop: The pushback bank: five follow-ups for each know-cold design, 5 min
The pushback bank: five follow-ups for each know-cold design
Context. The follow-ups are the author's selection, with invented numbers. Cover the right-hand column, read a follow-up aloud, and answer before you look.
Table: Design, The follow-up, One-line answer (40 rows)
| Design | The follow-up | One-line answer |
|---|---|---|
| 1 URL shortener | 301 or 302? | 302 with a short max-age while clicks may matter; 301 is the saving if we stop counting. |
| 1 URL shortener | Sequential codes can be walked. | The counter goes through a keyed permutation, so codes look random; 0.17% of guesses hit, so rate-limit misses. |
| 1 URL shortener | A new link 404s on its first click. | Return the code after the primary commits; on a replica miss, ask the primary before a 404. |
| 1 URL shortener | One link goes viral. | The cache takes it; an in-process cache on each app server keeps it off one cache node. |
| 1 URL shortener | How many shards? | 11,574 a second over an assumed 2,000 a shard is 5.8, so six, each with two replicas (18 nodes); the six carry the peak if the cache dies. I would load-test. |
| Rate limiter | Why not a fixed window? | Twice the limit at a boundary (Figure 8.4). |
| Rate limiter | Two gateways read the count together. | Check, refill and decrement in one atomic script; in RaceTest get-then-set let about 800 through a limit of 100 (lost updates), get-then-INCR up to 107. |
| Rate limiter | The store dies. | Neither fail closed nor wide open: a local limit of the limit over the gateways, about 1 s of counters lost. |
| Rate limiter | One key sends 100,000 a second. | Sharding cannot split a key: lease 100 tokens at a time to each gateway, so the store sees one call in 100. At 5,000 a second, change nothing. |
| Rate limiter | The limit is global across two regions. | Exchange usage every 10 s; steady traffic overshoots 167 (17%), a flood up to 100%, so each region spends at most half what remains; exact caps go to a home region. |
| 6 Notifications | Does a campaign delay an OTP? | No: separate lanes, workers and provider quota; the campaign takes 83 minutes, the OTP seconds. |
| 6 Notifications | A user opts out after the campaign is queued. | Preferences are checked at send time, not at enqueue time. |
| 6 Notifications | The SMS provider fails. | Fail over to the second; an OTP may repeat, a payment alert checks the status table first. |
| 6 Notifications | Why is the bill so large? | SMS: 1% of 250 million a day is Rs 3 crore a month, so push first, SMS only for OTP and fraud. |
| 6 Notifications | Send to a million users from one call. | A job walks the segment in pages with a cursor, so a crash resumes where it stopped. |
| 2 News feed | Why not push to everyone? | One 20-million-follower post is 20 million writes, 173 seconds of the whole site's mean push load of 115,741 a second. |
| 2 News feed | Where is the threshold? | 10,000 followers, assumed; lower makes reads merge more lists, higher floods the queue. |
| 2 News feed | My own post is missing. | The author's own list is written synchronously; "own post missing" is a metric. |
| 2 News feed | A huge post lags the queue. | Nothing is lost; lower the threshold for the duration and add workers by partition. |
| 2 News feed | Ranking next quarter? | A separate call with a 50 ms timeout that reorders candidates, falling back to newest first; version the cursor. |
| 4 Chat | How is order kept? | One writer per conversation assigns the next number; client clocks are never used. |
| 4 Chat | The client sees 40, then 42. | It asks for the gap with sync. |
| 4 Chat | A crash after the append, before delivery. | The log has it; the recipient syncs from its last number on reconnect. |
| 4 Chat | A gateway with 67,000 sockets dies. | Clients reconnect with jitter, the registry moves them, and the log loses nothing. |
| 4 Chat | Why not a flag per message for read? | A cursor per member per conversation, one number, instead of a flag on every message. |
| 3 Payments ledger | The client retries. | Key, request hash and entries in one commit; the same key with a new amount is an error. |
| 3 Payments ledger | The bank times out. Did the money go? | Unknown, so pending; ask the bank by my transfer id, never show a false failed. |
| 3 Payments ledger | Why not two-phase commit with the bank? | The bank is not a participant and 2PC blocks on a lost coordinator; a saga with compensation. |
| 3 Payments ledger | Two payments of 400 on a balance of 500. | Subtract 400 where the balance is at least 400, so the second fails; a hot account (16 debits a second at a 62 ms commit) splits into sub-accounts. |
| 3 Payments ledger | How do you shard? | By account; cross-shard is a saga through a clearing account, and I alert on its oldest balance. |
| 7 Flash sale | Why not reserve at add-to-cart? | Most carts never buy, so window-shoppers would hold the sale. |
| 7 Flash sale | Why not a Redis counter? | Its replication is asynchronous, so a failover can lose a decrement; Redis says no, the database says yes. |
| 7 Flash sale | The stock row is hot. | About 500 holds a second a row at 2 ms; 20 rows of 50 give 10,000. |
| 7 Flash sale | Payment lands after the hold expired. | Confirm needs RESERVED and an unexpired hold; otherwise refund automatically. |
| 7 Flash sale | Ten times the crowd? | Nothing inside changes; the edge absorbs it and the gate admits by stock, not by people. |
| 5 Ride matching | Why not store every update? | 2.16 TB a day; keep the latest 100 MB in memory and sample a trail to cold storage. |
| 5 Ride matching | Two matchers pick one driver. | Compare-and-set by driver from available to offered, with a lease. |
| 5 Ride matching | The rider is at a cell edge. | Search the rider's cell and its neighbours, about nine cells. |
| 5 Ride matching | The driver never answers. | The lease ends after 10 seconds, the driver returns to the pool, the offer moves on. |
| 5 Ride matching | A driver's phone goes quiet. | Three missed updates, 12 seconds, remove the driver from the index. |
Practice with an AI: the prompt kit in the appendix has 3 cards for this lesson.
Sources
20 sources
- Giuseppe DeCandia et al., "Dynamo: Amazon's Highly Available Key-Value Store", SOSP 2007, read 31 Oct 2026.
- Diego Ongaro and John Ousterhout, "In Search of an Understandable Consensus Algorithm", raft.github.io/raft.pdf, read 31 Oct 2026. Terms and majority commit.
- Martin Kleppmann, "How to do distributed locking", martin.kleppmann.com, 8 Feb 2016, read 1 Nov 2026. Fencing tokens.
- Daniel Abadi, "Consistency Tradeoffs in Modern Distributed Database System Design", IEEE Computer, Feb 2012, read 1 Nov 2026. PACELC.
- Seth Gilbert and Nancy Lynch, "Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services", 2002, opened 1 Nov 2026; text not extractable.
- Jim Gray and Leslie Lamport, "Consensus on Transaction Commit", 2004, arXiv cs/0408036, opened 2 Nov 2026.
- Hector Garcia-Molina and Kenneth Salem, "Sagas", SIGMOD 1987, read 1 Nov 2026.
- PostgreSQL 18 documentation, 13.2 "Transaction Isolation", postgresql.org/docs/18/transaction-iso.html, read 1 Nov 2026.
- Rajesh Nishtala et al., "Scaling Memcache at Facebook", NSDI 2013, read 31 Oct 2026. Look-aside cache, leases.
- David Karger et al., "Consistent Hashing and Random Trees", STOC 1997, abstract read 31 Oct 2026.
- Google SRE book, "Load Balancing in the Datacenter", sre.google, re-read 2 Nov 2026.
- Google SRE book, "Handling Overload" and "Addressing Cascading Failures", sre.google, read 4 Nov 2026.
- Google SRE Workbook, "Canarying Releases", sre.google/workbook/canarying-releases/, re-read 2 Nov 2026.
- Elastic, "Near real-time search" (older guide), read 31 Oct 2026.
- Stripe, "Idempotent requests", docs.stripe.com/api/idempotent_requests, read 31 Oct 2026.
- Redis, Lua scripting introduction, redis.io/docs, read 2 Nov 2026. Scripts run atomically.
- Reserve Bank of India, "Storage of Payment System Data", RBI/2017-18/153, 6 Apr 2018, rbi.org.in, read 2 Nov 2026.
- OWASP Top 10, 2025 edition, top10.owasp.org/2025, read 1 Nov 2026.
- Digital Personal Data Protection Act, 2023, section 8(7), as reproduced at dpdpa.in/chapter2.htm, read 4 Nov 2026 (a mirror, not the Gazette).
- Programs run with
java --source 21on OpenJDK 25, 4 Nov 2026, each withAssertionErrorchecks: RaceTest, printed with the rate limiter (ten runs, three checks). All passed. Every volume, price and count in the designs is invented.