Staff and Principal loops

The prompt. Interviewer: "Kosh is a lending company with 34 engineering teams. Every team schedules background work its own way. We had five incidents from it in six months. You are the Staff engineer in the platform org. What would you do in the next year?" Kosh, its numbers and its people are invented.

Minutes 0 to 5: clarify, and ask for the numbers you need. "What kinds of work: cron-style jobs, or data pipelines too? What were the five incidents? Is there an owner today? What growth do you expect in job runs?" Interviewer: "Cron-style jobs; pipelines have their own tool. Two incidents were jobs that ran twice after a VM failed over; one sent 41,000 duplicate EMI reminder messages. Three were jobs that silently stopped; the worst, a mandate retry file, was missed for three days. No owner. Assume runs grow 40% a year."

Minutes 5 to 15: frame and measure before choosing. "Before any choice I would spend two weeks on an inventory, because I cannot plan a migration I cannot count. Say it finds 2,400 jobs on 46 VMs, averaging 30 runs a day each, so 72,000 runs a day. I would tier them by what a failure costs: Tier 1 moves money or meets a regulator's deadline, say 180 jobs; Tier 2 is customer-facing, 620; Tier 3 is internal, 1,600. There are two failure classes, and they need different fixes. Silent stops need detection. Double runs need a rule that a scheduled run happens at most once, enforced by the scheduler, which no monitoring gives you."

Minutes 15 to 30: four options, priced over three years. "I will put people at an invented loaded cost of Rs 60 lakh per engineer-year and a managed service at an invented Rs 0.25 per run. Self-hosted infrastructure I assume grows at half the rate of runs, because capacity is added in steps."

"A is cheapest and fixes only the silent stops; it leaves double runs on every failover, so it cannot be the answer, but its alert wrapper is a two-week win I would ship first whatever we choose. B costs the most and a scheduler is not Kosh's product; I would build only if no option met a hard constraint. C and D are within 7% of each other at 40% growth, and D is cheaper below about 26% growth a year. So cost does not decide this. C keeps job payloads inside our own cloud account, which a lending company should check with its counsel before sending them to a vendor, and it does not grow its bill with every run. People covers build and on-call: C is 5.0 engineer-years over three, about 1.7 a year; D is 2.5, about 0.8, so half an engineer on call. I choose C, if the platform team can staff 1.5 engineers on call. If it cannot, D."

"One cost is the same for B, C and D and is easy to hide: moving 2,400 jobs at about two hours each is 4,800 hours, close to 2.7 engineer-years at about 1,800 working hours a year (author's estimate, not measured), paid by 34 teams, not by me. It has to be on their roadmaps before I promise a date."

Minutes 30 to 40: the migration, with a way back. "Month one, the inventory and the alert wrapper. From month two, no new cron jobs on the old VMs, announced with the date. Months one to three, the engine and a pilot of 60 Tier 3 jobs from three volunteer teams. A job with side effects cannot run on both systems at once, so for two weeks the new engine fires in dry-run mode and logs 'would run'; we compare its log with cron's, then flip a flag per job. The old cron line is disabled, not deleted, for 30 days: rollback is one flag. Months three to six, Tier 3 by the teams themselves with a tool that converts crontab lines. Months six to nine, Tier 2. Months nine to twelve, Tier 1, one job at a time, with the owning team, in business hours. Every run carries a key made of the job and its scheduled time, and the engine refuses a second trigger with the same key; that is what ends double runs. Month twelve, the 46 VMs go off, with an exception list a director signs."

Minutes 40 to 45: risks, owners, measures. "Biggest risk: teams cannot fit 4,800 hours; the conversion tool and doing Tier 1 with them are the mitigation. Second: one engine outage now stops every job; two zones, a runbook, and a manual trigger for Tier 1. The platform team owns the engine; each team owns its jobs and their alerts. I would report four numbers monthly: double runs, missed-run alerts, percentage migrated per tier, and VMs left."

Gave up / revisit when: about Rs 43 lakh over three years if growth stays flat, and 1.5 engineers on call; revisit if growth runs under 26% a year for two quarters, or the on-call cannot be staffed, and move to D.

Annotation: what moved the interviewer's notes up was the inventory before the choice, two failure classes with two different fixes, "do less" priced as an option, the admission that cost does not decide between C and D, and the migration hours named as other teams' cost. What would have moved them down: drawing a scheduler's internals for twenty minutes, which is the Senior answer to a different question.

Director (invented): Tell me about something you changed across teams you did not lead.

Farhan Qureshi (invented): I retired a customer database that seven teams wrote to directly. It had no owner, and in one year three schema changes by one team had broken another team in production. I counted first: the write logs showed seven writing services, and I asked the teams who read it. I wrote one page with three options: a review process for schema changes, which cost nothing but relied on people remembering; a big-bang move to a new service, which needed three quarters of feature freeze; and a staged one, an owner team with an API, reads moved first, then writes team by team, each team's direct credentials revoked on a date published three months ahead. I took it to each tech lead one to one before any group meeting. The collections team refused, because their quarter-end close fell on their date. I moved their date by six weeks and lent them two engineers for three weeks. My director chose the staged plan. All seven teams were off direct writes in eleven months, schema-change outages went from three that year to none the next, and the platform org used the same dated-notice policy for two later retirements. What I got wrong: I took the teams' word on readers instead of reading the logs for them too, missed fourteen reporting jobs until month six, and slipped two months. I now start every retirement from access logs.

Annotation: reach (seven teams, a policy reused twice), evidence before persuasion, the owner met before the meeting, a hold-out handled by moving the plan rather than escalating, a twelve-month check (zero outages, policy in use without him), and a miss about method, not luck.

Gave up / revisit when: two engineers for three weeks and six weeks of schedule for one hold-out; revisit if a team needs direct access for a regulator, with an exception a director signs.

VP (invented): Tell me about a time you changed a decision above your level.

Lakshmi Rao (invented): Our CTO's staff had approved an eighteen-month rewrite of all nine mobile backend services into a new stack, because the app's slowest screens took 1.8 seconds at p95. I owned none of the nine. I asked the rewrite's sponsor, Arvind, whether I could measure first, and he agreed. Tracing a week of traffic showed that about 70% of the p95 time on the slow screens came from three endpoints, each calling the pricing service once per item. I wrote a counter-proposal with Arvind as co-author, not against him: fix the three call patterns in the old stack in one quarter, rewrite two services that had a real reason to move, and set a policy that new services use the new stack and old ones move only when a team is already changing them. I rejected stopping the rewrite outright, because Arvind's other reason was real: the old stack was getting harder to hire for. We presented together. The CTO took the narrower plan. P95 on those screens fell from 1.8 seconds to 650 milliseconds in two quarters, and about twenty engineers planned for the rewrite went back to product work. Two years on, the policy still decides which stack a new service uses. What I got wrong: my first draft treated hiring as a minor point; Arvind was right, and the policy exists because he pushed.

Annotation: company scope (nine services, every new service after), a decision above her changed through its sponsor, measurement before argument, the other side's best reason kept, a number a year on, and a miss that credits someone else.

Gave up / revisit when: a cleaner single stack in eighteen months; revisit when fewer than three services remain on the old stack, and finish the move.

VP (invented, at a lender): What is the hardest technical problem in our space?

Lakshmi: From outside, month-end collections: mandate debits bunch into three days, and a failed debit is a missed EMI. First I would want the split of failures between our side and the banks', because it decides whether the fix is code or routing.

VP: Next year I can fund one big bet. What would you stop to pay for it?

Lakshmi: Something with a high run cost and few users, say a second reporting stack: perhaps six engineers freed, its users given a dated notice. I would bring that stop, priced, with the bet's kill test.

VP: And if the bet fails?

Lakshmi: You hear it from me on the kill-test date, with the cost and what we keep.

Annotation: the business before the technology, another group's work stopped and priced, and the failure plan offered unasked.

Asking for the level with evidence. Candidate: "Thank you. Before we discuss numbers, can I see the Staff definition you used? I think the panel did not see the cross-org part of my work. I can send a one-page doc on the database retirement I led across seven teams by Thursday, if the panel would read it." Recruiter: "I will ask the hiring manager." Annotation: evidence, a date, a process; no number yet.

If the level stays at Senior. Candidate: "I understand the level is Senior. I would consider it with two things in writing: a review for Staff at nine months against the scope we discussed, and a stock grant at the top of the Senior band, since the role you described includes the platform migration. Can both go in the offer email?" Annotation: the level conceded, a dated review asked for, and the grant tied to the scope they described. Whether either is granted depends on the company; the writing makes it checkable. If neither moves and you hold a Staff offer elsewhere, that written offer is the only real lever (lesson 13).

Nisha Varghese (INVENTED, Principal): "Checkout failed on 2.1% of attempts across our four apps, and each app fixed it alone. I argued for one shared checkout, and the CTO funded fourteen engineers for three quarters on my page. The page held a kill test: by month six, two apps live and failures under 1.5%. At month six one app was live, at 2.0%. A week of payment-gateway logs, which I should have read before asking for anyone, showed about two thirds of failures were bank-side timeouts no checkout could fix. I told the CTO the bet had failed and asked to stop. It cost about 84 engineer-months. We kept the retry module and moved eight engineers to routing payments away from slow banks: 1.4% two quarters later. Since then any proposal above 20 engineer-months reaches review with its kill test and the data that could disprove it, read first. Three have been cut there, one of them mine."

Page: who funded it, a kill test set before the money, the stop called by its author, the cost, the salvage, a rule that outlived the bet. At Staff, tell the same shape inside one organisation.

VP: Who wanted to keep going?

Nisha: Two of the four app leads. The Android lead had six engineers mid-build and wanted one more quarter. I showed him the gateway split: with two thirds of failures bank-side, a perfect shared checkout reaches about 1.4% at best, for fourteen engineers. He agreed to stop if his retry module was kept, and it was. I had written the kill test; moving it after missing it would make every later one worthless.

VP: Why not read the logs first?

Nisha: I should have. The four apps' failure reports agreed, so I took them as the cause; nobody had joined them to the gateway's codes. That was two days of work I skipped because the case looked obvious, and it is why the rule reads the disproving data before review, not after funding.

VP: Would you make the bet again?

Nisha: On what I knew, yes; on what two days would have shown me, no.

Annotation: names and their reasons, not "the team"; the kill test held by its author; a miss owned without defence and turned into a rule. Weak: "everyone agreed", or blaming the banks.

VP (invented): Tell me about someone you grew to Staff.

Lakshmi: Devika (invented) was a senior on the pricing team, the best debugger I knew, whose cross-team decisions went up to a director because they would not write the page. After the rewrite review I handed them the three call patterns, which touched four teams, and asked for the proposal, not the code. I commented on two drafts, sat silent in the first review and stopped attending in month three. When the catalogue team refused, Devika priced the refusal instead of escalating, and catalogue moved. Fourteen months later they were Staff, and their batching rule is in the API guidelines. What I got wrong: in month two I rewrote a paragraph of their page, and they could not defend it. I comment now; I never edit.

VP: How much of that was you?

Lakshmi: The choice of problem and the month I stopped going. Their promotion case cites the review they ran alone in month four.

VP: What did it cost you?

Lakshmi: A slower quarter: their page took five weeks where mine would take one, and I told my director first.

Page: the gap as a behaviour, a problem handed over, the month you stopped attending, the promotion and a rule that outlived it, a miss.

The design handed over. The payments team wants to move its ledger from one PostgreSQL primary (350 transfers a second at peak, 35% busy, which fits lesson 8's budget of about 1,000 a node) to a key-value store sharded by transfer id, cut over in one weekend, for Rs 40 lakh a year more. The reason given: "we will need ten times the load".

What I ask. Where does the primary saturate, measured, not forecast? At 35% busy it saturates near 2.9 times today's load and crosses lesson 8's two-thirds-busy line (66%) at about 1.9 times. So the question is when 1.9 times arrives: at 40% growth a year, in about two years; if load doubles yearly, in under one. The move is needed; the date comes from growth, not from "ten times". How is a balance read when one account's entries sit on every shard? What does rollback look like on Monday morning if the new store disagrees with the old?

What I block, and what lifts it. The one-weekend cutover: money moves by dual writes, a verifier that compares both stores every night, and reads moved first behind a flag; I lift the block when the plan shows those steps and a rollback at each. The shard key: by transfer id, every balance scatters; shard by account and move money between shards through a clearing account with a saga, or show me why balance reads do not matter.

What I let go. The choice of store brand, the client library and the dashboards: each is cheap to change later and the team owns it. The growth forecast goes to finance as an open question with a date, because it sets the deadline.

How I close. "Approve with two conditions: a staged migration with a verifier, finished before the primary passes 66% busy, and an account shard key. Everything else is yours." Written down, with the conditions.

Signal: three questions that each change the plan, two blocks each with the condition that lifts it, a deadline from the numbers, and the rest handed back to its owner. The trap: reading "35% busy" as "years away". Utilisation gives the multiple (1 / 0.35, about 2.9); only growth gives the date.

The document. Kosh one level up: 34 teams in five organisations, 140 services, one cloud region. After a five-hour regional outage, the CTO's draft makes every service active-active across two regions within 18 months, about doubling the Rs 24 crore yearly cloud bill and taking two quarters from every team. Every number is invented.

Minutes 0 to 5: the decision. "What does this feed, by when, and who signs?" CTO: "The board's budget in March; the CFO co-signs."

Minutes 5 to 15: the numbers the page lacks. "What does an outage hour cost, and where? How often do we expect one?" CTO: "About Rs 60 lakh an hour, 90% of it from 12 services that move money. Assume five hours every two years." "Then the expected loss is 5 × 60 ÷ 2 = Rs 1.5 crore a year, against Rs 24 crore a year extra: sixteen times. It is insurance, and the question is how much to buy."

Minutes 15 to 30: tier it, and price each tier. "Tier 1, the 12 money services, is about 15% of the bill: a second copy costs about Rs 3.6 crore a year and covers 90% of the outage cost. I buy it anyway: a lost or doubled debit breaches the regulator's rules, so it is a duty, not an insurance bet. Tier 2, 40 customer-facing services, is 35%: Rs 8.4 crore a year to protect at most Rs 15 lakh. Tier 3, 88 internal services, gets backups restorable within a day, about Rs 0.5 crore. Year one is about Rs 4 crore, not 24; Tier 2 is the CFO's call, with its price."

Minutes 30 to 45: block, let go. "I block two things, each expensive to reverse. Last-writer-wins on Tier 1 loses or doubles a payment: give each account a home region for its writes, or run a synchronous standby; the block lifts when a failover drill loses and doubles nothing. And one date for all 140: Tier 1 moves a service at a time, done when its drill passes, and only the 12 owning teams pay hours. I let go of routing, tooling and runbooks: cheap to change, and each team's own."

Minutes 45 to 55: what stops, the rule, the kill test. "Stop active-active for Tiers 2 and 3, which frees most of the two quarters. Its price, said plainly: in a regional outage the app is down while the region is, debits and approved disbursals keep running, and internal tools return within a day; the five organisation heads sign that. The rule that outlives the plan: every new service declares its tier at design review, and no Tier 1 service launches without a passed drill. Kill test: by month six the ledger and disbursal fail over in under 30 minutes with nothing lost or doubled, or we stop and rethink."

Minutes 55 to 60: close in writing. "Fund Tier 1 and the backups, with no last-writer-wins on money and a drill per service; Tier 2 returns at month six." CTO: "The board asked for fully active-active." "Then show them cover per rupee by tier; all 140 is their call, with the price on it."

Signal: the missing frame (insurance against the expected loss), most of the plan stopped and priced, blocks only where reversal is expensive, a rule for every new service. Down-level trap: 30 careful minutes on the ledger's replication.

CTO (invented, at a lender): What would your first year look like?

Lakshmi: Which decision is waiting for this seat? CTO: Our four business lines run three payment stacks. Lakshmi: Then quarter one counts each stack's cost, incidents and users, and each line head sees their numbers first. Quarter two, one page: the survivor, what stops, a kill test, signed by the four heads and you. By month twelve one stack is off.

CTO: Your rewrite story is one organisation's mobile backend. Why is that Principal?

Lakshmi: The nine services were one organisation's; the policy was not. It now decides the stack for every new service in all five organisations, 31 so far, none of them mine.

Lakshmi (closing): What would make you doubt me? CTO: A quarter spent counting while three stacks break. Lakshmi: Fair. In month one the stack with the most incidents gets the fixes its team already wants; you would see its incidents fall before my page.

What the CTO files: the decision asked for first; reach as a rule five organisations use without her; the doubt answered with a dated fix.