Two-week programme · Day 13 of 14
You can't roll back a phone
Part 1Basics
On a server, a bad release can be rolled back in minutes. On phones, it can't. Once an update is installed it stays, and a fix must pass store review and then wait for users to update.
So the goal shifts to catching problems while few people have them, and being able to switch off a broken feature remotely. Principal engineers are often judged on exactly this kind of system.
Words you need today
- Staged rollout
- Releasing to 1% of users, then 10%, then everyone, with a check at each step.
- Crash-free sessions
- The share of app sessions with no crash, the headline health number.
- ANR
- Application Not Responding: Android's name for a frozen main thread.
- Telemetry
- Measurements the app sends back about how it's behaving.
- Feature flag
- A remote setting that turns a feature on or off without an app update.
- Kill switch
- A flag whose job is to turn something off in an emergency.
The story
Version 8.2 goes to 1% of users on Monday. On one manufacturer's phones, a new SDK starts during the background wake-up and crashes the process. Remember Day 3: nothing on screen shows it. Priya's phone just stops syncing, her outbox quietly grows, and she sees clocks instead of ticks.
Under the old setup, you'd learn this five days later from store reviews. Now phones report crash-free sessions and outbox depth by version and device. Within an hour the new version looks worse than the old one on those phones, so the rollout halts itself. A remote flag turns off the SDK for those who already updated.
This fix needed no new sync design. It needed telemetry, a staged rollout, a flag, and the organisation to run them.
Part 2Today's question
How do you keep all of this working as 150 engineers keep changing it?
By the end of this lesson, you should be able to:
- Design a staged rollout with gates
- Choose release signals, including domain health
- Describe how you'd lead change across teams
Part 3In detail
Step by step
- Ship to 1%. Version 8.2 goes out to a small group on Monday.
- Phones report health. Crashes and outbox depth get buffered on disk and uploaded in batches, as the outbox does.
- Compare new with old. Compare the same device types over the same hours, because a global average can hide a problem on one manufacturer's phones.
- Halt automatically. With enough sessions and a clearly worse result, the rollout stops itself.
- Switch it off. A kill switch disables the broken SDK for those who already updated.
- Fix forward. The corrected version goes through the same stages.
Fixes reach phones slowly
A server fix lands in minutes. A mobile fix waits for store review and for users to update, so detection has to happen while 1% of users have the bug. Old versions also live for years, which is why Day 2's migrations must handle skipped versions.
Telemetry is another outbox
Events buffer on disk and upload in batches, so an offline crash still arrives. The same durable-queue idea carries your release signals. Include domain health such as outbox depth, because a crash rate alone would miss a silent sync stall.
Flags need the same discipline
Assignment must be stable and identical across platforms. Log exposure when the user reaches the feature, and give every flag an owner and an expiry.
Principal work means moving people
Put build time, flaky tests, crash-free sessions and lead time on one dashboard everyone sees. Pick the worst, fix it with one or two teams, publish the result, and let others ask to join.
The code, explained
This is the real code, and our tests run it. Read each block, then the notes under it.
Deciding whether a rollout continues
This would have stopped version 8.2 at 1%. It compares like with like, and waits for enough data.
1data class Health(val sessions: Int, val crashFree: Double, val outboxP90: Int)2 3enum class Rollout { CONTINUE, WAIT, HALT }4 5/** Compare new with old on the same devices over the same hours. */6fun judge(new: Health, old: Health, minSessions: Int = 20_000): Rollout {7 if (new.sessions < minSessions) return Rollout.WAIT // too little data to judge8 val moreCrashes = new.crashFree < old.crashFree - 0.0029 val syncStalled = new.outboxP90 > old.outboxP90 * 2 // a stall shows no crash at all10 return if (moreCrashes || syncStalled) Rollout.HALT else Rollout.CONTINUE11}12 13private val STAGES = listOf(1, 10, 50, 100)14 15fun nextStage(percent: Int): Int = STAGES.firstOrNull { it > percent } ?: 100new.sessions < minSessions- Too little data proves nothing. Wait, rather than react to noise.
old.crashFree - 0.002- A small allowance, so normal variation doesn't halt every release.
new.outboxP90 > old.outboxP90 * 2- Catches a sync stall, which shows up in no crash report. Domain health needs its own signal.
Compiled and tested by npm run test:mobile.
Telemetry that survives being offline
It's an outbox for measurements, so a crash in a basement still gets reported.
1data class Event(val id: String, val name: String, val critical: Boolean)2 3/** An outbox for measurements. It keeps crashes when full, and drops routine events first. */4class TelemetryBuffer(private val maxEvents: Int = 500) {5 private val queue = ArrayDeque<Event>()6 val size: Int get() = queue.size7 8 fun record(e: Event) {9 if (queue.size >= maxEvents) {10 val victim = queue.firstOrNull { !it.critical } ?: queue.first()11 queue.remove(victim)12 }13 queue.addLast(e)14 }15 16 /** Removes a batch only after [upload] succeeds, so a failed upload loses nothing. */17 suspend fun flush(batch: Int = 50, upload: suspend (List<Event>) -> Boolean): Int {18 val events = queue.take(batch)19 if (events.isEmpty() || !upload(events)) return 020 repeat(events.size) { queue.removeFirst() }21 return events.size22 }23}24 25/** Server side. A resend must not inflate the crash rate. */26class EventIntake {27 private val seen = HashSet<String>()28 fun accept(batch: List<Event>): List<Event> = batch.filter { seen.add(it.id) }29}queue.firstOrNull { !it.critical }- When full, routine events go first. Crash reports are the last to be dropped.
if (events.isEmpty() || !upload(events))- Events leave the queue only after a successful upload.
class EventIntake- The server removes repeats, so a resend can't inflate the crash rate.
Compiled and tested by npm run test:mobile.
Stable variant assignment
The same user must get the same variant on any device, offline, with no server call.
1data class Variant(val name: String, val weight: Int)2 3data class Experiment(4 val key: String,5 val salt: String,6 val variants: List<Variant>,7 val rolloutPercent: Int = 100,8 val killed: Boolean = false,9)10 11/** Pure assignment: the same user and experiment always get the same variant, on any device. */12class FlagEvaluator(private val onExposure: (experiment: String, variant: String) -> Unit) {13 private val logged = HashSet<String>()14 15 fun variantFor(userId: String, exp: Experiment, default: String = "control"): String {16 if (exp.killed) return default // kill switch beats everything17 val bucket = bucketOf("${exp.salt}:$userId")18 if (bucket >= exp.rolloutPercent * 100) return default19 val total = exp.variants.sumOf { it.weight }20 var point = bucketOf("${exp.salt}:variant:$userId") % total21 val chosen = exp.variants.first { v -> (point < v.weight).also { if (!it) point -= v.weight } }.name22 // Log once per session, and only when the user actually reaches the feature.23 if (logged.add(exp.key)) onExposure(exp.key, chosen)24 return chosen25 }26}27 28/** FNV-1a into 0..9999: stable across platforms, unlike hashCode() between languages. */29fun bucketOf(s: String): Int {30 var h = 0x811C9DC5.toInt()31 for (b in s.encodeToByteArray()) {32 h = h xor (b.toInt() and 0xFF)33 h *= 1677761934 }35 return ((h.toLong() and 0xFFFFFFFFL) % 10_000).toInt()36}if (exp.killed) return default- The kill switch wins over everything.
bucketOf("${exp.salt}:$userId")- A new salt gives each experiment an independent split.
logged.add(exp.key)- Logs exposure once, when the feature is reached, so unexposed users don't dilute results.
FNV-1a- hashCode differs between languages. A hash written out by hand gives Android and iOS the same answer.
Compiled and tested by npm run test:mobile.
What can go wrong
- A failure shows nothing on screen, like a silent sync stall
- Measure domain health, such as outbox depth, and not only crashes.
- Too little data at 1%
- Set a minimum session count before judging.
- Old flags pile up
- Give each flag an owner and an expiry date.
- Teams resist the process
- Show better numbers from one team first, and make the good path the easy one.
Decisions to defend
- How do you ship to 150 engineers' users? Weekly train, staged, with automatic gates, not monthly big release. Small batches make each bad change cheaper to find and stop.
Part 4Questions and answers
Answer out loud first, then open each one.
A kill switch and a rollout halt: why both?
Twenty teams resist a new release process. What do you do?
Part 5Going further
The three questions, for today
- Where does it live?
- Telemetry buffers on disk, as the outbox does.
- Who runs the next step?
- An upload job on the phone, and an automatic gate on the server.
- What if it runs twice?
- Event IDs remove duplicates, so a repeat doesn't inflate the crash rate.
Practise
- Mock45 minMobile platform for 150 engineers: builds, releases, flags
No code in this round. Bring numbers and a plan for moving 20 teams.
Optional reference pages
You don’t need these for the lesson. They hold more detail if you want it later.
One day left. Tomorrow you get a problem you haven't seen, and the same three questions.