/Tracks

Two-week programme · Day 13 of 14

You can't roll back a phone

Study with AI

Part 1Basics

On a server, a bad release can be rolled back in minutes. On phones, it can't. Once an update is installed it stays, and a fix must pass store review and then wait for users to update.

So the goal shifts to catching problems while few people have them, and being able to switch off a broken feature remotely. Principal engineers are often judged on exactly this kind of system.

Words you need today

Staged rollout
Releasing to 1% of users, then 10%, then everyone, with a check at each step.
Crash-free sessions
The share of app sessions with no crash, the headline health number.
ANR
Application Not Responding: Android's name for a frozen main thread.
Telemetry
Measurements the app sends back about how it's behaving.
Feature flag
A remote setting that turns a feature on or off without an app update.
Kill switch
A flag whose job is to turn something off in an emergency.

The story

Version 8.2 goes to 1% of users on Monday. On one manufacturer's phones, a new SDK starts during the background wake-up and crashes the process. Remember Day 3: nothing on screen shows it. Priya's phone just stops syncing, her outbox quietly grows, and she sees clocks instead of ticks.

Under the old setup, you'd learn this five days later from store reviews. Now phones report crash-free sessions and outbox depth by version and device. Within an hour the new version looks worse than the old one on those phones, so the rollout halts itself. A remote flag turns off the SDK for those who already updated.

This fix needed no new sync design. It needed telemetry, a staged rollout, a flag, and the organisation to run them.

Part 2Today's question

How do you keep all of this working as 150 engineers keep changing it?

By the end of this lesson, you should be able to:

  • Design a staged rollout with gates
  • Choose release signals, including domain health
  • Describe how you'd lead change across teams

Part 3In detail

Step by step

  1. Ship to 1%. Version 8.2 goes out to a small group on Monday.
  2. Phones report health. Crashes and outbox depth get buffered on disk and uploaded in batches, as the outbox does.
  3. Compare new with old. Compare the same device types over the same hours, because a global average can hide a problem on one manufacturer's phones.
  4. Halt automatically. With enough sessions and a clearly worse result, the rollout stops itself.
  5. Switch it off. A kill switch disables the broken SDK for those who already updated.
  6. Fix forward. The corrected version goes through the same stages.

Fixes reach phones slowly

A server fix lands in minutes. A mobile fix waits for store review and for users to update, so detection has to happen while 1% of users have the bug. Old versions also live for years, which is why Day 2's migrations must handle skipped versions.

Telemetry is another outbox

Events buffer on disk and upload in batches, so an offline crash still arrives. The same durable-queue idea carries your release signals. Include domain health such as outbox depth, because a crash rate alone would miss a silent sync stall.

Flags need the same discipline

Assignment must be stable and identical across platforms. Log exposure when the user reaches the feature, and give every flag an owner and an expiry.

Principal work means moving people

Put build time, flaky tests, crash-free sessions and lead time on one dashboard everyone sees. Pick the worst, fix it with one or two teams, publish the result, and let others ask to join.

The code, explained

This is the real code, and our tests run it. Read each block, then the notes under it.

Deciding whether a rollout continues

This would have stopped version 8.2 at 1%. It compares like with like, and waits for enough data.

ReleaseGate.ktKotlin
1data class Health(val sessions: Int, val crashFree: Double, val outboxP90: Int)2 3enum class Rollout { CONTINUE, WAIT, HALT }4 5/** Compare new with old on the same devices over the same hours. */6fun judge(new: Health, old: Health, minSessions: Int = 20_000): Rollout {7  if (new.sessions < minSessions) return Rollout.WAIT    // too little data to judge8  val moreCrashes = new.crashFree < old.crashFree - 0.0029  val syncStalled = new.outboxP90 > old.outboxP90 * 2    // a stall shows no crash at all10  return if (moreCrashes || syncStalled) Rollout.HALT else Rollout.CONTINUE11}12 13private val STAGES = listOf(1, 10, 50, 100)14 15fun nextStage(percent: Int): Int = STAGES.firstOrNull { it > percent } ?: 100
15 linesUTF-8Kotlin
new.sessions < minSessions
Too little data proves nothing. Wait, rather than react to noise.
old.crashFree - 0.002
A small allowance, so normal variation doesn't halt every release.
new.outboxP90 > old.outboxP90 * 2
Catches a sync stall, which shows up in no crash report. Domain health needs its own signal.

Compiled and tested by npm run test:mobile.

Telemetry that survives being offline

It's an outbox for measurements, so a crash in a basement still gets reported.

ReleaseGate.ktKotlin
1data class Event(val id: String, val name: String, val critical: Boolean)2 3/** An outbox for measurements. It keeps crashes when full, and drops routine events first. */4class TelemetryBuffer(private val maxEvents: Int = 500) {5  private val queue = ArrayDeque<Event>()6  val size: Int get() = queue.size7 8  fun record(e: Event) {9    if (queue.size >= maxEvents) {10      val victim = queue.firstOrNull { !it.critical } ?: queue.first()11      queue.remove(victim)12    }13    queue.addLast(e)14  }15 16  /** Removes a batch only after [upload] succeeds, so a failed upload loses nothing. */17  suspend fun flush(batch: Int = 50, upload: suspend (List<Event>) -> Boolean): Int {18    val events = queue.take(batch)19    if (events.isEmpty() || !upload(events)) return 020    repeat(events.size) { queue.removeFirst() }21    return events.size22  }23}24 25/** Server side. A resend must not inflate the crash rate. */26class EventIntake {27  private val seen = HashSet<String>()28  fun accept(batch: List<Event>): List<Event> = batch.filter { seen.add(it.id) }29}
29 linesUTF-8Kotlin
queue.firstOrNull { !it.critical }
When full, routine events go first. Crash reports are the last to be dropped.
if (events.isEmpty() || !upload(events))
Events leave the queue only after a successful upload.
class EventIntake
The server removes repeats, so a resend can't inflate the crash rate.

Compiled and tested by npm run test:mobile.

Stable variant assignment

The same user must get the same variant on any device, offline, with no server call.

FlagEvaluator.ktKotlin
1data class Variant(val name: String, val weight: Int)2 3data class Experiment(4  val key: String,5  val salt: String,6  val variants: List<Variant>,7  val rolloutPercent: Int = 100,8  val killed: Boolean = false,9)10 11/** Pure assignment: the same user and experiment always get the same variant, on any device. */12class FlagEvaluator(private val onExposure: (experiment: String, variant: String) -> Unit) {13  private val logged = HashSet<String>()14 15  fun variantFor(userId: String, exp: Experiment, default: String = "control"): String {16    if (exp.killed) return default                         // kill switch beats everything17    val bucket = bucketOf("${exp.salt}:$userId")18    if (bucket >= exp.rolloutPercent * 100) return default19    val total = exp.variants.sumOf { it.weight }20    var point = bucketOf("${exp.salt}:variant:$userId") % total21    val chosen = exp.variants.first { v -> (point < v.weight).also { if (!it) point -= v.weight } }.name22    // Log once per session, and only when the user actually reaches the feature.23    if (logged.add(exp.key)) onExposure(exp.key, chosen)24    return chosen25  }26}27 28/** FNV-1a into 0..9999: stable across platforms, unlike hashCode() between languages. */29fun bucketOf(s: String): Int {30  var h = 0x811C9DC5.toInt()31  for (b in s.encodeToByteArray()) {32    h = h xor (b.toInt() and 0xFF)33    h *= 1677761934  }35  return ((h.toLong() and 0xFFFFFFFFL) % 10_000).toInt()36}
36 linesUTF-8Kotlin
if (exp.killed) return default
The kill switch wins over everything.
bucketOf("${exp.salt}:$userId")
A new salt gives each experiment an independent split.
logged.add(exp.key)
Logs exposure once, when the feature is reached, so unexposed users don't dilute results.
FNV-1a
hashCode differs between languages. A hash written out by hand gives Android and iOS the same answer.

Compiled and tested by npm run test:mobile.

What can go wrong

A failure shows nothing on screen, like a silent sync stall
Measure domain health, such as outbox depth, and not only crashes.
Too little data at 1%
Set a minimum session count before judging.
Old flags pile up
Give each flag an owner and an expiry date.
Teams resist the process
Show better numbers from one team first, and make the good path the easy one.

Decisions to defend

  • How do you ship to 150 engineers' users? Weekly train, staged, with automatic gates, not monthly big release. Small batches make each bad change cheaper to find and stop.

Part 4Questions and answers

Answer out loud first, then open each one.

A kill switch and a rollout halt: why both?
A halt stops new users from getting the bug. A kill switch helps those who already have it. You need both, because you can't recall an installed app.
Twenty teams resist a new release process. What do you do?
Show one team's better numbers before asking for anything, keep the path easy, and make the old way visibly slower. Mandates without evidence rarely last.

Part 5Going further

The three questions, for today

Where does it live?
Telemetry buffers on disk, as the outbox does.
Who runs the next step?
An upload job on the phone, and an automatic gate on the server.
What if it runs twice?
Event IDs remove duplicates, so a repeat doesn't inflate the crash rate.

Practise

Optional reference pages

You don’t need these for the lesson. They hold more detail if you want it later.

One day left. Tomorrow you get a problem you haven't seen, and the same three questions.

← Day 12: Nearly all reads, on a mid-range phone

AI Engineer · Day 13 of 14