Two-week programme · Day 4 of 14
2.1 seconds at 4:12 pm
Part 1Basics
The worker is running. Now it has two jobs: send the phone's changes up, and bring everyone else's changes down. Doing both is called a sync run.
To know what to bring down, the phone keeps a bookmark called a cursor. It works like a page number in a shared logbook: I have read up to here, so give me what comes after.
Words you need today
- Push
- Sending the phone's pending changes to the server.
- Pull
- Fetching changes made elsewhere from the server.
- Cursor
- The phone's bookmark in the server's list of changes, such as c_88411.
- Single-flight lock
- A rule that only one sync run may happen at a time.
- Backoff
- Waiting longer after each failed attempt before trying again.
- Jitter
- A random extra delay, so thousands of phones don't all retry in the same instant.
The story
The worker calls the sync engine, which takes a single-flight lock so no second run can overlap, and reads its saved cursor, c_88411. That cursor means: I have seen every change up to here.
First it pushes. The outbox holds 37 ops from Priya's day, sent in batches of 25 and 12. The server answers 36 applied and 1 conflict. The conflict is op 7f3a. One local transaction deletes the 36 finished ops and marks their jobs synced, then marks j-84 as conflict and stores the server's value beside hers.
Then it pulls: changes after c_88411. It receives 140, applies them in one transaction, and stores c_88551 in that same transaction. The run took 2.1 seconds. Priya glances down and sees eleven green ticks and one amber needs review.
Part 2Today's question
What exactly happens in one sync run, and how does it recover from a failure at any step?
By the end of this lesson, you should be able to:
- Narrate a whole sync run from memory
- Explain push before pull
- Handle a crash at each step
Part 3In detail
Step by step
- Take the lock. If another run is active, stop. This run has nothing to add.
- Read the outbox. 37 changes are waiting. Send them in batches of up to 25.
- Handle each result. 36 succeeded, so delete them from the outbox and mark their jobs synced. One conflicted, so mark job j-84 as conflict and keep the server's value next to Priya's.
- Pull after the cursor. Ask for changes after c_88411. The server returns 140.
- Apply in one transaction. Write the 140 changes and the new cursor c_88551 together, so the bookmark can never move ahead of the data.
- Release and report. Free the lock and tell the worker whether anything is left.
Push first, then pull
Pushing first means the pull returns the merged result of your own changes too, so the phone ends the run matching the server. Pulling first would apply changes you are about to affect, and you would need another pull anyway.
Save the cursor with the data
If the cursor commits after the changes and the app dies in between, you pull the same changes again, which is harmless if applying is idempotent. Committing them together removes even that. Saving the cursor first is the real bug: a crash then skips changes forever.
Each failure has one response
Offline: stop, and let the network constraint wake you. A 429 or 5xx: back off with jitter, keep the op. A 4xx validation error: park that op and tell the user, so one bad op doesn't block everything behind it forever.
Op 7f3a on the wire
1POST /v1/sync/push2{ "device_id": "d-7f2", "ops": [3 { "op_id": "0192e0c4-7f3a", "entity": "job", "id": "j-84",4 "op": "update", "base_versions": { "answers.valve_3": 4 },5 "fields": { "answers.valve_3": "leaking" } } ] }6 7→ { "results": [{ "op_id": "0192e0c4-7f3a", "status": "conflict",8 "server": { "value": "replaced", "version": 5, "by": "dev" } }] }9 10GET /v1/sync/pull?cursor=c_88411&limit=50011→ { "changes": [ … 140 … ], "next_cursor": "c_88551", "has_more": false }Why did the server refuse, and who is Dev? That is tomorrow.
The code, explained
This is the real code, and our tests run it. Read each block, then the notes under it.
The contract between phone and server
Write this on the board early. These few types decide how the whole system behaves when things fail.
1sealed interface OpResult {2 val opId: String3 data class Applied(override val opId: String, val seq: Long) : OpResult4 data class Conflict(override val opId: String, val theirs: Theirs) : OpResult5 data class Rejected(override val opId: String, val reason: String) : OpResult6}7 8/** One line of the server's change log. */9data class Change(val seq: Long, val jobId: String, val field: String, val value: String, val version: Int)10data class PullPage(val changes: List<Change>, val next: Long, val hasMore: Boolean)11 12/** 429 or 5xx, or no network at all. Keep the ops and try again later. */13class TryLater(val afterMs: Long? = null) : IOException("try later")14 15/** 410: this device's cursor is older than the log keeps. */16class CursorTooOld(val snapshotUrl: String) : IOException("cursor too old")17 18interface SyncApi {19 suspend fun push(ops: List<Op>): List<OpResult>20 suspend fun pull(cursor: Long, limit: Int): PullPage21}OpResult- Three answers per op: applied, conflict or rejected. One batch can contain all three.
val seq: Long- The line number in the server's log. The cursor is simply the last number the phone has read.
TryLater- 429, 5xx or no network. A separate type means retry code can't confuse this with a bad request.
CursorTooOld- The server's 410. The phone must rebuild from a snapshot, because some deletes may be gone from the log.
Compiled and tested by npm run test:mobile.
One sync run
This is the 2.1 seconds at 4:12 pm. Read it top to bottom: lock, push until empty, pull until caught up, report.
1class SyncEngine(2 private val db: LocalDb,3 private val api: SyncApi,4 private val batchSize: Int = 25,5 private val pageSize: Int = 500,6) {7 private val lock = Mutex()8 9 suspend fun run(hasTime: () -> Boolean = { true }): RunOutcome {10 if (!lock.tryLock()) return RunOutcome.AlreadyRunning // single flight11 var applied = 012 var conflicts = 013 var pulled = 014 try {15 while (db.outbox.isNotEmpty() && hasTime()) {16 val results = api.push(db.outbox.take(batchSize)) // may throw: ops stay queued17 applyResults(results)18 applied += results.count { it is OpResult.Applied }19 conflicts += results.count { it is OpResult.Conflict }20 }21 var more = true22 while (more && hasTime()) {23 val page = api.pull(db.cursor, pageSize)24 applyPage(page)25 pulled += page.changes.size26 more = page.hasMore27 }28 return RunOutcome.Finished(applied, conflicts, pulled, morePending = db.outbox.isNotEmpty() || more)29 } catch (e: TryLater) {30 return RunOutcome.RetryLater(e.afterMs)31 } catch (e: CursorTooOld) {32 return RunOutcome.RebuildNeeded(e.snapshotUrl)33 } finally {34 lock.unlock()35 }36 }37 // applyResults and applyPage follow38}lock.tryLock()- Single flight. If a run is already active, this one returns at once, because that run will see any new ops.
db.outbox.take(batchSize)- Takes 25 at a time, and reads them fresh each loop, so edits made during the run go out too.
api.push(...)- If this throws, nothing was removed locally. The ops wait for the next run.
while (more && hasTime())- Keeps pulling pages until the server has no more, or the budget ends.
finally { lock.unlock() }- Runs even after a failure, so one bad run can't block sync forever.
Compiled and tested by npm run test:mobile.
Applying the server's answers
The push came back. One transaction records every outcome, so the outbox and the job states always agree.
1/** One transaction: finished ops leave the outbox together with their job's new state. */2private fun applyResults(results: List<OpResult>) = db.transaction {3 for (r in results) {4 val op = outbox.first { it.opId == r.opId }5 outbox = outbox - op6 val job = jobs.getValue(op.jobId)7 val updated = when (r) {8 // A later success on this job must not clear a conflict or a failure.9 is OpResult.Applied ->10 if (job.syncState == SyncState.PENDING && outbox.none { it.jobId == op.jobId }) {11 job.copy(syncState = SyncState.SYNCED)12 } else job13 is OpResult.Conflict -> job.copy(syncState = SyncState.CONFLICT, theirs = r.theirs)14 is OpResult.Rejected -> job.copy(syncState = SyncState.FAILED, error = r.reason)15 }16 jobs = jobs + (op.jobId to updated)17 }18}outbox = outbox - op- A finished op leaves the queue. Applied, conflict and rejected ops all stop being retried.
job.syncState == SyncState.PENDING- Priya's story found this bug: a later success on the same job used to clear its amber badge. Only a pending job may turn synced.
outbox.none { it.jobId == op.jobId }- A job only shows a tick once all its ops are done.
OpResult.Conflict- Stores the other person's value next to hers, ready for the conflict screen.
Compiled and tested by npm run test:mobile.
Applying a pulled page
Changes and the new cursor commit together. That one choice removes a whole family of sync bugs.
1/** One transaction: the data and the cursor can never disagree. */2private fun applyPage(page: PullPage) = db.transaction {3 for (c in page.changes) {4 val job = jobs[c.jobId] ?: Job(c.jobId)5 val unsent = outbox.any { it.jobId == c.jobId && it.field == c.field }6 val inConflict = job.theirs?.field == c.field7 jobs = jobs + (c.jobId to job.copy(8 // Never overwrite something the user changed and the server has not settled.9 answers = if (unsent || inConflict) job.answers else job.answers + (c.field to c.value),10 version = maxOf(job.version, c.version),11 ))12 }13 cursor = page.next14}unsent || inConflict- Never overwrite something the user changed and the server hasn't settled. This is what keeps leaking on Priya's screen.
maxOf(job.version, c.version)- The version still moves forward, so her eventual choice is based on version 5.
cursor = page.next- Saved in the same transaction. If the phone dies, both the changes and the bookmark roll back, and the pull simply repeats.
Compiled and tested by npm run test:mobile.
When to try again
After an outage, every phone wants to retry. This spreads them out.
1/** Full jitter: anywhere from zero up to the exponential cap, so phones spread out after an outage. */2fun retryDelayMs(attempt: Int, random: Random, baseMs: Long = 1_000, capMs: Long = 300_000): Long {3 val exponential = baseMs shl attempt.coerceIn(0, 20)4 return random.nextLong(minOf(exponential, capMs) + 1)5}baseMs shl attempt- Doubles the wait each attempt: 1, 2, 4, 8 seconds and so on.
minOf(exponential, capMs)- Stops growing at five minutes, so a phone never waits for hours.
random.nextLong(...)- Full jitter picks anywhere from zero up to that limit. Without it, phones that failed together retry together.
Compiled and tested by npm run test:mobile.
What can go wrong
- The server applied the push but the reply was lost
- The ops stay in the outbox and get resent. The server recognises their op_ids and does nothing twice.
- The server is overloaded (429 or 5xx)
- Keep the ops, and wait using backoff with jitter.
- The server rejects one op as invalid (4xx)
- Set that op aside and tell the user, so one bad op doesn't block the rest.
- The phone dies during the pull
- The transaction hasn't committed, so the cursor hasn't moved. The next run repeats the pull safely.
Decisions to defend
- How do you stop parallel sync runs? Single-flight lock in the engine, not trust the scheduler alone. Sync can also start when the app opens, outside the scheduler, so only the engine sees every caller.
Part 4Questions and answers
Answer out loud first, then open each one.
The phone dies after the server applied the push but before it got the response. What happens?
Why must the pull apply in one transaction?
Part 5Going further
The three questions, for today
- Where does it live?
- In the outbox until acknowledged. The cursor commits with the pulled changes.
- Who runs the next step?
- The worker runs the engine, and a follow-up job takes any leftovers.
- What if it runs twice?
- Known op_ids return stored results, and re-applying a pull changes nothing.
Practise
- Build30 minRetry with exponential backoff and jitter
Exponential backoff with jitter. Ask why jitter matters when 200,000 phones retry at once.
Optional reference pages
You don’t need these for the lesson. They hold more detail if you want it later.
Thirty-six ops went through and one did not. Tomorrow: two people changed one valve, and no clock can settle it.