/Tracks

Two-week programme · Day 4 of 14

2.1 seconds at 4:12 pm

Study with AI

Part 1Basics

The worker is running. Now it has two jobs: send the phone's changes up, and bring everyone else's changes down. Doing both is called a sync run.

To know what to bring down, the phone keeps a bookmark called a cursor. It works like a page number in a shared logbook: I have read up to here, so give me what comes after.

Words you need today

Push
Sending the phone's pending changes to the server.
Pull
Fetching changes made elsewhere from the server.
Cursor
The phone's bookmark in the server's list of changes, such as c_88411.
Single-flight lock
A rule that only one sync run may happen at a time.
Backoff
Waiting longer after each failed attempt before trying again.
Jitter
A random extra delay, so thousands of phones don't all retry in the same instant.

The story

The worker calls the sync engine, which takes a single-flight lock so no second run can overlap, and reads its saved cursor, c_88411. That cursor means: I have seen every change up to here.

First it pushes. The outbox holds 37 ops from Priya's day, sent in batches of 25 and 12. The server answers 36 applied and 1 conflict. The conflict is op 7f3a. One local transaction deletes the 36 finished ops and marks their jobs synced, then marks j-84 as conflict and stores the server's value beside hers.

Then it pulls: changes after c_88411. It receives 140, applies them in one transaction, and stores c_88551 in that same transaction. The run took 2.1 seconds. Priya glances down and sees eleven green ticks and one amber needs review.

Part 2Today's question

What exactly happens in one sync run, and how does it recover from a failure at any step?

By the end of this lesson, you should be able to:

  • Narrate a whole sync run from memory
  • Explain push before pull
  • Handle a crash at each step

Part 3In detail

Step by step

  1. Take the lock. If another run is active, stop. This run has nothing to add.
  2. Read the outbox. 37 changes are waiting. Send them in batches of up to 25.
  3. Handle each result. 36 succeeded, so delete them from the outbox and mark their jobs synced. One conflicted, so mark job j-84 as conflict and keep the server's value next to Priya's.
  4. Pull after the cursor. Ask for changes after c_88411. The server returns 140.
  5. Apply in one transaction. Write the 140 changes and the new cursor c_88551 together, so the bookmark can never move ahead of the data.
  6. Release and report. Free the lock and tell the worker whether anything is left.

Push first, then pull

Pushing first means the pull returns the merged result of your own changes too, so the phone ends the run matching the server. Pulling first would apply changes you are about to affect, and you would need another pull anyway.

Save the cursor with the data

If the cursor commits after the changes and the app dies in between, you pull the same changes again, which is harmless if applying is idempotent. Committing them together removes even that. Saving the cursor first is the real bug: a crash then skips changes forever.

Each failure has one response

Offline: stop, and let the network constraint wake you. A 429 or 5xx: back off with jitter, keep the op. A 4xx validation error: park that op and tell the user, so one bad op doesn't block everything behind it forever.

Op 7f3a on the wire

op-7f3a-wire.httpHTTP
1POST /v1/sync/push2{ "device_id": "d-7f2", "ops": [3  { "op_id": "0192e0c4-7f3a", "entity": "job", "id": "j-84",4    "op": "update", "base_versions": { "answers.valve_3": 4 },5    "fields": { "answers.valve_3": "leaking" } } ] }6 7→ { "results": [{ "op_id": "0192e0c4-7f3a", "status": "conflict",8     "server": { "value": "replaced", "version": 5, "by": "dev" } }] }9 10GET /v1/sync/pull?cursor=c_88411&limit=50011→ { "changes": [ … 140 … ], "next_cursor": "c_88551", "has_more": false }
11 linesUTF-8HTTP

Why did the server refuse, and who is Dev? That is tomorrow.

The code, explained

This is the real code, and our tests run it. Read each block, then the notes under it.

The contract between phone and server

Write this on the board early. These few types decide how the whole system behaves when things fail.

SyncEngine.ktKotlin
1sealed interface OpResult {2  val opId: String3  data class Applied(override val opId: String, val seq: Long) : OpResult4  data class Conflict(override val opId: String, val theirs: Theirs) : OpResult5  data class Rejected(override val opId: String, val reason: String) : OpResult6}7 8/** One line of the server's change log. */9data class Change(val seq: Long, val jobId: String, val field: String, val value: String, val version: Int)10data class PullPage(val changes: List<Change>, val next: Long, val hasMore: Boolean)11 12/** 429 or 5xx, or no network at all. Keep the ops and try again later. */13class TryLater(val afterMs: Long? = null) : IOException("try later")14 15/** 410: this device's cursor is older than the log keeps. */16class CursorTooOld(val snapshotUrl: String) : IOException("cursor too old")17 18interface SyncApi {19  suspend fun push(ops: List<Op>): List<OpResult>20  suspend fun pull(cursor: Long, limit: Int): PullPage21}
21 linesUTF-8Kotlin
OpResult
Three answers per op: applied, conflict or rejected. One batch can contain all three.
val seq: Long
The line number in the server's log. The cursor is simply the last number the phone has read.
TryLater
429, 5xx or no network. A separate type means retry code can't confuse this with a bad request.
CursorTooOld
The server's 410. The phone must rebuild from a snapshot, because some deletes may be gone from the log.

Compiled and tested by npm run test:mobile.

One sync run

This is the 2.1 seconds at 4:12 pm. Read it top to bottom: lock, push until empty, pull until caught up, report.

SyncEngine.ktKotlin
1class SyncEngine(2  private val db: LocalDb,3  private val api: SyncApi,4  private val batchSize: Int = 25,5  private val pageSize: Int = 500,6) {7  private val lock = Mutex()8 9  suspend fun run(hasTime: () -> Boolean = { true }): RunOutcome {10    if (!lock.tryLock()) return RunOutcome.AlreadyRunning     // single flight11    var applied = 012    var conflicts = 013    var pulled = 014    try {15      while (db.outbox.isNotEmpty() && hasTime()) {16        val results = api.push(db.outbox.take(batchSize))     // may throw: ops stay queued17        applyResults(results)18        applied += results.count { it is OpResult.Applied }19        conflicts += results.count { it is OpResult.Conflict }20      }21      var more = true22      while (more && hasTime()) {23        val page = api.pull(db.cursor, pageSize)24        applyPage(page)25        pulled += page.changes.size26        more = page.hasMore27      }28      return RunOutcome.Finished(applied, conflicts, pulled, morePending = db.outbox.isNotEmpty() || more)29    } catch (e: TryLater) {30      return RunOutcome.RetryLater(e.afterMs)31    } catch (e: CursorTooOld) {32      return RunOutcome.RebuildNeeded(e.snapshotUrl)33    } finally {34      lock.unlock()35    }36  }37  // applyResults and applyPage follow38}
38 linesUTF-8Kotlin
lock.tryLock()
Single flight. If a run is already active, this one returns at once, because that run will see any new ops.
db.outbox.take(batchSize)
Takes 25 at a time, and reads them fresh each loop, so edits made during the run go out too.
api.push(...)
If this throws, nothing was removed locally. The ops wait for the next run.
while (more && hasTime())
Keeps pulling pages until the server has no more, or the budget ends.
finally { lock.unlock() }
Runs even after a failure, so one bad run can't block sync forever.

Compiled and tested by npm run test:mobile.

Applying the server's answers

The push came back. One transaction records every outcome, so the outbox and the job states always agree.

SyncEngine.ktKotlin
1/** One transaction: finished ops leave the outbox together with their job's new state. */2private fun applyResults(results: List<OpResult>) = db.transaction {3  for (r in results) {4    val op = outbox.first { it.opId == r.opId }5    outbox = outbox - op6    val job = jobs.getValue(op.jobId)7    val updated = when (r) {8      // A later success on this job must not clear a conflict or a failure.9      is OpResult.Applied ->10        if (job.syncState == SyncState.PENDING && outbox.none { it.jobId == op.jobId }) {11          job.copy(syncState = SyncState.SYNCED)12        } else job13      is OpResult.Conflict -> job.copy(syncState = SyncState.CONFLICT, theirs = r.theirs)14      is OpResult.Rejected -> job.copy(syncState = SyncState.FAILED, error = r.reason)15    }16    jobs = jobs + (op.jobId to updated)17  }18}
18 linesUTF-8Kotlin
outbox = outbox - op
A finished op leaves the queue. Applied, conflict and rejected ops all stop being retried.
job.syncState == SyncState.PENDING
Priya's story found this bug: a later success on the same job used to clear its amber badge. Only a pending job may turn synced.
outbox.none { it.jobId == op.jobId }
A job only shows a tick once all its ops are done.
OpResult.Conflict
Stores the other person's value next to hers, ready for the conflict screen.

Compiled and tested by npm run test:mobile.

Applying a pulled page

Changes and the new cursor commit together. That one choice removes a whole family of sync bugs.

SyncEngine.ktKotlin
1/** One transaction: the data and the cursor can never disagree. */2private fun applyPage(page: PullPage) = db.transaction {3  for (c in page.changes) {4    val job = jobs[c.jobId] ?: Job(c.jobId)5    val unsent = outbox.any { it.jobId == c.jobId && it.field == c.field }6    val inConflict = job.theirs?.field == c.field7    jobs = jobs + (c.jobId to job.copy(8      // Never overwrite something the user changed and the server has not settled.9      answers = if (unsent || inConflict) job.answers else job.answers + (c.field to c.value),10      version = maxOf(job.version, c.version),11    ))12  }13  cursor = page.next14}
14 linesUTF-8Kotlin
unsent || inConflict
Never overwrite something the user changed and the server hasn't settled. This is what keeps leaking on Priya's screen.
maxOf(job.version, c.version)
The version still moves forward, so her eventual choice is based on version 5.
cursor = page.next
Saved in the same transaction. If the phone dies, both the changes and the bookmark roll back, and the pull simply repeats.

Compiled and tested by npm run test:mobile.

When to try again

After an outage, every phone wants to retry. This spreads them out.

SyncEngine.ktKotlin
1/** Full jitter: anywhere from zero up to the exponential cap, so phones spread out after an outage. */2fun retryDelayMs(attempt: Int, random: Random, baseMs: Long = 1_000, capMs: Long = 300_000): Long {3  val exponential = baseMs shl attempt.coerceIn(0, 20)4  return random.nextLong(minOf(exponential, capMs) + 1)5}
5 linesUTF-8Kotlin
baseMs shl attempt
Doubles the wait each attempt: 1, 2, 4, 8 seconds and so on.
minOf(exponential, capMs)
Stops growing at five minutes, so a phone never waits for hours.
random.nextLong(...)
Full jitter picks anywhere from zero up to that limit. Without it, phones that failed together retry together.

Compiled and tested by npm run test:mobile.

What can go wrong

The server applied the push but the reply was lost
The ops stay in the outbox and get resent. The server recognises their op_ids and does nothing twice.
The server is overloaded (429 or 5xx)
Keep the ops, and wait using backoff with jitter.
The server rejects one op as invalid (4xx)
Set that op aside and tell the user, so one bad op doesn't block the rest.
The phone dies during the pull
The transaction hasn't committed, so the cursor hasn't moved. The next run repeats the pull safely.

Decisions to defend

  • How do you stop parallel sync runs? Single-flight lock in the engine, not trust the scheduler alone. Sync can also start when the app opens, outside the scheduler, so only the engine sees every caller.

Part 4Questions and answers

Answer out loud first, then open each one.

The phone dies after the server applied the push but before it got the response. What happens?
On the next run the ops are still in the outbox, so they go again. The server sees known op_ids and returns the stored results. Nothing applies twice, and that is exactly why op_id exists.
Why must the pull apply in one transaction?
The screen must never show half a page of changes. One transaction also lets the query emit once, so the list redraws once.

Part 5Going further

The three questions, for today

Where does it live?
In the outbox until acknowledged. The cursor commits with the pulled changes.
Who runs the next step?
The worker runs the engine, and a follow-up job takes any leftovers.
What if it runs twice?
Known op_ids return stored results, and re-applying a pull changes nothing.

Practise

Optional reference pages

You don’t need these for the lesson. They hold more detail if you want it later.

Thirty-six ops went through and one did not. Tomorrow: two people changed one valve, and no clock can settle it.

← Day 3: Ninety-two minutes of nothing

AI Engineer · Day 4 of 14