/Tracks

Two-week programme · Day 6 of 14

Inside the push, and out to Dev's tablet

Study with AI

Part 1Basics

The phone did its part. Now look at the server. It must apply each change exactly once, even though phones resend things, and must put every change in an order that any device can catch up on.

Picture one numbered logbook. Each change gets the next line number, and a device just asks for the lines after the last one it read.

Words you need today

Idempotent
Safe to run twice: the second run changes nothing.
Change log
A table with one row per change, in order. Devices read from it.
Sequence number
The line number in that log, such as 88412.
Postgres, Debezium, Kafka
The server's database, a tool that reads its committed changes, and a queue that passes them to other services.
Tombstone
A marker saying this record was deleted, so devices learn to delete it too.
Thundering herd
Thousands of clients returning at the same moment, often right after an outage.

The story

Priya's two batches reach different API pods. For each op the pod opens one transaction: check op_id, merge the fields, update the row, take the next sequence number for this tenant, and write to the change log. Her edit on j-81 gets the next number in the log.

Then her phone hops between cell towers and never receives the answer to batch two. The engine resends the same 12 ops. The server finds their op_ids, returns the stored results and writes nothing.

From there, Debezium reads the change log out of Postgres and publishes to Kafka. A notifier coalesces changes per user and sends Dev's tablet one push: changes available. The tablet pulls after its own cursor and shows the new state. No server ever pushed data at it; push only said to come and look.

Part 2Today's question

How does one server stay correct and alive when every phone retries?

By the end of this lesson, you should be able to:

  • Walk through a push transaction
  • Explain how a device catches up by cursor
  • Plan for an outage's return traffic

Part 3In detail

Step by step

  1. Check the op_id. If this ID was already applied, return the saved result and stop. That is what makes a retry harmless.
  2. Merge the fields. Apply the rules from yesterday.
  3. Take the next sequence number. Take it under a lock for this customer, so the numbers appear in the order changes commit.
  4. Write it all in one transaction. The record, the op_id and the change log row commit together.
  5. Publish the change. Debezium reads the committed log and passes it through Kafka.
  6. Tell other devices to look. A notifier sends Dev's tablet one push: changes available. The tablet pulls after its own cursor, as Priya's phone did.

Do the arithmetic first

The inspection app has 120,000 active technicians, which works out to about 42 sync requests per second on average and about 400 at peak. That fits on a modest cluster. The hard moment is an outage: everyone returns at once.

Sequence numbers make the cursor work

A cursor can only promise I have seen everything up to here if numbers appear in commit order. So each tenant gets one ordered sequence, and the change log becomes the single thing every device reads.

Protect the API from its own clients

Phones retry forever, in step. Send Retry-After, have clients add jitter, and rate limit per device, so that recovery doesn't become a second outage.

Deletions need tombstones, kept for a window such as 90 days. A device older than that window gets a 410 and rebuilds from a snapshot.

One op, one server transaction

one-op-one.sqlSQL
1BEGIN;2 3-- A retry inserts nothing here, and the stored result goes back instead.4INSERT INTO applied_op (op_id, tenant_id, result)5VALUES ('0192e0c4-9c21', 't1', :result)6ON CONFLICT (op_id) DO NOTHING;7 8UPDATE job9   SET answers = jsonb_set(answers, '{valve_3}', '"leaking"'), version = version + 110 WHERE id = 'j-81';11 12-- This row lock orders commits, so cursors never skip a change.13UPDATE tenant_seq SET seq = seq + 1 WHERE tenant_id = 't1' RETURNING seq;  -- :next14 15INSERT INTO change_log (tenant_id, seq, entity, entity_id, fields)16VALUES ('t1', :next, 'job', 'j-81', '{"answers.valve_3":"leaking"}');17 18COMMIT;
18 linesUTF-8SQL

The dedupe, the write and the log entry commit together, the same trick as Day 2's outbox, now on the server.

The code, explained

This is the real code, and our tests run it. Read each block, then the notes under it.

Applying one op on the server

Each op must apply exactly once, however many times it arrives. In production this is one Postgres transaction per op.

SyncServer.ktKotlin
1/** The server side of sync. Each op is one transaction: check, merge, number, log. */2class SyncServer(startSeq: Long, private val logFloor: Long = 0) : SyncApi {3  private class Row(val fields: MutableMap<String, Stored> = HashMap(), var version: Int = 0)4 5  private val rows = HashMap<String, Row>()6  private val applied = HashMap<String, OpResult>()     // op_id -> the answer we gave7  private val log = ArrayList<Change>()8  private var seq = startSeq9 10  /** Rows written, so a test can prove that a retry writes nothing. */11  var writes = 012    private set13  var offline = false14 15  fun seedJob(jobId: String, version: Int) { rows[jobId] = Row(version = version) }16 17  override suspend fun push(ops: List<Op>): List<OpResult> {18    if (offline) throw TryLater()19    return synchronized(this) { ops.map(::applyOne) }20  }21 22  private fun applyOne(op: Op): OpResult {23    applied[op.opId]?.let { return it }                 // a retry: same answer, no write24    val row = rows.getOrPut(op.jobId) { Row() }25    val result = when (val m = merge(op.field, op.value, op.baseVersion, row.fields[op.field])) {26      is Merge.Refuse -> OpResult.Conflict(op.opId, Theirs(op.field, m.current.value, row.version, m.current.by))27      is Merge.Apply -> {28        row.version += 129        row.fields[op.field] = Stored(m.value, row.version, op.by)30        seq += 131        log += Change(seq, op.jobId, op.field, m.value, row.version)32        writes += 133        OpResult.Applied(op.opId, seq)34      }35    }36    applied[op.opId] = result37    return result38  }39  // pull follows40}
40 linesUTF-8Kotlin
applied[op.opId]?.let { return it }
The first line. A retry returns the saved answer and writes nothing. A conflict counts too, so a retry can't turn it into a success.
merge(op.field, ...)
The same function as the merge lesson.
seq += 1
The next line number in the log. In production a per-customer row lock keeps these in commit order.
log += Change(...)
Every change goes here. Other devices read only this.

Compiled and tested by npm run test:mobile.

Serving a pull

A device says where it got to, and gets what comes after.

SyncServer.ktKotlin
1override suspend fun pull(cursor: Long, limit: Int): PullPage {2  if (offline) throw TryLater()3  if (cursor < logFloor) throw CursorTooOld("/snapshots/latest")4  return synchronized(this) {5    val after = log.filter { it.seq > cursor }6    val page = after.take(limit)7    PullPage(page, page.lastOrNull()?.seq ?: cursor, hasMore = after.size > limit)8  }9}
9 linesUTF-8Kotlin
cursor < logFloor
This device is older than the log keeps, so answer 410 and send it to a snapshot.
it.seq > cursor
Only what this device hasn't read. Asking twice is harmless.
hasMore = after.size > limit
Tells the phone to keep pulling. Large catch-ups arrive in safe pieces.

Compiled and tested by npm run test:mobile.

Telling other devices to look

A push doesn't carry the change. It just says something changed, so Dev's tablet pulls by cursor like any other device.

SyncServer.ktKotlin
1/** Push says "come and look". It carries no data, and one push covers many changes. */2class ChangeNotifier(private val sendPush: (device: String) -> Unit) {3  fun onCommitted(changes: List<Change>, devicesFor: (jobId: String) -> Set<String>, author: String) {4    changes.flatMap { devicesFor(it.jobId) }5      .toSet()6      .minus(author)7      .forEach(sendPush)8  }9}
9 linesUTF-8Kotlin
flatMap { devicesFor(it.jobId) }
Finds who cares about each change.
.toSet()
Five changes for one tablet still mean one push, which saves battery and provider quota.
.minus(author)
The phone that made the change doesn't need telling.

Compiled and tested by npm run test:mobile.

What can go wrong

A reply gets lost and the phone retries
The op_id check returns the saved result, and nothing is written twice.
The service was down, and every phone returns at once
Send Retry-After, rely on client jitter, and rate limit per device.
A push never arrives
Nothing is lost. Push is only a hint, and the next pull catches up.
A device was off for 120 days
Its cursor is older than the log keeps, so the server replies 410, and the device rebuilds from a snapshot.

Decisions to defend

  • How do devices learn about changes? Push says changes available, then the device pulls by cursor, not push carries the data. Pushes get dropped and throttled. The pull is the guarantee, and the push is only a hint.

Part 4Questions and answers

Answer out loud first, then open each one.

Why not simply use a database auto-increment for the cursor?
Auto-increment values get assigned before commit, so a slower transaction can commit a lower number after a client has read past it. That client then misses a change forever. Taking the sequence under a per-tenant lock prevents this.
A tablet has been off for 120 days. What happens?
Its cursor is older than the 90-day tombstone window, so deletes it needs may be gone. The server returns 410 and a snapshot URL, and the tablet rebuilds locally. Unsent ops must still push first.

Part 5Going further

The three questions, for today

Where does it live?
In Postgres: the row, the applied op and the change log entry, committed together.
Who runs the next step?
Debezium and Kafka carry it on, then a notifier prompts other devices.
What if it runs twice?
The applied-op table catches a repeat and returns the stored result.

Practise

Optional reference pages

You don’t need these for the lesson. They hold more detail if you want it later.

Text is small. Tomorrow a teammate records a 412 MB video in a pump room, and the outbox alone can't carry that.

← Day 5: Leaking or replaced?

AI Engineer · Day 6 of 14