/Practice

The sync protocol reels

The sync protocol1 / 5
1 / 5

Problem

A user taps Save. You write to the database, then call the API. If the app is killed between those steps, the server never hears about the change. Call the API first instead, and a failed request loses the edit entirely.

2 / 5

Fix

Write the change and an outbox row in the same local transaction. The outbox row says which entity changed, what changed, and carries a unique op_id. One transaction saves both or neither, so an edit can never exist without the intent to send it.

3 / 5

Draining it

The sync engine reads outbox rows in creation order, sends them, and deletes each row only after the server confirms it. If the process dies halfway, the remaining rows are simply sent on the next run.

4 / 5

Details that matter

Keep operations on the same entity in order and batch the rest. Store the attempt count and next retry time on each row. Move permanently rejected operations to a failed state the user can see, instead of retrying them forever.

In practiceRestaurant version: the waiter writes the order on the pad before walking to the kitchen.
Interview answer

Say it in the interview

Every user write commits the row and an outbox entry in one transaction, so intent is never lost. A sync engine drains the outbox in order, deletes entries only after the server confirms, and parks permanent failures where the user can see them. A crash at any point is recoverable.

The outbox pattern
#outbox
The sync protocol2 / 5
1 / 5

Lost response

The app sends a change, the server applies it, and the response is lost as the phone switches from Wi-Fi to mobile data. To the app the request failed. A naive retry applies it twice: two comments, two orders, two payments.

2 / 5

Op_id

Each change gets a UUID generated on the device at the moment the user acts. The server records the op_ids it has processed. When one arrives again, it returns the original result and changes nothing.

3 / 5

How long to remember

Keep processed op_ids for longer than any realistic retry, often several days, in Redis or a table with a unique index. UUIDv7 suits this well: unique, time-ordered and friendly to indexes.

4 / 5

Retrying politely

Use exponential backoff with full jitter: wait a random time between zero and a growing limit. Honour Retry-After. Retry timeouts, 5xx and 429; send 400 and 403 to a failed state instead.

In practiceSay it: “Every write is idempotent by op_id, so the client can retry without fear.”
Interview answer

Say it in the interview

Every mutation carries a UUIDv7 op_id generated on the device. The server stores processed IDs and returns the original result for duplicates, so retries are always safe. Clients back off exponentially with full jitter and honour Retry-After, which keeps recovery calm after an outage.

Idempotency: making retries safe
#idempotency
The sync protocol3 / 5
1 / 5

Asking what changed

To pull, the app asks the server for everything since it last synced. The obvious approach is a timestamp: rows where updated_at is later than my last sync. It is also the source of a whole family of bugs.

2 / 5

Why timestamps break

Device clocks are often wrong, and server clocks drift between machines. Rows written in the same millisecond straddle the boundary, and a transaction that commits late carries an earlier timestamp than rows you already pulled. Data quietly goes missing.

3 / 5

Sequence numbers

Instead, the server stamps every change with a number from a monotonic sequence per tenant or user. The device stores the last number it applied, its cursor, and asks for changes after it. The order is total, and nothing slips through.

4 / 5

Saving the cursor

Apply a page of changes and save the new cursor in the same database transaction. If the app dies in between, the next run reapplies the page, which is harmless because applying a change twice gives the same result.

In practiceKeep the cursor opaque, like c_1043, so you can change what it means without breaking old apps.
Interview answer

Say it in the interview

We never sync by timestamps, because clocks drift and late commits slip through. The server stamps each change with a per-tenant sequence number and the device keeps an opaque cursor. Applying a page and saving the cursor happen in one transaction, so a crash only replays a harmless page.

Cursors and server sequence numbers
#cursor
Deletes and devices that vanish for weeks
Choosing a conflict strategy