Design every step so it is safe to kill and resume.
if retries < 5 { backoff(attempt) }
The sync protocol reels
The sync protocol1 / 5
1 / 5
Problem
A user taps Save. You write to the database, then call the API. If the app is killed between those steps, the server never hears about the change. Call the API first instead, and a failed request loses the edit entirely.
2 / 5
Fix
Write the change and an outbox row in the same local transaction. The outbox row says which entity changed, what changed, and carries a unique op_id. One transaction saves both or neither, so an edit can never exist without the intent to send it.
3 / 5
Draining it
The sync engine reads outbox rows in creation order, sends them, and deletes each row only after the server confirms it. If the process dies halfway, the remaining rows are simply sent on the next run.
4 / 5
Details that matter
Keep operations on the same entity in order and batch the rest. Store the attempt count and next retry time on each row. Move permanently rejected operations to a failed state the user can see, instead of retrying them forever.
In practiceRestaurant version: the waiter writes the order on the pad before walking to the kitchen.
Interview answer
Say it in the interview
Every user write commits the row and an outbox entry in one transaction, so intent is never lost. A sync engine drains the outbox in order, deletes entries only after the server confirms, and parks permanent failures where the user can see them. A crash at any point is recoverable.
1 / 5
Problem
A user taps Save. You write to the database, then call the API. If the app is killed between those steps, the server never hears about the change. Call the API first instead, and a failed request loses the edit entirely.
2 / 5
Fix
Write the change and an outbox row in the same local transaction. The outbox row says which entity changed, what changed, and carries a unique op_id. One transaction saves both or neither, so an edit can never exist without the intent to send it.
3 / 5
Draining it
The sync engine reads outbox rows in creation order, sends them, and deletes each row only after the server confirms it. If the process dies halfway, the remaining rows are simply sent on the next run.
4 / 5
Details that matter
Keep operations on the same entity in order and batch the rest. Store the attempt count and next retry time on each row. Move permanently rejected operations to a failed state the user can see, instead of retrying them forever.
In practiceRestaurant version: the waiter writes the order on the pad before walking to the kitchen.
Interview answer
Say it in the interview
Every user write commits the row and an outbox entry in one transaction, so intent is never lost. A sync engine drains the outbox in order, deletes entries only after the server confirms, and parks permanent failures where the user can see them. A crash at any point is recoverable.
The outbox pattern
#outbox
❚❚
The sync protocol2 / 5
1 / 5
Lost response
The app sends a change, the server applies it, and the response is lost as the phone switches from Wi-Fi to mobile data. To the app the request failed. A naive retry applies it twice: two comments, two orders, two payments.
2 / 5
Op_id
Each change gets a UUID generated on the device at the moment the user acts. The server records the op_ids it has processed. When one arrives again, it returns the original result and changes nothing.
3 / 5
How long to remember
Keep processed op_ids for longer than any realistic retry, often several days, in Redis or a table with a unique index. UUIDv7 suits this well: unique, time-ordered and friendly to indexes.
4 / 5
Retrying politely
Use exponential backoff with full jitter: wait a random time between zero and a growing limit. Honour Retry-After. Retry timeouts, 5xx and 429; send 400 and 403 to a failed state instead.
In practiceSay it: “Every write is idempotent by op_id, so the client can retry without fear.”
Interview answer
Say it in the interview
Every mutation carries a UUIDv7 op_id generated on the device. The server stores processed IDs and returns the original result for duplicates, so retries are always safe. Clients back off exponentially with full jitter and honour Retry-After, which keeps recovery calm after an outage.
1 / 5
Lost response
The app sends a change, the server applies it, and the response is lost as the phone switches from Wi-Fi to mobile data. To the app the request failed. A naive retry applies it twice: two comments, two orders, two payments.
2 / 5
Op_id
Each change gets a UUID generated on the device at the moment the user acts. The server records the op_ids it has processed. When one arrives again, it returns the original result and changes nothing.
3 / 5
How long to remember
Keep processed op_ids for longer than any realistic retry, often several days, in Redis or a table with a unique index. UUIDv7 suits this well: unique, time-ordered and friendly to indexes.
4 / 5
Retrying politely
Use exponential backoff with full jitter: wait a random time between zero and a growing limit. Honour Retry-After. Retry timeouts, 5xx and 429; send 400 and 403 to a failed state instead.
In practiceSay it: “Every write is idempotent by op_id, so the client can retry without fear.”
Interview answer
Say it in the interview
Every mutation carries a UUIDv7 op_id generated on the device. The server stores processed IDs and returns the original result for duplicates, so retries are always safe. Clients back off exponentially with full jitter and honour Retry-After, which keeps recovery calm after an outage.
Idempotency: making retries safe
#idempotency
❚❚
The sync protocol3 / 5
1 / 5
Asking what changed
To pull, the app asks the server for everything since it last synced. The obvious approach is a timestamp: rows where updated_at is later than my last sync. It is also the source of a whole family of bugs.
2 / 5
Why timestamps break
Device clocks are often wrong, and server clocks drift between machines. Rows written in the same millisecond straddle the boundary, and a transaction that commits late carries an earlier timestamp than rows you already pulled. Data quietly goes missing.
3 / 5
Sequence numbers
Instead, the server stamps every change with a number from a monotonic sequence per tenant or user. The device stores the last number it applied, its cursor, and asks for changes after it. The order is total, and nothing slips through.
4 / 5
Saving the cursor
Apply a page of changes and save the new cursor in the same database transaction. If the app dies in between, the next run reapplies the page, which is harmless because applying a change twice gives the same result.
In practiceKeep the cursor opaque, like c_1043, so you can change what it means without breaking old apps.
Interview answer
Say it in the interview
We never sync by timestamps, because clocks drift and late commits slip through. The server stamps each change with a per-tenant sequence number and the device keeps an opaque cursor. Applying a page and saving the cursor happen in one transaction, so a crash only replays a harmless page.
1 / 5
Asking what changed
To pull, the app asks the server for everything since it last synced. The obvious approach is a timestamp: rows where updated_at is later than my last sync. It is also the source of a whole family of bugs.
2 / 5
Why timestamps break
Device clocks are often wrong, and server clocks drift between machines. Rows written in the same millisecond straddle the boundary, and a transaction that commits late carries an earlier timestamp than rows you already pulled. Data quietly goes missing.
3 / 5
Sequence numbers
Instead, the server stamps every change with a number from a monotonic sequence per tenant or user. The device stores the last number it applied, its cursor, and asks for changes after it. The order is total, and nothing slips through.
4 / 5
Saving the cursor
Apply a page of changes and save the new cursor in the same database transaction. If the app dies in between, the next run reapplies the page, which is harmless because applying a change twice gives the same result.
In practiceKeep the cursor opaque, like c_1043, so you can change what it means without breaking old apps.
Interview answer
Say it in the interview
We never sync by timestamps, because clocks drift and late commits slip through. The server stamps each change with a per-tenant sequence number and the device keeps an opaque cursor. Applying a page and saving the cursor happen in one transaction, so a crash only replays a harmless page.
Cursors and server sequence numbers
#cursor
❚❚
Deletes and devices that vanish for weeksChoosing a conflict strategy