Design every step so it is safe to kill and resume.
if retries < 5 { backoff(attempt) }
Backend and scale reels
Backend and scale1 / 4
1 / 5
No memory of clients
Each sync request carries everything the server needs: the device’s cursor and a batch of operations with op_ids. No session lives on any server, so any instance can answer any request, and you add capacity by adding instances.
2 / 5
Change log
Every write produces a change-log entry with a sequence number per partition, usually per tenant or user. A pull is a cheap range read on that log, and partitioning keeps each read small as the log grows.
3 / 5
Change data capture
A transactional outbox table on the server, or Debezium reading the Postgres write-ahead log, turns writes into an ordered stream in Kafka. Search indexing, thumbnails, snapshots and push notifications consume it outside the request path.
4 / 5
Bytes never touch it
Because uploads go straight to object storage through presigned URLs, terabytes of media never pass through your API servers. They handle small JSON and stay cheap to run.
Interview answer
Say it in the interview
The sync API is stateless: every request carries its cursor and op_ids, so any instance serves any device. Writes land in a partitioned change log, CDC streams them to Kafka for indexing and notifications, and file bytes go straight to object storage. Scaling means adding pods.
1 / 5
No memory of clients
Each sync request carries everything the server needs: the device’s cursor and a batch of operations with op_ids. No session lives on any server, so any instance can answer any request, and you add capacity by adding instances.
2 / 5
Change log
Every write produces a change-log entry with a sequence number per partition, usually per tenant or user. A pull is a cheap range read on that log, and partitioning keeps each read small as the log grows.
3 / 5
Change data capture
A transactional outbox table on the server, or Debezium reading the Postgres write-ahead log, turns writes into an ordered stream in Kafka. Search indexing, thumbnails, snapshots and push notifications consume it outside the request path.
4 / 5
Bytes never touch it
Because uploads go straight to object storage through presigned URLs, terabytes of media never pass through your API servers. They handle small JSON and stay cheap to run.
Interview answer
Say it in the interview
The sync API is stateless: every request carries its cursor and op_ids, so any instance serves any device. Writes land in a partitioned change log, CDC streams them to Kafka for indexing and notifications, and file bytes go straight to object storage. Scaling means adding pods.
Why the backend scales sideways
#backend
❚❚
Backend and scale2 / 4
1 / 4
Thundering herd
After an outage, every device holds a backlog and retries at the same moment, so your own clients become the biggest threat to recovery. Sending one push to every user at once causes the same spike.
2 / 4
On the client
Full jitter spreads retries randomly across the backoff window. Coalescing triggers means one device runs one sync, not five. A remote-config kill switch lets you pause sync entirely without shipping a release.
3 / 4
On the server
Rate-limit per device and per tenant, and return 429 with Retry-After to set the pace. Spread push fan-out over several minutes, and use collapse keys so a device receives one changes-available message, not fifty.
Interview answer
Say it in the interview
To prevent retry storms, clients use full-jitter backoff and coalesced triggers, the server rate-limits and returns 429 with Retry-After, and push fan-out is staggered with collapse keys. A remote kill switch lets us pause sync without shipping a release.
1 / 4
Thundering herd
After an outage, every device holds a backlog and retries at the same moment, so your own clients become the biggest threat to recovery. Sending one push to every user at once causes the same spike.
2 / 4
On the client
Full jitter spreads retries randomly across the backoff window. Coalescing triggers means one device runs one sync, not five. A remote-config kill switch lets you pause sync entirely without shipping a release.
3 / 4
On the server
Rate-limit per device and per tenant, and return 429 with Retry-After to set the pace. Spread push fan-out over several minutes, and use collapse keys so a device receives one changes-available message, not fifty.
Interview answer
Say it in the interview
To prevent retry storms, clients use full-jitter backoff and coalesced triggers, the server rate-limits and returns 429 with Retry-After, and push fan-out is staggered with collapse keys. A remote kill switch lets us pause sync without shipping a release.
Surviving your own users
#reliability
❚❚
Backend and scale3 / 4
1 / 4
Requests
One million daily users syncing ten times a day is ten million requests, about 115 per second on average. Peaks run five to ten times higher, so plan for around a thousand per second, plus room for retry storms.
2 / 4
Bytes
If each user uploads 20 MB a day, that is 20 TB a day of ingress, all of it straight to object storage. The interesting costs become storage tiers, CDN egress and deduplication by hash.
3 / 4
Log
Fifty changes per user per day at about 1 KB each is 50 GB a day of change log. With 90 days of tombstone retention, you need partitioning and archiving from the start.
In practiceSay the arithmetic out loud. It is what panels remember.
Interview answer
Say it in the interview
A million daily users syncing ten times a day is about 115 requests per second, with peaks near a thousand. At 20 MB of uploads each, that is 20 TB a day, all direct to object storage. A change log near 50 GB a day needs partitioning from the first day.
1 / 4
Requests
One million daily users syncing ten times a day is ten million requests, about 115 per second on average. Peaks run five to ten times higher, so plan for around a thousand per second, plus room for retry storms.
2 / 4
Bytes
If each user uploads 20 MB a day, that is 20 TB a day of ingress, all of it straight to object storage. The interesting costs become storage tiers, CDN egress and deduplication by hash.
3 / 4
Log
Fifty changes per user per day at about 1 KB each is 50 GB a day of change log. With 90 days of tombstone retention, you need partitioning and archiving from the start.
In practiceSay the arithmetic out loud. It is what panels remember.
Interview answer
Say it in the interview
A million daily users syncing ten times a day is about 115 requests per second, with peaks near a thousand. At 20 MB of uploads each, that is 20 TB a day, all direct to object storage. A change log near 50 GB a day needs partitioning from the first day.