/Practice

Backend and scale reels

Backend and scale1 / 4
1 / 5

No memory of clients

Each sync request carries everything the server needs: the device’s cursor and a batch of operations with op_ids. No session lives on any server, so any instance can answer any request, and you add capacity by adding instances.

2 / 5

Change log

Every write produces a change-log entry with a sequence number per partition, usually per tenant or user. A pull is a cheap range read on that log, and partitioning keeps each read small as the log grows.

3 / 5

Change data capture

A transactional outbox table on the server, or Debezium reading the Postgres write-ahead log, turns writes into an ordered stream in Kafka. Search indexing, thumbnails, snapshots and push notifications consume it outside the request path.

4 / 5

Bytes never touch it

Because uploads go straight to object storage through presigned URLs, terabytes of media never pass through your API servers. They handle small JSON and stay cheap to run.

Interview answer

Say it in the interview

The sync API is stateless: every request carries its cursor and op_ids, so any instance serves any device. Writes land in a partitioned change log, CDC streams them to Kafka for indexing and notifications, and file bytes go straight to object storage. Scaling means adding pods.

Why the backend scales sideways
#backend
Backend and scale2 / 4
1 / 4

Thundering herd

After an outage, every device holds a backlog and retries at the same moment, so your own clients become the biggest threat to recovery. Sending one push to every user at once causes the same spike.

2 / 4

On the client

Full jitter spreads retries randomly across the backoff window. Coalescing triggers means one device runs one sync, not five. A remote-config kill switch lets you pause sync entirely without shipping a release.

3 / 4

On the server

Rate-limit per device and per tenant, and return 429 with Retry-After to set the pace. Spread push fan-out over several minutes, and use collapse keys so a device receives one changes-available message, not fifty.

Interview answer

Say it in the interview

To prevent retry storms, clients use full-jitter backoff and coalesced triggers, the server rate-limits and returns 429 with Retry-After, and push fan-out is staggered with collapse keys. A remote kill switch lets us pause sync without shipping a release.

Surviving your own users
#reliability
Backend and scale3 / 4
1 / 4

Requests

One million daily users syncing ten times a day is ten million requests, about 115 per second on average. Peaks run five to ten times higher, so plan for around a thousand per second, plus room for retry storms.

2 / 4

Bytes

If each user uploads 20 MB a day, that is 20 TB a day of ingress, all of it straight to object storage. The interesting costs become storage tiers, CDN egress and deduplication by hash.

3 / 4

Log

Fifty changes per user per day at about 1 KB each is 50 GB a day of change log. With 90 days of tombstone retention, you need partitioning and archiving from the start.

In practiceSay the arithmetic out loud. It is what panels remember.
Interview answer

Say it in the interview

A million daily users syncing ten times a day is about 115 requests per second, with peaks near a thousand. At 20 MB of uploads each, that is 20 TB a day, all direct to object storage. A change log near 50 GB a day needs partitioning from the first day.

Capacity maths for the whiteboard
#capacity
Scaling in stages, with gates