/Practice

Huge files and huge data reels

Huge files and huge data1 / 3
1 / 6

Why one request fails

A single 4 GB upload over mobile data will almost certainly be interrupted. A network switch, a tunnel or the app being killed sends it back to zero. The answer is to split the file and make every piece resumable on its own.

2 / 6

Prepare

Copy the file into app storage first, because a gallery or picker URI can disappear. Hash it with SHA-256 while copying, and compress or transcode if the product allows. Record all of it in a file-transfer table before a byte leaves the phone.

3 / 6

Ask before sending

The app sends the hash and size to the server. If the server already holds that content, the app skips the upload. Otherwise it starts a multipart upload and returns presigned URLs, one per part, valid for a few minutes.

4 / 6

Send parts, save progress

Parts of 8 to 16 MB go straight to object storage, two to four at a time. After each part lands, the app writes its ETag to the database. A kill now costs at most the parts that were in flight.

5 / 6

Finish

When every part is done, the app asks the server to complete the upload. The server verifies the hash, links the file to its record and queues thumbnails or virus scans. That record change then syncs like any other.

In practiceS3 parts must be at least 5 MB, except the last. Abort stale multipart uploads with a lifecycle rule.
Interview answer

Say it in the interview

Before upload, the app copies large files into its storage, hashes them with SHA-256 and deduplicates them. Then S3 multipart with presigned part URLs, saving each part’s ETag as it lands, so a kill costs at most the parts in flight. The server verifies the hash on completion.

Uploading 4 GB on a bad network
#uploads
Huge files and huge data2 / 3
1 / 5

Download less

Most users never need every file. Show thumbnails first and fetch the full file when it is opened. Prefetch only what is likely to be needed, such as today’s jobs, and only on Wi-Fi while charging.

2 / 5

Resume, verify, rename

Use HTTP Range requests to continue from the last byte received. Write to a temporary file, verify its hash, then atomically rename it into place, so a half-written file is never mistaken for a complete one.

3 / 5

Serve from the edge

Put files behind a CDN with short-lived signed URLs and content-addressed keys, such as the hash. A file then never changes at a given address and can be cached forever.

4 / 5

Respect the phone

Keep a storage budget and evict the least recently used files first, sparing pinned ones. Handle low-disk errors calmly, and keep re-downloadable files out of cloud backups.

Interview answer

Say it in the interview

Downloads are lazy: thumbnails first, prefetch only on Wi-Fi while charging. We resume with Range requests, verify hashes and rename atomically. Files sit behind a CDN with signed, content-addressed URLs, and each device keeps a storage budget with LRU eviction.

Downloads, storage and caching
#downloads
Huge files and huge data3 / 3
1 / 5

Paging does not scale

A new device on a big account may need millions of records. Paging them through the API 500 at a time means thousands of requests and hours of waiting. Users give up long before it finishes.

2 / 5

Snapshots

Instead, the server builds a compressed snapshot per scope on a schedule, as a SQLite file or newline-delimited JSON, and serves it from a CDN. The app downloads it once, imports it in the background and pulls deltas from the snapshot’s cursor.

3 / 5

Sync less

Partial replication limits a device to what its user needs: their team, their region or the last 90 days. Older data is fetched on demand from the API. Most users never notice, and the database stays small.

4 / 5

Import fast

Parse responses as a stream rather than loading them whole. Insert in large transactions, and pause full-text indexing during the import, rebuilding it at the end.

In practiceSize pages by bytes, not rows: roughly 100 to 500 KB each.
Interview answer

Say it in the interview

New devices bootstrap from a prebuilt, CDN-served snapshot per scope, then pull deltas from its cursor. Partial replication keeps each device to the data its user needs, and imports stream, batch and defer indexing. First sync goes from hours of paging to minutes.

First sync with two million rows
#datasets