Two-week programme · Day 7 of 14
Arjun's 412 MB video
Part 1Basics
An edit is a few bytes. A video can be hundreds of megabytes, and a phone may lose signal or get closed partway through sending it. Starting over each time would waste hours and data.
So we cut the file into small numbered parts and send them one by one. Like moving house box by box: if the van breaks down, only the boxes still on it need another trip.
Words you need today
- Object storage
- A service built to hold large files, such as Amazon S3.
- Multipart upload
- Sending a file as separate parts that get joined at the end.
- Presigned URL
- A temporary link that lets the phone upload straight to storage without going through your server.
- ETag
- A receipt returned for each uploaded part. You need all of them to finish the upload.
- Hash
- A short fingerprint of a file. Same file, same fingerprint.
- Background URLSession
- An iOS service that keeps transferring files even after your app has been suspended.
The story
Arjun, on Priya's crew, films a pump room and taps Submit. The app copies the 412 MB file into its own storage, hashing it on the way, and asks the server to start. The server returns 52 presigned URLs of 8 MB each, so the bytes go straight to object storage and never touch your API.
At part 20 the phone moves from Wi-Fi to 4G. Three parts fail and retry. At part 31 the URLs expire, so the app fetches fresh ones for the remaining parts only. At part 38 Arjun swipes the app away, and iOS cancels the transfers. An hour later he opens the app, it reads a table showing 37 parts done, and it finishes the rest.
Total bytes sent twice: about 32 MB of 412 MB. Without part checkpoints, the swipe alone would have cost 296 MB.
Part 2Today's question
How do you move a huge file over a bad network from an app the user can close?
By the end of this lesson, you should be able to:
- Draw the multipart upload protocol
- Say what each OS allows in the background
- Explain why metadata and bytes travel separately
Part 3In detail
Step by step
- Copy and fingerprint. Copy the video into the app's own storage, computing its hash as you go, so it reads the file once.
- Ask the server to start. Send the hash and size. If the server already has this file, you're done. If not, it returns 52 presigned URLs.
- Send parts, record receipts. Send four parts at a time. After each one, write its ETag to a table on disk.
- Resume after any interruption. After a failure or restart, read the table and send only the parts without an ETag.
- Finish and verify. Send all part numbers and ETags. The server checks the hash and only then links the video to the job.
Separate the bytes from the record
The inspection syncs first through the normal outbox, and it just says a video is coming. The file follows on its own track. That way a slow upload never blocks Priya's 37 small edits.
The three questions, again
Where does it live? In app storage, with one row per part. Who runs the next step? WorkManager with a foreground service on Android, and a background URLSession on iOS. What if it runs twice? A part is safe to resend, and the file hash lets the server skip a file it already has.
Trust nothing until it's verified
The server checks the final hash before it links the video to the job. Until then the job shows uploading, not done. Tell the user the truth, because they will drive away from the site on the strength of that tick.
Progress that survives being killed
1CREATE TABLE transfer_part (2 file_id TEXT NOT NULL,3 part_no INTEGER NOT NULL,4 etag TEXT, -- NULL until this part succeeds5 PRIMARY KEY (file_id, part_no)6);7 8-- Arjun reopens the app: which parts still need sending?9SELECT part_no FROM transfer_part10 WHERE file_id = 'f-412' AND etag IS NULL11 ORDER BY part_no;Part checkpoints are to bytes what the outbox is to edits: durable progress that a retry can pick up.
The code, explained
This is the real code, and our tests run it. Read each block, then the notes under it.
Cutting a file into parts
This is simple arithmetic, and getting the last part wrong corrupts uploads.
1data class Part(val no: Int, val offset: Long, val length: Long)2 3fun planParts(size: Long, partSize: Long = 8 * MIB): List<Part> {4 val count = ((size + partSize - 1) / partSize).toInt()5 return List(count) { i ->6 val offset = i * partSize7 Part(i + 1, offset, minOf(partSize, size - offset))8 }9}(size + partSize - 1) / partSize- Rounds up, so 412 MB makes 52 parts of 8 MB.
minOf(partSize, size - offset)- The last part is usually shorter.
Part(i + 1, ...)- Object stores number parts from 1.
Compiled and tested by npm run test:mobile.
Sending parts that survive being killed
Arjun's upload. The receipts map stands for a table on disk, so a killed app loses almost nothing.
1/**2 * Sends only what the table says is missing. [receipts] stands for the transfer_part3 * table: a part number maps to its ETag once that part succeeds.4 */5class PartUploader(6 private val parts: List<Part>,7 private val receipts: MutableMap<Int, String>,8 private val freshUrls: suspend (List<Part>) -> Map<Int, String>,9 private val send: suspend (Part, url: String) -> String,10 private val parallel: Int = 4,11) {12 val remaining: List<Part> get() = parts.filter { it.no !in receipts }13 14 suspend fun run() {15 var urls = freshUrls(remaining)16 while (remaining.isNotEmpty()) {17 val outcomes = coroutineScope {18 remaining.take(parallel).map { part ->19 async { part to runCatching { send(part, urls.getValue(part.no)) } }20 }.awaitAll()21 }22 // Save every success first, so a failure next to it costs nothing.23 outcomes.forEach { (part, r) -> r.onSuccess { etag -> receipts[part.no] = etag } }24 val failure = outcomes.firstNotNullOfOrNull { (_, r) -> r.exceptionOrNull() }25 when (failure) {26 null -> Unit27 is UrlExpired -> urls = freshUrls(remaining) // only the parts still missing28 else -> throw failure29 }30 }31 }32 33 /** The body of POST /files/complete, in part order. */34 fun manifest(): List<Pair<Int, String>> = receipts.toSortedMap().map { (no, etag) -> no to etag }35}parts.filter { it.no !in receipts }- What is left. This one line is why resuming needs no special case: a fresh start and a resume run the same code.
runCatching { send(...) }- One failed part doesn't cancel the others in flight.
outcomes.forEach { ... receipts[...] = etag }- Saves every success before looking at failures, so finished work always counts.
is UrlExpired -> urls = freshUrls(remaining)- Fetches new URLs only for the parts still missing, then carries on.
else -> throw failure- Anything else ends this run. The next run picks up from the receipts.
Compiled and tested by npm run test:mobile.
What can go wrong
- The network switches mid-part
- That part fails and retries on its own. The others aren't affected.
- The URLs expire
- Request new URLs for the remaining parts only.
- The user swipes the app away on iOS
- iOS cancels the transfers. On next open, the table shows what's done, and sending continues.
- A slow video would hold up small edits
- Bytes travel separately, so edits keep syncing while the video uploads.
Decisions to defend
- How do bytes reach storage? Presigned URLs, straight to object storage, not upload through your API. At 48 TB a day, your API servers would become an expensive, fragile proxy.
Part 4Questions and answers
Answer out loud first, then open each one.
How large should a part be?
iOS killed the app and the background session continues. How does the app learn it finished?
Part 5Going further
The three questions, for today
- Where does it live?
- In app storage, with one row per part.
- Who runs the next step?
- WorkManager with a foreground service on Android, background URLSession on iOS.
- What if it runs twice?
- Parts can be resent safely, and the file hash skips what the server has.
Practise
- Build40 minResumable multipart uploader
The core of today. Test it by failing a part and killing it midway.
Optional reference pages
You don’t need these for the lesson. They hold more detail if you want it later.
You now know the whole system. Tomorrow you say it aloud, to a clock, as one 45-minute answer.