/Tracks

Two-week programme · Day 7 of 14

Arjun's 412 MB video

Study with AI

Part 1Basics

An edit is a few bytes. A video can be hundreds of megabytes, and a phone may lose signal or get closed partway through sending it. Starting over each time would waste hours and data.

So we cut the file into small numbered parts and send them one by one. Like moving house box by box: if the van breaks down, only the boxes still on it need another trip.

Words you need today

Object storage
A service built to hold large files, such as Amazon S3.
Multipart upload
Sending a file as separate parts that get joined at the end.
Presigned URL
A temporary link that lets the phone upload straight to storage without going through your server.
ETag
A receipt returned for each uploaded part. You need all of them to finish the upload.
Hash
A short fingerprint of a file. Same file, same fingerprint.
Background URLSession
An iOS service that keeps transferring files even after your app has been suspended.

The story

Arjun, on Priya's crew, films a pump room and taps Submit. The app copies the 412 MB file into its own storage, hashing it on the way, and asks the server to start. The server returns 52 presigned URLs of 8 MB each, so the bytes go straight to object storage and never touch your API.

At part 20 the phone moves from Wi-Fi to 4G. Three parts fail and retry. At part 31 the URLs expire, so the app fetches fresh ones for the remaining parts only. At part 38 Arjun swipes the app away, and iOS cancels the transfers. An hour later he opens the app, it reads a table showing 37 parts done, and it finishes the rest.

Total bytes sent twice: about 32 MB of 412 MB. Without part checkpoints, the swipe alone would have cost 296 MB.

Part 2Today's question

How do you move a huge file over a bad network from an app the user can close?

By the end of this lesson, you should be able to:

  • Draw the multipart upload protocol
  • Say what each OS allows in the background
  • Explain why metadata and bytes travel separately

Part 3In detail

Step by step

  1. Copy and fingerprint. Copy the video into the app's own storage, computing its hash as you go, so it reads the file once.
  2. Ask the server to start. Send the hash and size. If the server already has this file, you're done. If not, it returns 52 presigned URLs.
  3. Send parts, record receipts. Send four parts at a time. After each one, write its ETag to a table on disk.
  4. Resume after any interruption. After a failure or restart, read the table and send only the parts without an ETag.
  5. Finish and verify. Send all part numbers and ETags. The server checks the hash and only then links the video to the job.

Separate the bytes from the record

The inspection syncs first through the normal outbox, and it just says a video is coming. The file follows on its own track. That way a slow upload never blocks Priya's 37 small edits.

The three questions, again

Where does it live? In app storage, with one row per part. Who runs the next step? WorkManager with a foreground service on Android, and a background URLSession on iOS. What if it runs twice? A part is safe to resend, and the file hash lets the server skip a file it already has.

Trust nothing until it's verified

The server checks the final hash before it links the video to the job. Until then the job shows uploading, not done. Tell the user the truth, because they will drive away from the site on the strength of that tick.

Progress that survives being killed

progress-that-survives.sqlSQL
1CREATE TABLE transfer_part (2  file_id TEXT NOT NULL,3  part_no INTEGER NOT NULL,4  etag    TEXT,               -- NULL until this part succeeds5  PRIMARY KEY (file_id, part_no)6);7 8-- Arjun reopens the app: which parts still need sending?9SELECT part_no FROM transfer_part10 WHERE file_id = 'f-412' AND etag IS NULL11 ORDER BY part_no;
11 linesUTF-8SQL

Part checkpoints are to bytes what the outbox is to edits: durable progress that a retry can pick up.

The code, explained

This is the real code, and our tests run it. Read each block, then the notes under it.

Cutting a file into parts

This is simple arithmetic, and getting the last part wrong corrupts uploads.

PartUploader.ktKotlin
1data class Part(val no: Int, val offset: Long, val length: Long)2 3fun planParts(size: Long, partSize: Long = 8 * MIB): List<Part> {4  val count = ((size + partSize - 1) / partSize).toInt()5  return List(count) { i ->6    val offset = i * partSize7    Part(i + 1, offset, minOf(partSize, size - offset))8  }9}
9 linesUTF-8Kotlin
(size + partSize - 1) / partSize
Rounds up, so 412 MB makes 52 parts of 8 MB.
minOf(partSize, size - offset)
The last part is usually shorter.
Part(i + 1, ...)
Object stores number parts from 1.

Compiled and tested by npm run test:mobile.

Sending parts that survive being killed

Arjun's upload. The receipts map stands for a table on disk, so a killed app loses almost nothing.

PartUploader.ktKotlin
1/**2 * Sends only what the table says is missing. [receipts] stands for the transfer_part3 * table: a part number maps to its ETag once that part succeeds.4 */5class PartUploader(6  private val parts: List<Part>,7  private val receipts: MutableMap<Int, String>,8  private val freshUrls: suspend (List<Part>) -> Map<Int, String>,9  private val send: suspend (Part, url: String) -> String,10  private val parallel: Int = 4,11) {12  val remaining: List<Part> get() = parts.filter { it.no !in receipts }13 14  suspend fun run() {15    var urls = freshUrls(remaining)16    while (remaining.isNotEmpty()) {17      val outcomes = coroutineScope {18        remaining.take(parallel).map { part ->19          async { part to runCatching { send(part, urls.getValue(part.no)) } }20        }.awaitAll()21      }22      // Save every success first, so a failure next to it costs nothing.23      outcomes.forEach { (part, r) -> r.onSuccess { etag -> receipts[part.no] = etag } }24      val failure = outcomes.firstNotNullOfOrNull { (_, r) -> r.exceptionOrNull() }25      when (failure) {26        null -> Unit27        is UrlExpired -> urls = freshUrls(remaining)    // only the parts still missing28        else -> throw failure29      }30    }31  }32 33  /** The body of POST /files/complete, in part order. */34  fun manifest(): List<Pair<Int, String>> = receipts.toSortedMap().map { (no, etag) -> no to etag }35}
35 linesUTF-8Kotlin
parts.filter { it.no !in receipts }
What is left. This one line is why resuming needs no special case: a fresh start and a resume run the same code.
runCatching { send(...) }
One failed part doesn't cancel the others in flight.
outcomes.forEach { ... receipts[...] = etag }
Saves every success before looking at failures, so finished work always counts.
is UrlExpired -> urls = freshUrls(remaining)
Fetches new URLs only for the parts still missing, then carries on.
else -> throw failure
Anything else ends this run. The next run picks up from the receipts.

Compiled and tested by npm run test:mobile.

What can go wrong

The network switches mid-part
That part fails and retries on its own. The others aren't affected.
The URLs expire
Request new URLs for the remaining parts only.
The user swipes the app away on iOS
iOS cancels the transfers. On next open, the table shows what's done, and sending continues.
A slow video would hold up small edits
Bytes travel separately, so edits keep syncing while the video uploads.

Decisions to defend

  • How do bytes reach storage? Presigned URLs, straight to object storage, not upload through your API. At 48 TB a day, your API servers would become an expensive, fragile proxy.

Part 4Questions and answers

Answer out loud first, then open each one.

How large should a part be?
Large enough to limit requests and small enough that a failure costs little. 8 MB is a common start on mobile. Object stores also cap the number of parts, so very large files need larger parts.
iOS killed the app and the background session continues. How does the app learn it finished?
The system relaunches the app in the background and delivers the session's events. If the user force-quit, iOS cancels the transfers, so the app resumes from the table on next open.

Part 5Going further

The three questions, for today

Where does it live?
In app storage, with one row per part.
Who runs the next step?
WorkManager with a foreground service on Android, background URLSession on iOS.
What if it runs twice?
Parts can be resent safely, and the file hash skips what the server has.

Practise

Optional reference pages

You don’t need these for the lesson. They hold more detail if you want it later.

You now know the whole system. Tomorrow you say it aloud, to a clock, as one 45-minute answer.

← Day 6: Inside the push, and out to Dev's tablet

AI Engineer · Day 7 of 14