/Learn
Mobile systems at scale · Part 4 of 7

Uploading 4 GB on one bar of signal

Big uploads on mobile networks work as small, resumable, verifiable steps. Build them here, and learn what iOS and Android allow in the background.

04
FilesS3iOSAndroid

Mobile engineeringAll mobile pages

Arjun's 412 MB video can't fit in an outbox row. This post gives files their own track, using the same three questions. Start from the basics in Day 7 →

Study with AI

If you only read this

  1. Never push file bytes through your API. Presigned part URLs send them straight to S3 or equivalent.
  2. Checkpoint every part in SQLite. A kill costs at most one part.
  3. iOS needs a background URLSession with file-based tasks; Android needs WorkManager or a user-initiated data transfer job.
The one ruleHash it, split it into parts, send parts straight to object storage, checkpoint each one, and finish with a server-side check.

01Shape of the problem

A 4 GB video on a moving train sees minutes of signal, then none, then a different network. A single HTTP request cannot survive that. The design has to assume the upload will be interrupted dozens of times and cost almost nothing each time.

Four ideas combine to make it work: break the file into independent parts, upload each part with its own URL, record which parts are done, and verify the whole thing at the end.

02Protocol, end to end

AppTransfer managerYour APIObject storageEnqueue upload (file path)SHA-256 while reading, in chunksPOST /uploads {sha256, size, parts}CreateMultipartUploadupload_id + presigned URLsPUT part 1 (8 MiB)ETagCheckpoint part 1 in SQLiteProcess killed. Hours later the OS wakes the job.GET /uploads/id (what is done?)ListPartsparts 1-12 donePUT parts 13…nPOST /uploads/id/complete {etags}CompleteMultipartUploadverified · linked to record
Solid arrows are requests; dashed arrows are responses.

Notice who talks to whom. Your API never sees the bytes. It creates the upload, hands out time-limited URLs and, at the end, completes and verifies it. That is what lets 5 million phones upload at once without your servers becoming the bottleneck.

03Hashing, parts and limits

Hash the file while reading it in chunks, so memory stays flat. Send the SHA-256 at the start. The server can answer 'already have it', which turns a duplicate upload into a no-op, and uses the hash at the end to verify integrity.

Choose the part size from the file size and the network. S3 multipart parts must be at least 5 MiB (except the last) and an upload can have at most 10,000 parts, so 8 to 16 MiB works well for gigabyte files: 4 GB at 8 MiB is 512 parts. Smaller parts lose less when interrupted and larger parts have less overhead, so adapt: start at 8 MiB and move up on Wi-Fi.

Upload state the transfer manager persists

upload-state-transfer.sqlSQL
1CREATE TABLE upload (2  id TEXT PRIMARY KEY, path TEXT NOT NULL, size INTEGER NOT NULL, sha256 TEXT NOT NULL,3  upload_id TEXT, part_size INTEGER NOT NULL, state TEXT NOT NULL,  -- queued|hashing|uploading|completing|done|failed4  attempts INTEGER NOT NULL DEFAULT 0, next_try_at INTEGER5);6CREATE TABLE upload_part (7  upload_id TEXT NOT NULL, n INTEGER NOT NULL, etag TEXT, bytes INTEGER NOT NULL,8  PRIMARY KEY (upload_id, n)          -- a row with an etag is a finished part9);
9 linesUTF-8SQL
Check you get rightVerify with the server, not only the client. After complete, the server computes or compares the checksum and only then marks the file usable. Otherwise a corrupted part can sit in your storage with a green tick.

04Android: who is allowed to run this

  • WorkManager runs resumable work with constraints (network, battery) and survives reboots. Use a foreground service with a progress notification for work the user is watching.
  • Android 14 added user-initiated data transfer jobs for large transfers the user started. They are exempt from the usual background limits, require a notification, and fit a 4 GB upload.
  • Doze and app standby delay jobs but do not cancel them. Some manufacturers add their own task killers, so treat background execution as best effort and resume instantly when the app opens.
  • Use OkHttp with a streaming request body for each part, reading from the file with an offset. Do not load a part into memory.

05iOS: the background URLSession rules

  • Large background uploads must be file-based: URLSession.uploadTask(with:fromFile:) in a background configuration. The system process performs the transfer and keeps going when your app is suspended or terminated.
  • Create one task per part. Write each part to its own temporary file first (or use a file with byte ranges through a custom stream where supported), because tasks upload a file, not a slice of memory.
  • When a task finishes, iOS relaunches your app in the background and calls the application delegate's handler for the session identifier. Reconnect the session with the same identifier, handle the delegate callbacks, then call the completion handler.
  • Set isDiscretionary for non-urgent uploads so the system picks good moments, and decide explicitly about allowsCellularAccess and allowsExpensiveNetworkAccess.
  • BGTaskScheduler is for deciding what to do next (finalising, retrying, hashing). It is not the transfer engine.
Interview lineOn iOS the upload is not my process. It belongs to the system. My job is to persist intent, hand over small file-based tasks and reconcile when I am relaunched.

06Downloads, Range requests and tus

Resumable downloads use HTTP Range requests. Send If-Range with the ETag you saw, so that if the file changed on the server you restart instead of splicing two versions together. Write to a .part file, verify the hash, then rename atomically.

If you do not want to run multipart logic yourself, the open tus protocol and managed services such as Cloudflare Stream or Mux handle resumable uploads with a standard client. Build it yourself when you need control over cost, residency or integrity checks; adopt tus or a vendor when upload is not your differentiator.

Resumable download sketch

resumable-download-sketch.ktKotlin
1val have = partFile.length()2val req = Request.Builder().url(url)3    .header("Range", "bytes=" + have + "-")4    .header("If-Range", savedEtag)            // restart if the object changed5    .build()6client.newCall(req).execute().use { resp ->7    when (resp.code) {8        206 -> append(resp.body!!.source(), partFile)   // continue9        200 -> { partFile.delete(); append(resp.body!!.source(), partFile) } // changed: start over10        416 -> verifyAndRename(partFile)                // already complete11        else -> throw RetryableHttp(resp.code)12    }13}
13 linesUTF-8Kotlin

07Tested code, explained

Cutting a file into parts

This is simple arithmetic, and getting the last part wrong corrupts uploads.

PartUploader.ktKotlin
1data class Part(val no: Int, val offset: Long, val length: Long)2 3fun planParts(size: Long, partSize: Long = 8 * MIB): List<Part> {4  val count = ((size + partSize - 1) / partSize).toInt()5  return List(count) { i ->6    val offset = i * partSize7    Part(i + 1, offset, minOf(partSize, size - offset))8  }9}
9 linesUTF-8Kotlin
(size + partSize - 1) / partSize
Rounds up, so 412 MB makes 52 parts of 8 MB.
minOf(partSize, size - offset)
The last part is usually shorter.
Part(i + 1, ...)
Object stores number parts from 1.

Compiled and tested by npm run test:mobile.

Sending parts that survive being killed

Arjun's upload. The receipts map stands for a table on disk, so a killed app loses almost nothing.

PartUploader.ktKotlin
1/**2 * Sends only what the table says is missing. [receipts] stands for the transfer_part3 * table: a part number maps to its ETag once that part succeeds.4 */5class PartUploader(6  private val parts: List<Part>,7  private val receipts: MutableMap<Int, String>,8  private val freshUrls: suspend (List<Part>) -> Map<Int, String>,9  private val send: suspend (Part, url: String) -> String,10  private val parallel: Int = 4,11) {12  val remaining: List<Part> get() = parts.filter { it.no !in receipts }13 14  suspend fun run() {15    var urls = freshUrls(remaining)16    while (remaining.isNotEmpty()) {17      val outcomes = coroutineScope {18        remaining.take(parallel).map { part ->19          async { part to runCatching { send(part, urls.getValue(part.no)) } }20        }.awaitAll()21      }22      // Save every success first, so a failure next to it costs nothing.23      outcomes.forEach { (part, r) -> r.onSuccess { etag -> receipts[part.no] = etag } }24      val failure = outcomes.firstNotNullOfOrNull { (_, r) -> r.exceptionOrNull() }25      when (failure) {26        null -> Unit27        is UrlExpired -> urls = freshUrls(remaining)    // only the parts still missing28        else -> throw failure29      }30    }31  }32 33  /** The body of POST /files/complete, in part order. */34  fun manifest(): List<Pair<Int, String>> = receipts.toSortedMap().map { (no, etag) -> no to etag }35}
35 linesUTF-8Kotlin
parts.filter { it.no !in receipts }
What is left. This one line is why resuming needs no special case: a fresh start and a resume run the same code.
runCatching { send(...) }
One failed part doesn't cancel the others in flight.
outcomes.forEach { ... receipts[...] = etag }
Saves every success before looking at failures, so finished work always counts.
is UrlExpired -> urls = freshUrls(remaining)
Fetches new URLs only for the parts still missing, then carries on.
else -> throw failure
Anything else ends this run. The next run picks up from the receipts.

Compiled and tested by npm run test:mobile.

08Interview questions

Why multipart with presigned URLs instead of one POST to your API?
Bytes bypass your servers, so cost and capacity do not depend on your API. Each part is independently retriable and resumable, and URLs expire, limiting abuse.
The app is killed at part 301 of 512. What happens?
On next run the transfer manager reads its checkpoint, asks the server which parts it has, uploads the missing ones and completes. At most one in-flight part is repeated.
How do you stop a user uploading the same video twice?
Send the SHA-256 first. If the server already has that content for the tenant, it links the existing object and returns done.
Why can't iOS upload a part straight from memory in the background?
Background sessions hand the transfer to a system daemon, which needs a file to read from. So write parts to temporary files and upload them with file-based tasks.
How do you handle a flaky captive-portal Wi-Fi that returns 200 with HTML?
Validate response status and expected headers such as the ETag. Treat anything unexpected as a retryable failure and back off with jitter.

Practise it:

FilesS3iOSAndroid
Written by

Rizvan

Builds mobile and AI systems at scale and writes the series he wished existed before his own principal loop: one rule, one analogy, something to break, and the interview kit.

All 7 parts
AI Engineer · written by Rizvan · part 4 of 7