Basics
What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.
1.1Definition
The one-paragraph summary, then where its responsibility starts and stops.
Transfer managers move large files between phone and object storage. They hash, split and send parts that succeed or fail alone, record progress after each, and survive network switches, app kills and reboots.
- Copy the source into app storage and hash it while copying
- Ask the server whether the content already exists
- Plan parts and upload several in parallel
- Persist each part’s ETag the moment it lands
- Refresh expired URLs and resume after any interruption
- Download with Range requests, verify and rename atomically
- Sync records (the sync engine does)
- Decide product rules like Wi-Fi only (it reads them as constraints)
- Hold file bytes in memory or in JavaScript
- Trust a file path that the system can revoke
1.2Key concepts
Ideas this deep dive relies on, in one sentence each.
Splitting a file into independently uploaded and retriable parts, assembled by the store.
A time-limited URL that lets a client talk to object storage directly.
Android library for deferrable, guaranteed background work.
iOS transfer session performed by a system daemon and surviving app termination.
DeepSpeed's sharding of optimiser state, gradients and weights across GPUs.
1.3Real-life run
A concrete example first, so the mechanics have something to attach to.
Arjun records a 412 MB walkthrough video of a pump room and taps Submit. The transfer manager copies the file into app storage while computing its SHA-256, so it reads the file once. It asks the server with POST /files/init; the hash is new, so the server starts a multipart upload and returns 52 presigned part URLs of 8 MB each, valid for 15 minutes.
Four parts upload at a time. After each one, the ETag goes into the transfer_part table. At part 20 the phone moves from Wi-Fi to 4G: three in-flight parts fail, retry after a short jittered delay and succeed. At part 31 the URLs have expired, the server returns 403, and the manager asks for fresh URLs for parts 31 to 52 only.
Arjun’s phone is an iPhone and he swipes the app away at part 38. iOS cancels the background transfers. When he opens the app an hour later, the manager reads the table, sees 37 parts done, requests URLs for parts 38 to 52 and finishes. It then sends POST /files/complete with all 52 part numbers and ETags; the server verifies the hash and links the video to the inspection.
Total bytes re-sent because of the network switch and the swipe: about 32 MB out of 412 MB. Without part checkpoints, the swipe alone would have cost 296 MB.
Question
Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.
2.1Target questions
Keep them in mind while you read the detail.
- Why copy the file before uploading?
- How do you choose part size?
- Why not one background task for the whole file on iOS?
- How do you prevent duplicate storage when users re-upload?
Model answers are in part 4, after the detail that supports them.
2.2Interview rounds
System design rounds where this component is part of a strong answer.
Detailed answer
How it actually works: the components, the states it moves through, a full run, and the code.
3.1Anatomy
The parts and how they connect.
- Public API
- enqueue(uri, owner, priority) returns a transfer ID immediately; observe(id) streams progress from the database; cancel(id) marks it cancelled so nothing restarts it.
- Preparer
- Copies the file into app storage, because content URIs and photo-library references can disappear, and computes SHA-256 in the same pass. Optional transcoding happens here, before any bytes leave the phone.
- Part planner
- Chooses part size from file size and network: at least 5 MB (the S3 minimum for all but the last part), usually 8–16 MB, and large enough to stay under 10,000 parts.
- URL broker
- Owns the upload ID and presigned URLs. It refreshes URLs in bulk when they are near expiry or a PUT returns 403, so expiry never fails a transfer.
- Worker pool
- Runs two to four part uploads concurrently, fewer on cellular or low battery. It records each part’s ETag in its own small transaction.
- Downloader
- Uses Range requests to resume, writes to a temporary file, verifies the hash, and renames into place so a half-written file is never visible.
3.2States
| State | Meaning |
|---|---|
| Queued | Accepted and persisted. Waiting for constraints such as Wi-Fi or charging. |
| Preparing | Copying, hashing, optionally compressing. |
| Checking | Asking the server whether the content already exists. |
| Uploading | Parts in flight; progress = parts done ÷ total. |
| Paused | System stopped us, network lost, or constraints no longer met. Resumes automatically. |
| Completing | All parts done; asking the server to assemble and verify. |
| Done | Linked to its record. Local copy can be evicted later. |
| Failed | Permanent error such as file deleted or hash mismatch; shown to the user. |
| Cancelled | User cancelled. Server multipart upload aborted. |
3.3Run loop
One run, step by step.
- The app computes the hash once, during the copy, and reuses it for deduplication and final verification.
- Each part commits independently, so any interruption costs only the parts in flight.
- Expired URLs are a normal event, refreshed in bulk, never a failure.
3.4Code
Working code for the core of it.
Transfer tables (SQLite)
1CREATE TABLE file_transfer (2 id TEXT PRIMARY KEY, owner_entity TEXT, owner_id TEXT,3 local_path TEXT NOT NULL, size INTEGER NOT NULL, sha256 TEXT NOT NULL, mime TEXT,4 direction TEXT NOT NULL, -- up | down5 priority TEXT NOT NULL, -- user | background6 state TEXT NOT NULL, -- queued | preparing | uploading | paused | completing | done | failed | cancelled7 upload_id TEXT, part_size INTEGER, part_count INTEGER,8 attempts INTEGER DEFAULT 0, next_retry_at INTEGER, last_error TEXT, remote_key TEXT,9 created_at INTEGER NOT NULL10);11CREATE TABLE transfer_part (12 transfer_id TEXT NOT NULL, part_no INTEGER NOT NULL,13 url TEXT, url_expires_at INTEGER, etag TEXT, done INTEGER DEFAULT 0,14 PRIMARY KEY (transfer_id, part_no)15);16CREATE INDEX transfer_ready ON file_transfer(state, priority, next_retry_at);Hash while copying (Kotlin)
1fun copyAndHash(src: InputStream, dst: File): String {2 val md = MessageDigest.getInstance("SHA-256")3 val buf = ByteArray(1 shl 20) // 1 MB buffer, constant memory4 dst.outputStream().buffered().use { out ->5 src.use { input ->6 while (true) {7 val n = input.read(buf); if (n < 0) break8 md.update(buf, 0, n); out.write(buf, 0, n)9 }10 }11 }12 return md.digest().joinToString("") { "%02x".format(it) }13}Part planning
1fun planParts(size: Long, onCellular: Boolean): Plan {2 val min = 5L * 1024 * 1024 // S3 minimum except last part3 val preferred = if (onCellular) 8L * 1024 * 1024 else 16L * 1024 * 10244 val forLimit = ceil(size / 9_000.0).toLong() // stay well under 10,000 parts5 val partSize = maxOf(min, preferred, forLimit)6 val count = ((size + partSize - 1) / partSize).toInt()7 return Plan(partSize, count)8}Streaming one part without loading it (OkHttp)
1class FileSliceBody(private val file: File, private val offset: Long, private val length: Long) : RequestBody() {2 override fun contentType() = "application/octet-stream".toMediaType()3 override fun contentLength() = length4 override fun writeTo(sink: BufferedSink) {5 RandomAccessFile(file, "r").use { raf ->6 raf.seek(offset)7 val buf = ByteArray(64 * 1024); var left = length8 while (left > 0) {9 val n = raf.read(buf, 0, minOf(buf.size.toLong(), left).toInt())10 sink.write(buf, 0, n); left -= n11 }12 }13 }14}15 16suspend fun uploadPart(t: Transfer, p: Part): String {17 val req = Request.Builder().url(p.url).put(FileSliceBody(t.file, (p.no - 1) * t.partSize, p.length)).build()18 client.newCall(req).await().use { res ->19 if (res.code == 403) throw UrlExpired(p.no) // broker refreshes, then retry20 if (!res.isSuccessful) throw IOException("part " + p.no + " failed: " + res.code)21 return res.header("ETag") ?: error("missing ETag")22 }23}Android: one WorkManager job per part
1fun enqueuePart(ctx: Context, transferId: String, part: Int) {2 val req = OneTimeWorkRequestBuilder<PartUploadWorker>()3 .setInputData(workDataOf("transfer" to transferId, "part" to part))4 .setConstraints(Constraints.Builder().setRequiredNetworkType(NetworkType.CONNECTED).setRequiresStorageNotLow(true).build())5 .setBackoffCriteria(BackoffPolicy.EXPONENTIAL, 10, TimeUnit.SECONDS)6 .addTag("transfer:$transferId") // cancel a whole transfer by tag7 .build()8 WorkManager.getInstance(ctx).enqueueUniqueWork("part:$transferId:$part", ExistingWorkPolicy.KEEP, req)9}10 11class PartUploadWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ctx, params) {12 override suspend fun doWork(): Result {13 val id = inputData.getString("transfer") ?: return Result.failure()14 val no = inputData.getInt("part", -1)15 val (t, p) = transfers.load(id, no) ?: return Result.success() // transfer was cancelled meanwhile16 if (p.etag != null) return Result.success() // already uploaded: safe to rerun17 setForeground(foregroundInfo(t)) // long upload: visible, not killed18 return try {19 transfers.recordPart(id, no, etag = uploadPart(t, p)) // durable before reporting success20 if (transfers.allPartsDone(id)) TransferTriggers.complete(applicationContext, id)21 Result.success()22 } catch (e: UrlExpired) {23 transfers.refreshUrl(id, no); Result.retry() // presigned URL timed out24 } catch (e: IOException) {25 if (runAttemptCount < 8) Result.retry() else Result.failure()26 }27 }28}Resuming a download
1val have = tmp.length()2val req = Request.Builder().url(signedUrl).header("Range", "bytes=" + have + "-").build()3client.newCall(req).await().use { res ->4 require(res.code == 206 || (have == 0L && res.code == 200)) { "server ignored Range" }5 FileOutputStream(tmp, true).use { out -> res.body!!.byteStream().copyTo(out) }6}7check(sha256(tmp) == expectedHash) { "corrupt download" }8tmp.renameTo(finalFile) // atomic on the same filesystem3.5Tested code, explained
A working implementation, then what each part does and why.
Cutting a file into parts
This is simple arithmetic, and getting the last part wrong corrupts uploads.
1data class Part(val no: Int, val offset: Long, val length: Long)2 3fun planParts(size: Long, partSize: Long = 8 * MIB): List<Part> {4 val count = ((size + partSize - 1) / partSize).toInt()5 return List(count) { i ->6 val offset = i * partSize7 Part(i + 1, offset, minOf(partSize, size - offset))8 }9}(size + partSize - 1) / partSize- Rounds up, so 412 MB makes 52 parts of 8 MB.
minOf(partSize, size - offset)- The last part is usually shorter.
Part(i + 1, ...)- Object stores number parts from 1.
Compiled and tested by npm run test:mobile.
Sending parts that survive being killed
Arjun's upload. The receipts map stands for a table on disk, so a killed app loses almost nothing.
1/**2 * Sends only what the table says is missing. [receipts] stands for the transfer_part3 * table: a part number maps to its ETag once that part succeeds.4 */5class PartUploader(6 private val parts: List<Part>,7 private val receipts: MutableMap<Int, String>,8 private val freshUrls: suspend (List<Part>) -> Map<Int, String>,9 private val send: suspend (Part, url: String) -> String,10 private val parallel: Int = 4,11) {12 val remaining: List<Part> get() = parts.filter { it.no !in receipts }13 14 suspend fun run() {15 var urls = freshUrls(remaining)16 while (remaining.isNotEmpty()) {17 val outcomes = coroutineScope {18 remaining.take(parallel).map { part ->19 async { part to runCatching { send(part, urls.getValue(part.no)) } }20 }.awaitAll()21 }22 // Save every success first, so a failure next to it costs nothing.23 outcomes.forEach { (part, r) -> r.onSuccess { etag -> receipts[part.no] = etag } }24 val failure = outcomes.firstNotNullOfOrNull { (_, r) -> r.exceptionOrNull() }25 when (failure) {26 null -> Unit27 is UrlExpired -> urls = freshUrls(remaining) // only the parts still missing28 else -> throw failure29 }30 }31 }32 33 /** The body of POST /files/complete, in part order. */34 fun manifest(): List<Pair<Int, String>> = receipts.toSortedMap().map { (no, etag) -> no to etag }35}parts.filter { it.no !in receipts }- What is left. This one line is why resuming needs no special case: a fresh start and a resume run the same code.
runCatching { send(...) }- One failed part doesn't cancel the others in flight.
outcomes.forEach { ... receipts[...] = etag }- Saves every success before looking at failures, so finished work always counts.
is UrlExpired -> urls = freshUrls(remaining)- Fetches new URLs only for the parts still missing, then carries on.
else -> throw failure- Anything else ends this run. The next run picks up from the receipts.
Compiled and tested by npm run test:mobile.
3.6Platforms
Android
- WorkManager unique work per transfer (upload- plus the ID) with network and battery constraints for background priority.
- For uploads the user started, a user-initiated data transfer job on Android 14 and later, with a progress notification and cancel action.
- OkHttp with a FileSliceBody so memory stays flat regardless of file size.
- Media3 Transformer for on-device video compression before upload.
iOS
- Background URLSession with one upload task per part file; the system keeps transferring while the app is suspended.
- handleEventsForBackgroundURLSession in the app delegate, and the stored completion handler called after events are delivered.
- BGContinuedProcessingTask (iOS 26+) or a short background task for the hashing and compression step.
- AVAssetExportSession for video compression.
React Native
- A native module with enqueueUpload(uri, priority) that returns a transfer ID; progress is read from SQLite.
- Pass file URIs only; bytes never enter the JS heap.
- react-native-background-upload or a custom module wrapping the native pieces above.
Q&A
The questions from part 2, now with model answers. Try each one out loud before opening it.
4.1Interview questions
Why copy the file before uploading?
How do you choose part size?
Why not one background task for the whole file on iOS?
How do you prevent duplicate storage when users re-upload?
Extra depth
Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.
5.1Technology choices
Show detailsHide details
| Need | Options | When to pick |
|---|---|---|
| Upload protocol | S3 multipart with presigned URLs · GCS resumable uploads · Azure block blobs · tus | Use your cloud’s native protocol; tus when you need a vendor-neutral, open resumable protocol. |
| Integrity | SHA-256 of the whole file · per-part checksums (S3 supports CRC32C, SHA-256 and others) | Whole-file hash for dedupe and evidence; per-part checksums to catch corruption in transit. |
| Concurrency | 1 · 2–4 · adaptive | Two on cellular, four on Wi-Fi; drop to one when the battery is low. |
| Compression | Media3 Transformer · AVAssetExportSession · none | Compress video when the product allows; never re-compress photos used as evidence. |
| Download delivery | CDN signed URLs · direct storage URLs | CDN with content-addressed keys for anything downloaded more than once. |
5.2How real products do it
Show detailsHide details
Summaries of what these teams and projects have described publicly. Check their own write-ups for detail.
| Who | What they describe |
|---|---|
| Google APIs | Publish a resumable upload protocol: create an upload session, send the file in chunks, and query the server for how many bytes it received before resuming. |
| tus | An open protocol for resumable uploads over HTTP, with server and client libraries in many languages; Vimeo is among its well-known adopters. |
| Dropbox API | Exposes upload sessions (start, append, finish) so large files are sent in pieces and committed at the end. |
| Amazon S3 | Multipart upload: up to 10,000 parts, each at least 5 MB except the last, assembled by a final complete call; incomplete uploads should be cleaned up with a lifecycle rule. |
5.3Metrics to emit
Show detailsHide details
| Metric | Why it matters |
|---|---|
| transfer.completion_rate | The headline number: share of enqueued files that reach done. |
| bytes_resent ÷ bytes_total | Shows whether checkpoints and part size are working. |
| time_to_done (p50, p95) by network | Sets expectations and catches regressions. |
| part.failure by reason | 403 expiry, timeouts and 5xx need different fixes. |
| dedupe.hit_rate | How often the server already had the bytes, so the app skipped the upload. |
| paused.age | Transfers stuck paused for days usually mean a missing constraint or a killed OEM job. |
5.4Edge cases
Show detailsHide details
| Case | How it is handled |
|---|---|
| Source file deleted or permission revoked | Already copied into app storage at enqueue time, so the transfer continues. |
| File changed after enqueue | Hash mismatch on complete; transfer marked failed and re-enqueued from the new file. |
| Presigned URL expired | 403 triggers a bulk refresh for remaining parts; not counted as a failure. |
| Disk nearly full | Refuse new copies with a clear message; evict uploaded files first. |
| Complete called twice | Server treats complete as idempotent by upload ID. |
| Abandoned upload on the server | Storage lifecycle rule aborts incomplete multipart uploads after a few days. |