/Learn
Deep dives · Client · low-level design

The transfer manager

LearnDeep diveTier 2 · MechanismsAnalyze20 minMobile systems

You will be able to: explain the transfer manager and answer its interview questions.

Prove it: Offline-first inspection app with huge uploads

Mobile engineeringReact Native in one place11 of 22Next: The server sync service →

Arjun's upload in full: the Wi-Fi switch, the expired URLs and the swipe. The part table here does for bytes what the outbox does for edits. Start from the basics in Day 7 →

Study with AI

Part 1 of 5

Basics

What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.

1.1Definition

The one-paragraph summary, then where its responsibility starts and stops.

Transfer managers move large files between phone and object storage. They hash, split and send parts that succeed or fail alone, record progress after each, and survive network switches, app kills and reboots.

Its job
  • Copy the source into app storage and hash it while copying
  • Ask the server whether the content already exists
  • Plan parts and upload several in parallel
  • Persist each part’s ETag the moment it lands
  • Refresh expired URLs and resume after any interruption
  • Download with Range requests, verify and rename atomically
Not its job
  • Sync records (the sync engine does)
  • Decide product rules like Wi-Fi only (it reads them as constraints)
  • Hold file bytes in memory or in JavaScript
  • Trust a file path that the system can revoke

1.2Key concepts

Ideas this deep dive relies on, in one sentence each.

01Multipart upload

Splitting a file into independently uploaded and retriable parts, assembled by the store.

02Presigned URL

A time-limited URL that lets a client talk to object storage directly.

03WorkManager

Android library for deferrable, guaranteed background work.

04Background URLSession

iOS transfer session performed by a system daemon and surviving app termination.

05ZeRO

DeepSpeed's sharding of optimiser state, gradients and weights across GPUs.

1.3Real-life run

A concrete example first, so the mechanics have something to attach to.

Arjun records a 412 MB walkthrough video of a pump room and taps Submit. The transfer manager copies the file into app storage while computing its SHA-256, so it reads the file once. It asks the server with POST /files/init; the hash is new, so the server starts a multipart upload and returns 52 presigned part URLs of 8 MB each, valid for 15 minutes.

Four parts upload at a time. After each one, the ETag goes into the transfer_part table. At part 20 the phone moves from Wi-Fi to 4G: three in-flight parts fail, retry after a short jittered delay and succeed. At part 31 the URLs have expired, the server returns 403, and the manager asks for fresh URLs for parts 31 to 52 only.

Arjun’s phone is an iPhone and he swipes the app away at part 38. iOS cancels the background transfers. When he opens the app an hour later, the manager reads the table, sees 37 parts done, requests URLs for parts 38 to 52 and finishes. It then sends POST /files/complete with all 52 part numbers and ETags; the server verifies the hash and links the video to the inspection.

Total bytes re-sent because of the network switch and the swipe: about 32 MB out of 412 MB. Without part checkpoints, the swipe alone would have cost 296 MB.

Part 2 of 5

Question

Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.

2.1Target questions

Keep them in mind while you read the detail.

  1. Why copy the file before uploading?
  2. How do you choose part size?
  3. Why not one background task for the whole file on iOS?
  4. How do you prevent duplicate storage when users re-upload?

Model answers are in part 4, after the detail that supports them.

2.2Interview rounds

System design rounds where this component is part of a strong answer.

Part 3 of 5

Detailed answer

How it actually works: the components, the states it moves through, a full run, and the code.

3.1Anatomy

The parts and how they connect.

TRANSFER MANAGER (NATIVE)SERVER AND STORAGEPublic APIenqueue · cancel ·observeTransfer queuepriority ·constraintsTransfer DBfiles · parts ·statePreparercopy · hash ·compressPart plannersize · count ·rangesWorker pool2–4 parts in flightURL brokerpresign · refreshUploaderPUT part → ETagDownloaderRange GET · verifyUpload APIinit · completeObject storagemultipartCDNsigned URLsenqueuepersistfilesha256 + sizepartspartrangeETag savedinitURLPUTGET Range
ClientServiceData store
Boxes are components; arrows show calls and data flow.
Public API
enqueue(uri, owner, priority) returns a transfer ID immediately; observe(id) streams progress from the database; cancel(id) marks it cancelled so nothing restarts it.
Preparer
Copies the file into app storage, because content URIs and photo-library references can disappear, and computes SHA-256 in the same pass. Optional transcoding happens here, before any bytes leave the phone.
Part planner
Chooses part size from file size and network: at least 5 MB (the S3 minimum for all but the last part), usually 8–16 MB, and large enough to stay under 10,000 parts.
URL broker
Owns the upload ID and presigned URLs. It refreshes URLs in bulk when they are near expiry or a PUT returns 403, so expiry never fails a transfer.
Worker pool
Runs two to four part uploads concurrently, fewer on cellular or low battery. It records each part’s ETag in its own small transaction.
Downloader
Uses Range requests to resume, writes to a temporary file, verifies the hash, and renames into place so a half-written file is never visible.

3.2States

StateMeaning
QueuedAccepted and persisted. Waiting for constraints such as Wi-Fi or charging.
PreparingCopying, hashing, optionally compressing.
CheckingAsking the server whether the content already exists.
UploadingParts in flight; progress = parts done ÷ total.
PausedSystem stopped us, network lost, or constraints no longer met. Resumes automatically.
CompletingAll parts done; asking the server to assemble and verify.
DoneLinked to its record. Local copy can be evicted later.
FailedPermanent error such as file deleted or hash mismatch; shown to the user.
CancelledUser cancelled. Server multipart upload aborted.

3.3Run loop

One run, step by step.

AppTransfer managerUpload APIObject storageTransfer DBenqueue(video, priority = user)copy + SHA-256 in one passPOST /files/init {sha256, size}upload_id + 52 part URLssave plan: 52 partsPUT parts 1–4 in parallelETag per partmark each part doneWi-Fi → 4G at part 20: in-flight parts retryURLs expire at part 31refresh URLs for 31–52PUT remaining partsPOST /files/complete {parts, ETags}hash verified · linked
Solid arrows are calls; dashed arrows are responses.
  1. The app computes the hash once, during the copy, and reuses it for deduplication and final verification.
  2. Each part commits independently, so any interruption costs only the parts in flight.
  3. Expired URLs are a normal event, refreshed in bulk, never a failure.

3.4Code

Working code for the core of it.

Transfer tables (SQLite)

transfer-tables-sqlite.sqlSQL
1CREATE TABLE file_transfer (2  id TEXT PRIMARY KEY, owner_entity TEXT, owner_id TEXT,3  local_path TEXT NOT NULL, size INTEGER NOT NULL, sha256 TEXT NOT NULL, mime TEXT,4  direction TEXT NOT NULL,           -- up | down5  priority TEXT NOT NULL,            -- user | background6  state TEXT NOT NULL,               -- queued | preparing | uploading | paused | completing | done | failed | cancelled7  upload_id TEXT, part_size INTEGER, part_count INTEGER,8  attempts INTEGER DEFAULT 0, next_retry_at INTEGER, last_error TEXT, remote_key TEXT,9  created_at INTEGER NOT NULL10);11CREATE TABLE transfer_part (12  transfer_id TEXT NOT NULL, part_no INTEGER NOT NULL,13  url TEXT, url_expires_at INTEGER, etag TEXT, done INTEGER DEFAULT 0,14  PRIMARY KEY (transfer_id, part_no)15);16CREATE INDEX transfer_ready ON file_transfer(state, priority, next_retry_at);
16 linesUTF-8SQL

Hash while copying (Kotlin)

hash-while-copying.ktKotlin
1fun copyAndHash(src: InputStream, dst: File): String {2  val md = MessageDigest.getInstance("SHA-256")3  val buf = ByteArray(1 shl 20)                       // 1 MB buffer, constant memory4  dst.outputStream().buffered().use { out ->5    src.use { input ->6      while (true) {7        val n = input.read(buf); if (n < 0) break8        md.update(buf, 0, n); out.write(buf, 0, n)9      }10    }11  }12  return md.digest().joinToString("") { "%02x".format(it) }13}
13 linesUTF-8Kotlin

Part planning

part-planning.ktKotlin
1fun planParts(size: Long, onCellular: Boolean): Plan {2  val min = 5L * 1024 * 1024                            // S3 minimum except last part3  val preferred = if (onCellular) 8L * 1024 * 1024 else 16L * 1024 * 10244  val forLimit = ceil(size / 9_000.0).toLong()          // stay well under 10,000 parts5  val partSize = maxOf(min, preferred, forLimit)6  val count = ((size + partSize - 1) / partSize).toInt()7  return Plan(partSize, count)8}
8 linesUTF-8Kotlin

Streaming one part without loading it (OkHttp)

streaming-one-part.ktKotlin
1class FileSliceBody(private val file: File, private val offset: Long, private val length: Long) : RequestBody() {2  override fun contentType() = "application/octet-stream".toMediaType()3  override fun contentLength() = length4  override fun writeTo(sink: BufferedSink) {5    RandomAccessFile(file, "r").use { raf ->6      raf.seek(offset)7      val buf = ByteArray(64 * 1024); var left = length8      while (left > 0) {9        val n = raf.read(buf, 0, minOf(buf.size.toLong(), left).toInt())10        sink.write(buf, 0, n); left -= n11      }12    }13  }14}15 16suspend fun uploadPart(t: Transfer, p: Part): String {17  val req = Request.Builder().url(p.url).put(FileSliceBody(t.file, (p.no - 1) * t.partSize, p.length)).build()18  client.newCall(req).await().use { res ->19    if (res.code == 403) throw UrlExpired(p.no)                // broker refreshes, then retry20    if (!res.isSuccessful) throw IOException("part " + p.no + " failed: " + res.code)21    return res.header("ETag") ?: error("missing ETag")22  }23}
23 linesUTF-8Kotlin

Android: one WorkManager job per part

android-one-workmanager.ktKotlin
1fun enqueuePart(ctx: Context, transferId: String, part: Int) {2  val req = OneTimeWorkRequestBuilder<PartUploadWorker>()3    .setInputData(workDataOf("transfer" to transferId, "part" to part))4    .setConstraints(Constraints.Builder().setRequiredNetworkType(NetworkType.CONNECTED).setRequiresStorageNotLow(true).build())5    .setBackoffCriteria(BackoffPolicy.EXPONENTIAL, 10, TimeUnit.SECONDS)6    .addTag("transfer:$transferId")                                   // cancel a whole transfer by tag7    .build()8  WorkManager.getInstance(ctx).enqueueUniqueWork("part:$transferId:$part", ExistingWorkPolicy.KEEP, req)9}10 11class PartUploadWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ctx, params) {12  override suspend fun doWork(): Result {13    val id = inputData.getString("transfer") ?: return Result.failure()14    val no = inputData.getInt("part", -1)15    val (t, p) = transfers.load(id, no) ?: return Result.success()     // transfer was cancelled meanwhile16    if (p.etag != null) return Result.success()                        // already uploaded: safe to rerun17    setForeground(foregroundInfo(t))                                   // long upload: visible, not killed18    return try {19      transfers.recordPart(id, no, etag = uploadPart(t, p))            // durable before reporting success20      if (transfers.allPartsDone(id)) TransferTriggers.complete(applicationContext, id)21      Result.success()22    } catch (e: UrlExpired) {23      transfers.refreshUrl(id, no); Result.retry()                     // presigned URL timed out24    } catch (e: IOException) {25      if (runAttemptCount < 8) Result.retry() else Result.failure()26    }27  }28}
28 linesUTF-8Kotlin

Resuming a download

resuming-download.ktKotlin
1val have = tmp.length()2val req = Request.Builder().url(signedUrl).header("Range", "bytes=" + have + "-").build()3client.newCall(req).await().use { res ->4  require(res.code == 206 || (have == 0L && res.code == 200)) { "server ignored Range" }5  FileOutputStream(tmp, true).use { out -> res.body!!.byteStream().copyTo(out) }6}7check(sha256(tmp) == expectedHash) { "corrupt download" }8tmp.renameTo(finalFile)                             // atomic on the same filesystem
8 linesUTF-8Kotlin

3.5Tested code, explained

A working implementation, then what each part does and why.

Cutting a file into parts

This is simple arithmetic, and getting the last part wrong corrupts uploads.

PartUploader.ktKotlin
1data class Part(val no: Int, val offset: Long, val length: Long)2 3fun planParts(size: Long, partSize: Long = 8 * MIB): List<Part> {4  val count = ((size + partSize - 1) / partSize).toInt()5  return List(count) { i ->6    val offset = i * partSize7    Part(i + 1, offset, minOf(partSize, size - offset))8  }9}
9 linesUTF-8Kotlin
(size + partSize - 1) / partSize
Rounds up, so 412 MB makes 52 parts of 8 MB.
minOf(partSize, size - offset)
The last part is usually shorter.
Part(i + 1, ...)
Object stores number parts from 1.

Compiled and tested by npm run test:mobile.

Sending parts that survive being killed

Arjun's upload. The receipts map stands for a table on disk, so a killed app loses almost nothing.

PartUploader.ktKotlin
1/**2 * Sends only what the table says is missing. [receipts] stands for the transfer_part3 * table: a part number maps to its ETag once that part succeeds.4 */5class PartUploader(6  private val parts: List<Part>,7  private val receipts: MutableMap<Int, String>,8  private val freshUrls: suspend (List<Part>) -> Map<Int, String>,9  private val send: suspend (Part, url: String) -> String,10  private val parallel: Int = 4,11) {12  val remaining: List<Part> get() = parts.filter { it.no !in receipts }13 14  suspend fun run() {15    var urls = freshUrls(remaining)16    while (remaining.isNotEmpty()) {17      val outcomes = coroutineScope {18        remaining.take(parallel).map { part ->19          async { part to runCatching { send(part, urls.getValue(part.no)) } }20        }.awaitAll()21      }22      // Save every success first, so a failure next to it costs nothing.23      outcomes.forEach { (part, r) -> r.onSuccess { etag -> receipts[part.no] = etag } }24      val failure = outcomes.firstNotNullOfOrNull { (_, r) -> r.exceptionOrNull() }25      when (failure) {26        null -> Unit27        is UrlExpired -> urls = freshUrls(remaining)    // only the parts still missing28        else -> throw failure29      }30    }31  }32 33  /** The body of POST /files/complete, in part order. */34  fun manifest(): List<Pair<Int, String>> = receipts.toSortedMap().map { (no, etag) -> no to etag }35}
35 linesUTF-8Kotlin
parts.filter { it.no !in receipts }
What is left. This one line is why resuming needs no special case: a fresh start and a resume run the same code.
runCatching { send(...) }
One failed part doesn't cancel the others in flight.
outcomes.forEach { ... receipts[...] = etag }
Saves every success before looking at failures, so finished work always counts.
is UrlExpired -> urls = freshUrls(remaining)
Fetches new URLs only for the parts still missing, then carries on.
else -> throw failure
Anything else ends this run. The next run picks up from the receipts.

Compiled and tested by npm run test:mobile.

3.6Platforms

Android

  • WorkManager unique work per transfer (upload- plus the ID) with network and battery constraints for background priority.
  • For uploads the user started, a user-initiated data transfer job on Android 14 and later, with a progress notification and cancel action.
  • OkHttp with a FileSliceBody so memory stays flat regardless of file size.
  • Media3 Transformer for on-device video compression before upload.

iOS

  • Background URLSession with one upload task per part file; the system keeps transferring while the app is suspended.
  • handleEventsForBackgroundURLSession in the app delegate, and the stored completion handler called after events are delivered.
  • BGContinuedProcessingTask (iOS 26+) or a short background task for the hashing and compression step.
  • AVAssetExportSession for video compression.

React Native

  • A native module with enqueueUpload(uri, priority) that returns a transfer ID; progress is read from SQLite.
  • Pass file URIs only; bytes never enter the JS heap.
  • react-native-background-upload or a custom module wrapping the native pieces above.
Part 4 of 5

Q&A

The questions from part 2, now with model answers. Try each one out loud before opening it.

4.1Interview questions

Why copy the file before uploading?
Picker and gallery URIs can be revoked or the user can edit the photo. A private copy with a recorded hash is the only stable source for a transfer that may take hours.
How do you choose part size?
Big enough to keep request overhead low and stay under the part limit, small enough that a lost part is cheap on cellular. 8 MB on cellular and 16 MB on Wi-Fi is a sensible start; measure resend ratio.
Why not one background task for the whole file on iOS?
If it fails at 90%, you restart from zero. One task per part gives checkpoints that survive relaunch.
How do you prevent duplicate storage when users re-upload?
Content addressing: the server looks up the SHA-256 at init and links the existing object.
Part 5 of 5

Extra depth

Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.

5.1Technology choices

Show detailsHide details
NeedOptionsWhen to pick
Upload protocolS3 multipart with presigned URLs · GCS resumable uploads · Azure block blobs · tusUse your cloud’s native protocol; tus when you need a vendor-neutral, open resumable protocol.
IntegritySHA-256 of the whole file · per-part checksums (S3 supports CRC32C, SHA-256 and others)Whole-file hash for dedupe and evidence; per-part checksums to catch corruption in transit.
Concurrency1 · 2–4 · adaptiveTwo on cellular, four on Wi-Fi; drop to one when the battery is low.
CompressionMedia3 Transformer · AVAssetExportSession · noneCompress video when the product allows; never re-compress photos used as evidence.
Download deliveryCDN signed URLs · direct storage URLsCDN with content-addressed keys for anything downloaded more than once.

5.2How real products do it

Show detailsHide details

Summaries of what these teams and projects have described publicly. Check their own write-ups for detail.

WhoWhat they describe
Google APIsPublish a resumable upload protocol: create an upload session, send the file in chunks, and query the server for how many bytes it received before resuming.
tusAn open protocol for resumable uploads over HTTP, with server and client libraries in many languages; Vimeo is among its well-known adopters.
Dropbox APIExposes upload sessions (start, append, finish) so large files are sent in pieces and committed at the end.
Amazon S3Multipart upload: up to 10,000 parts, each at least 5 MB except the last, assembled by a final complete call; incomplete uploads should be cleaned up with a lifecycle rule.

5.3Metrics to emit

Show detailsHide details
MetricWhy it matters
transfer.completion_rateThe headline number: share of enqueued files that reach done.
bytes_resent ÷ bytes_totalShows whether checkpoints and part size are working.
time_to_done (p50, p95) by networkSets expectations and catches regressions.
part.failure by reason403 expiry, timeouts and 5xx need different fixes.
dedupe.hit_rateHow often the server already had the bytes, so the app skipped the upload.
paused.ageTransfers stuck paused for days usually mean a missing constraint or a killed OEM job.

5.4Edge cases

Show detailsHide details
CaseHow it is handled
Source file deleted or permission revokedAlready copied into app storage at enqueue time, so the transfer continues.
File changed after enqueueHash mismatch on complete; transfer marked failed and re-enqueued from the new file.
Presigned URL expired403 triggers a bulk refresh for remaining parts; not counted as a failure.
Disk nearly fullRefuse new copies with a clear message; evict uploaded files first.
Complete called twiceServer treats complete as idempotent by upload ID.
Abandoned upload on the serverStorage lifecycle rule aborts incomplete multipart uploads after a few days.
AI Engineer · The transfer manager