/Interview
Interview room · AI system design · 40 minutes

On-device AI versus cloud: a hybrid design

PracticeSystem designTier 3 · Edge casesCreate40 minModel serving

You will be able to: design on-device AI versus cloud: a hybrid design in 40 minutes, and defend each trade-off.

AI engineeringDesign rounds

Study with AI

Ready40:00
ClarifyEstimateHigh-level designDeep diveTrade-offs and wrap-up
Part 1 of 5

Basics

Start with the product and the ideas it depends on. If these are clear, the design that follows will feel obvious rather than memorised.

1.1Product in plain words

Who uses it, what a normal day looks like, and why it is harder than it seems.

Deciding where three AI features run: smart reply suggestions in chat, short summaries of long service descriptions, and recognising products in photos. Some phones can run AI models themselves; others need the cloud.

Who uses it

  • Users: want instant, private features that work offline.
  • Product: wants the features on every phone, not just new ones.
  • Finance: wants cloud AI costs to stay low.

A day in the life

  1. On a new phone, tapping Summarise returns a summary in about a second, with no network call.
  2. On an older phone, the same button calls the cloud and takes a little longer.
  3. Offline on an older phone, the button shows the first two sentences instead of failing.
  4. Taking a photo of a product identifies it on the phone; the photo never leaves the device.
  5. A weekly dashboard shows which phones run which path, and rules change remotely if a model is slow on some devices.

Why it is hard

Phones differ hugely

The same model can take 300 ms on one phone and 5 seconds on another.

Models are big

Even small language models are hundreds of megabytes.

Privacy

Photos and messages are personal; sending them to a server needs a reason and consent.

Same experience everywhere

Users shouldn’t notice which path served them.

Think of it like this

Like a shop that answers simple questions at the counter and phones head office only for the hard ones, and still gives you a leaflet if the phone line is down.

1.2Key concepts

The building blocks this design uses. Each one appears again in the answer.

01On-device model

A small model that runs on the phone's NPU or GPU. It is private, free per request and works offline, but it is less capable and varies by device.

02Cloud model

A large model on servers: the most capable and consistent, but every call costs money, needs a network and sends data off the device.

03Quantisation

Storing a model's weights with fewer bits, for example 4-bit, so it fits in phone memory and runs faster, for a small loss in quality.

04Hybrid routing

A router decides per request: on device when the phone can, cloud as the fallback, and a simple non-AI answer when both fail.

05Model delivery

Models are hundreds of megabytes, so they download after install, on Wi-Fi, and update separately from the app.

06Device fragmentation

Android phones differ hugely in chips and memory. Capability checks plus remote rules decide which devices get which path.

Learn them properly in the deep dives:

Part 2 of 5

Question

Now read the question closely. Agree on what you are building, ask what is unclear, and work out the numbers before you draw anything.

2.1Brief and scope

The question exactly as the interviewer asks it, and what they want to learn from your answer.

Interviewer

We want three AI features in our app: smart replies in chat, summarising long service descriptions, and identifying products in photos. Leadership wants low cost and strong privacy. Decide what runs on the device and what runs in the cloud, and design how they work together.

What it tests: On-device runtimes, model size and quantisation, device fragmentation, and routing that degrades gracefully.

The answer in one sentence. An on-device AI router that checks capability, consent and remote rules, then runs a system model, a bundled model or the cloud, with a non-AI fallback.

How to spend the 40 minutes

  1. Clarify~4 minAsk questions, agree on scope and write down assumptions.
  2. Estimate~4 minRough numbers for users, data and traffic, and what they imply.
  3. High-level design~10 minComponents, data flow and the key API on the board.
  4. Deep dive~14 minThe two or three hardest parts, with real mechanisms.
  5. Trade-offs and wrap-up~8 minDecisions, failure modes, and what you would do next.

2.2Clarify the problem

The first five minutes. Each answer removes a guess from your design.

You askInterviewer answers
Which devices must be supported?Android from 4 GB RAM and iOS from iPhone 12.
Must features work offline?Smart replies and photo recognition should; summaries may need a network.
Can photos leave the device?Only with consent; default is on-device.
App size limits?The base download should not grow by more than 20 MB.

2.3Estimate the scale

Say the arithmetic out loud. What matters is the order of magnitude and what it forces you to do.

MetricEstimateHow
Small text model on device≈ 1–3B params, 4-bitabout 0.5–1.5 GB on disk
Image classifier≈ 5–20 MBfits in the app or downloads on demand
Device share able to run a local LLMvaries widelymeasure with telemetry, not assumptions
Cloud cost avoidedproportional to local shareevery on-device request is free at the margin
Part 3 of 5

Detailed answer

The design itself, from the big picture down to the code: components first, then one request end to end, then the parts interviewers push hardest on.

3.1Architecture

A capability check on the device decides, per feature and per device, whether to run locally, call the cloud, or fall back to a simpler non-AI behaviour.

ON DEVICECLOUDFeature codereplies · summary ·scanAI routercapability + policyOS modelsApple FoundationModels · GeminiBundled modelsCore ML · LiteRTModel downloaderon-demand packsTelemetrylatency · successAI gatewayauth · quotasHosted LLMsummariesVision servicefallbackrecognitionModel CDNversioned packsRemote configrouting rulesQuality dashboardby device classif availablecloudfetchrules
ClientServiceData storeAI componentExternal
Reference design. Boxes show components; arrows show who calls whom.
AI router
A small library every feature calls. It checks device capability (OS version, RAM, chip, whether a system model is present), user consent and remote rules, then picks local, cloud or fallback.
System models
iOS 26 exposes an on-device language model through Apple’s Foundation Models framework, and supported Android devices expose Gemini Nano through ML Kit’s GenAI APIs. Using them costs no app size.
Bundled models
Task-specific models such as an image classifier run through Core ML on iOS and LiteRT (formerly TensorFlow Lite) on Android, using the GPU or neural engine where present.
On-demand packs
Larger models download after install, on Wi-Fi, with versioning and integrity checks, so the base app stays small.
Telemetry by device class
Latency, battery impact, crash rate and quality signals per device class decide routing rules, which change remotely without a release.

3.2Key flow, step by step

Summarising a service description on two different phones.

UserFeatureAI routerOn-device modelCloud LLMtap Summarisesummarise(text, max 60 words)New phone with a system modelrun locallysummary in 1.2 sOlder phone, no local modelPOST /summarisesummary in 1.8 sOlder phone, offlinefallback: first 2 sentences
Solid arrows are requests; dashed arrows are responses.
  1. Features never call a model directly; they call the router with a task and constraints.
  2. Each path produces the same output shape, so UI code is identical.
  3. Offline with no local model, the feature degrades to a non-AI behaviour instead of failing.

3.3Front end

Here the front end owns the decision. The router is client code, and the UI must make three different execution paths feel like one feature, including downloads, consent and fallbacks.

Screen map

Chatsmart replies rowService pageSummarise buttonProduct scancamera + resultConsent sheetuse cloud forphotos?Model downloadsize · Wi-Fi onlyAI settingson-device onlytoggleFallback statesnon-AI versionDebug overlaysource · latencyneeds packofflinenot recognised
ClientService
Screens and how users move between them.
Smart replies
Three chips above the keyboard, generated on device after each received message, with no spinner; if they are not ready in 300 ms, they simply don’t appear.
Summarise
A button that expands to the summary; the label never mentions where it ran, but internal builds show the source and latency.
Model download
Shown once when a feature needs a pack: size, Wi-Fi only by default, progress, cancel. The feature works in its fallback form meanwhile.
Consent sheet
Asked only when the on-device recogniser fails and cloud recognition could help; the answer is remembered and changeable in settings.

Client layers

LayerTechnologyResponsibility
UICompose / SwiftUIFeature screens identical across paths
AI routerKotlin / Swift libraryCapability check, consent, remote rules, timeouts
System modelsApple Foundation Models (iOS 26) · Gemini Nano via ML Kit GenAIOn-device text generation where available
Bundled modelsCore ML · LiteRTImage classification and small tasks
Model packsDownloader + checksumOn-demand models on Wi-Fi
TelemetryPer device classLatency, success, battery impact

Every state the user can see

StateWhat the user seesHow it works
ReadyFeature works; no indicatorA path is available
Preparing‘Getting smart replies ready’ onceModel pack downloading
LocalInstant resultOn-device model answered
CloudSame result, slightly slowerRouter chose cloud
FallbackSimpler non-AI versionOffline and no local model
Consent neededSheet with clear choicePhoto would leave the device

Code

One interface over each platform’s on-device model (Kotlin Multiplatform)

one-interface-over.ktKotlin
1// Shared code sees one interface. Android backs it with Gemini Nano (AICore) and iOS with2// Apple's Foundation Models framework, each in its own platform source set.3enum class Availability { READY, DOWNLOADING, UNSUPPORTED }4 5interface OnDeviceModel {6  suspend fun availability(): Availability7  suspend fun generate(instructions: String, input: String): String8}9 10class Summariser(private val local: OnDeviceModel, private val cloudOrFallback: suspend (String) -> AiResult<String>) {11  suspend fun summarise(text: String): AiResult<String> {12    if (local.availability() != Availability.READY) return cloudOrFallback(text)   // older device or model not ready13    return try {14      val out = local.generate("Summarise service descriptions in 60 words or fewer.", text)15      AiResult.Ok(out, Source.SYSTEM_MODEL)16    } catch (e: CancellationException) {17      throw e                                          // never swallow cancellation18    } catch (e: Exception) {19      cloudOrFallback(text)20    }21  }22}
22 linesUTF-8Kotlin

Router with a time budget and fallback (Kotlin)

router-time-budget.ktKotlin
1suspend fun summarise(text: String): AiResult<String> {2  val rule = remoteRules.summarise3  if (capabilities.hasSystemModel && capabilities.ramGb >= rule.minRamGbLocal) {4    withTimeoutOrNull(rule.localTimeoutMs) { local.summarise(text) }?.let { return AiResult.Ok(it, Source.SYSTEM_MODEL) }5  }6  if (network.isOnline) return runCatching { cloud.summarise(text) }7    .map { AiResult.Ok(it, Source.CLOUD) }8    .getOrElse { fallback(text) }9  return fallback(text)                                          // first two sentences, never an error10}11 12private fun fallback(text: String) = AiResult.Fallback(text.split(". ").take(2).joinToString(". ") + ".", "offline")
12 linesUTF-8Kotlin

3.4Data model and APIs

The tables and contracts that make the design concrete.

Router interface (Kotlin)

router-interface-kotlin.ktKotlin
1interface AiRouter {2  suspend fun summarise(text: String, maxWords: Int): AiResult<String>3}4sealed interface AiResult<out T> {5  data class Ok<T>(val value: T, val source: Source, val ms: Long) : AiResult<T>6  data class Fallback<T>(val value: T, val reason: String) : AiResult<T>7}8enum class Source { SYSTEM_MODEL, BUNDLED_MODEL, CLOUD }
8 linesUTF-8Kotlin

Routing rule (remote config)

routing-rule-remote.jsonJSON
1{ "summarise": {2    "prefer": ["SYSTEM_MODEL", "CLOUD"],3    "min_ram_gb_local": 6,4    "cloud_requires_consent": false,5    "disable_local_on": ["SM-A0*"] },6  "photo_identify": { "prefer": ["BUNDLED_MODEL", "CLOUD"], "cloud_requires_consent": true } }
6 linesUTF-8JSON

3.5Tested code, explained

The core of this design as working code, with what each part does and why.

What 4-bit loading actually saves

The weights shrink, and the constants needed to decode them cost a little extra.

nf4-memory.tsTypeScript
1/** Bits per weight spent on quantisation constants: one 32-bit constant per block of 64 weights. */2export function constantBits(doubleQuant: boolean): number {3  // Double quantisation stores 8-bit constants, plus one 32-bit constant per 256 of those.4  return doubleQuant ? 8 / 64 + 32 / (64 * 256) : 32 / 64;5}6 7export function baseModelGiB(params: number, weightBits: number, doubleQuant: boolean): number {8  return (params * (weightBits + constantBits(doubleQuant))) / 8 / 1024 ** 3;9}
9 linesUTF-8TypeScript
32 / 64
One 32-bit constant per 64 weights.
8 / 64 + 32 / (64 * 256)
Double quantisation also compresses those constants.

Tested by npm run test:modules.

How much memory one token costs

This arithmetic sets how many requests a GPU can serve at once. Weights are fixed, and the KV cache grows with every token.

kv-size.tsTypeScript
1/** Bytes of KV cache one token costs: a key and a value, per layer, per KV head. */2export function kvBytesPerToken(m: { layers: number; kvHeads: number; headDim: number; bytesPerValue: number }): number {3  return 2 * m.layers * m.kvHeads * m.headDim * m.bytesPerValue;4}5 6/** Requests that fit in the memory left after loading the weights. */7export function concurrentRequests(freeGiB: number, tokensPerRequest: number, bytesPerToken: number): number {8  return Math.floor((freeGiB * 1024 ** 3) / (tokensPerRequest * bytesPerToken));9}
9 linesUTF-8TypeScript
2 * m.layers * ...
A key and a value, for each layer, each head and each dimension.
freeGiB
What's left after loading the weights. Divide by the cost of a whole request, and you have your concurrency limit.

Tested by npm run test:modules.

3.6Hard parts, in depth

Where interviewers spend the second half of the round.

Making models fit

Quantisation stores weights in 8 or 4 bits instead of 16 or 32, cutting size and memory several-fold with a small quality loss you must measure. Distillation trains a small model to imitate a big one for a single task.

For narrow tasks such as smart replies, a small fine-tuned model often beats a general model at a fraction of the size.

Fragmentation is the real problem

The same model can run in 300 ms on one phone and 5 seconds on another, or crash on low memory. Ship with conservative rules, collect per-device telemetry, and widen local routing as data proves it is safe.

Privacy as an architecture choice

Photos identified on device never leave it. When the cloud is needed, send the minimum: a cropped region rather than a full photo, and only with consent. Say this explicitly; privacy is a product feature, not a footnote.

Part 4 of 5

Q&A

The questions that come after the design, with model answers. Answer each one out loud before opening it.

4.1Follow-up questions

What the interviewer asks once the main design is on the board.

How do you update a bundled model without an app release?
Versioned model packs from a CDN, with the app declaring which model versions it supports.
How do you evaluate on-device quality?
An offline test set run on a device farm across device classes, plus anonymous quality signals such as ‘summary accepted’ rates.
When would you not use on-device AI?
When the task needs fresh data, tools, long context or strong reasoning, or when the device population is mostly low-end.

4.2Front-end follow-ups

Questions aimed at the client: rendering, state and devices.

How do you keep the experience identical across paths?
Same prompt, same output limits and the same result type from the router; the UI never branches on the source.
How do you decide which devices run locally?
Start conservative, collect latency, crash and battery telemetry per device class, and widen local routing through remote rules as data proves it safe.

4.3Incidents, probes and traps

Realistic production incidents and pointed probes, grouped by topic.

The product 1

ProbeWhy not run everything in the cloud?
Cost per request, network latency, privacy of photos and messages, and offline use. On-device is free at the margin and private, when the device can handle it.

Requirements 1

TrapWhat do candidates forget to ask?
The real device distribution and app size limits. Without them you cannot decide what runs locally.

Scale 1

ProbeHow do you estimate how many devices can run a local model?
From telemetry: OS version, RAM, chip and whether a system model is available, by market. Do not assume flagship hardware.

Architecture 2

IncidentAfter enabling on-device summaries, crashes rose on 4 GB phones.
Out-of-memory from loading the model alongside the app. Tighten routing rules by RAM and device class, check available memory before loading, and disable the local path remotely for affected models.
ProbeHow do you update a bundled model without an app release?
Versioned model packs on a CDN, with the app declaring which versions it supports and verifying hashes before use.

The key flow 1

IncidentUsers complain summaries differ wildly between phones.
Local and cloud paths used different prompts or limits. Share templates and output limits, evaluate both on one test set, and prefer cloud where quality matters most.

Front end 2

IncidentUsers on mobile data say the app downloaded a gigabyte.
The model pack downloaded over cellular. Default to Wi-Fi only, show the size and ask first.
ProbeShould the UI tell users where AI ran?
Not normally, but privacy-sensitive features should say ‘processed on your phone’, and sending photos to the cloud needs explicit consent.

Data and APIs 1

TrapWhy route through an interface instead of calling models directly?
It lets you swap paths, test with fakes, record telemetry in one place and change rules remotely.

The hard parts 2

IncidentAfter quantisation, quality dropped badly in Hindi.
Quantisation often hurts low-resource languages more. Evaluate per language and keep higher precision or the cloud path for those languages.
ProbeHow do you control battery impact?
Run only on user action, use GPU or neural engine delegates, keep outputs short, and measure energy with platform profilers.

Trade-offs 1

ProbeWhy bundle the image classifier but download the language model?
The classifier is a few megabytes; the language model is hundreds. Bundling keeps the install small and most users never need the larger model.

Failures 2

IncidentA corrupted model file causes repeated crashes on start-up.
Verify hashes before loading, load lazily rather than at start-up, and add a crash-loop guard that disables the local path.
IncidentThe iOS system model is unavailable because the user turned Apple Intelligence off.
Check availability every time and fall back to cloud or the non-AI version without errors.
Part 5 of 5

Extra depth

What turns a good answer into a strong one: the decisions you rejected, how the system fails, and the signals of seniority.

5.1Trade-offs and decisions

Every choice has a cost. Naming it is what interviewers listen for.

Show detailsHide details
DecisionChosenRejectedWhy
Smart repliesOn deviceCloudLatency matters, works offline, private, and the task is small.
SummariesSystem model, else cloudCloud onlyFree when available; cloud covers older phones.
Photo recognitionBundled classifier, cloud fallbackCloud onlyPrivate by default and offline; cloud handles rare products.
Model deliveryOn-demand packsBundle everythingKeeps the install small; most users never need every model.

5.2Failure modes

What breaks in production, and how the design keeps working when it does.

Show detailsHide details
What breaksHow the design copes
Local model is slow on some phonesTelemetry flags the device class; remote rule routes it to cloud.
Model download corrupt or partialHash check, resume, keep the previous version until the new one verifies.
Cloud and local give different answersSame prompt templates and output limits; evaluate both paths on the same test set.
Battery drain complaintsRun heavy inference only on user action, never in the background.

5.3Performance and accessibility

The client-side details that separate a demo from a product.

Performance

  • Run inference only on user action or when a chat message arrives, never in the background.
  • Warm the model when the screen opens, not at app start.
  • Cap local generation length; long outputs drain battery.

Accessibility and localisation

  • AI-generated text is labelled as such where it matters.
  • Smart reply chips are reachable and readable by screen readers.
  • Users can turn on-device AI features off completely.

5.4Senior versus principal

The same design, answered at two levels. Use the rubric to score yourself.

Senior answer

A senior answer picks a runtime and runs a model on the phone.

Principal answer

A principal answer designs a router with capability checks, remote rules and fallbacks, explains quantisation and fragmentation, makes privacy a design input, and decides per feature with clear criteria.

Score yourself

5.5Whiteboard

Draw the design here, in this order. It stays in this browser.

  1. Feature → AI router decision box
  2. Three paths: system model, bundled model, cloud
  3. Model downloader and CDN
  4. Remote config feeding routing rules
  5. Telemetry by device class
Tip: draw boxes first, then arrows, then label them with Text.
AI Engineer · interview room