/Learn
Deep dives · AI handbook · integration patterns

Integrating LLMs into products

LearnDeep diveTier 2 · MechanismsAnalyze20 minAgents and apps

You will be able to: explain integrating LLMs into products and answer its interview questions.

Prove it: An AI assistant inside a mobile app

AI engineeringDeep dives

Study with AI

Part 1 of 5

Basics

What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.

1.1Definition

The one-paragraph summary, then where its responsibility starts and stops.

LLM features behave like ordinary software with one unreliable, slow, expensive dependency. The patterns from payments or search apply: a gateway, contracts, retries, fallbacks, caching, budgets and observability.

Patterns that work
  • One internal gateway in front of all model providers
  • Versioned prompt templates with owners
  • Structured outputs validated against a schema
  • Streaming to the client with cancel
  • Fallback models and graceful degradation
  • Per-feature budgets and per-user limits
Anti-patterns
  • Calling providers directly from mobile apps with embedded keys
  • Free-text parsing of model output with regex
  • Unbounded retries on a slow provider
  • Logging full prompts with personal data
  • Shipping prompt changes without an evaluation run

1.2Adding AI to an existing app

A concrete example first, so the mechanics have something to attach to.

A product team wants to add three features in a quarter: listing description rewrite, review summaries and a support assistant. Instead of each team calling providers directly, the platform team ships an LLM gateway in two weeks.

The gateway authenticates internal callers, looks up the prompt template version for the feature, routes to the configured model, enforces JSON schemas on output, retries once on transient errors, falls back to a second provider, and records tokens and cost per feature.

Review summaries run nightly through the provider’s batch API at a lower price. Description rewrites stream to the mobile app over server-sent events. The support assistant uses tools through the same gateway.

When one provider has an outage, traffic moves to the fallback automatically. When cost spikes, the dashboard shows which feature and prompt version caused it.

Part 2 of 5

Question

Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.

2.1Target questions

Keep them in mind while you read the detail.

  1. Why put a gateway in front of LLM providers?
  2. How do you get reliable structured data from a model?
  3. How do you add an LLM to a mobile app safely?

Model answers are in part 4, after the detail that supports them.

2.2Interview rounds

System design rounds where this component is part of a strong answer.

Part 3 of 5

Detailed answer

How it actually works: the components, the states it moves through, a full run, and the code.

3.1An LLM gateway

The parts and how they connect.

CALLERSLLM GATEWAYPROVIDERSMobile + webSSE streamsBackend servicessync callsBatch jobsnightlyAuth + quotasper feature · userPrompt registryversionedCachesprefix · semanticRoutermodel per featureValidatorJSON schema ·guardrailsMetering + tracestokens · costProvider AprimaryProvider BfallbackSelf-hostedopen modelsoutput
ClientServiceData storeAI componentExternal
Boxes are components; arrows show calls and data flow.
Gateway
One service owns provider keys, routing, retries, fallbacks, quotas and metering. Teams call a stable internal API instead of provider SDKs.
Prompt registry
Templates stored with versions and owners; features reference a version, and rollbacks are configuration.
Structured output
Ask for JSON that matches a schema, using the provider’s schema or tool features where available, then validate. Invalid output triggers one repair attempt, then a fallback.
Resilience
Timeouts sized to the feature, one retry with jitter for transient errors, circuit breakers per provider, and a fallback model or non-AI path.
Metering
Tokens, latency and cost per feature, prompt version and model; daily budgets with alerts.

3.2Patterns at a glance

NameMeaning
StreamingServer-sent events from gateway to client; render in small batches; support cancel.
Structured outputsJSON schema in the request, validation on the response.
FallbackSecond provider or smaller model; then a non-AI experience.
Batch processingOffline jobs through batch APIs at lower cost.
Prompt cachingStable prefixes reused across calls for lower cost and latency.

3.3A resilient LLM call

One run, step by step.

ServiceGatewayProvider AProvider BValidatorfeature=review_summary, inputtemplate v7, model routingrequest (timeout 8 s)503one retry with jittertimeoutfallback requestJSON outputvalidate schemavalidresult + usage
Solid arrows are calls; dashed arrows are responses.
  1. Retries stop at a limit, and fallback runs automatically.
  2. The caller receives the same schema whichever provider answered.

3.4Code

Working code for the core of it.

Gateway request contract

gateway-request-contract.jsonJSON
1POST /v1/llm/generate2{ "feature": "review_summary", "template_version": "v7",3  "input": { "reviews": ["..."], "language": "en" },4  "output_schema": { "type": "object", "required": ["summary", "pros", "cons"],5    "properties": { "summary": { "type": "string", "maxLength": 400 },6      "pros": { "type": "array", "items": { "type": "string" } },7      "cons": { "type": "array", "items": { "type": "string" } } } },8  "stream": false, "user_id": "u-91" }
8 linesUTF-8JSON

Validate and repair once (TypeScript)

validate-repair-once.tsTypeScript
1const out = await callModel(req);2let parsed = safeJson(out.text);3if (!schema.validate(parsed)) {4  const fix = await callModel({ ...req, messages: [...req.messages,5    { role: 'user', content: 'Return valid JSON matching the schema. Errors: ' + schema.errors() }] });6  parsed = safeJson(fix.text);7  if (!schema.validate(parsed)) return fallback(req.feature);   // non-AI path or cached result8}9return parsed;
9 linesUTF-8TypeScript

3.5Tested code, explained

A working implementation, then what each part does and why.

Assembling a prompt

A prompt is a program with a structure. Keep it in code, so you can test and version it.

prompt-builder.tsTypeScript
1export interface Example {2  input: string;3  output: string;4}5 6/** Assembles a few-shot prompt: rules first, examples next, the real input last. */7export function buildPrompt(opts: { task: string; examples?: Example[]; schema?: object; reason?: boolean; input: string }): string {8  const parts = [opts.task];9  if (opts.reason) parts.push("Work through the problem step by step, then give the final answer.");10  if (opts.schema) parts.push(`Reply with JSON matching this schema:\n${JSON.stringify(opts.schema)}`);11  for (const e of opts.examples ?? []) parts.push(`Input: ${e.input}\nOutput: ${e.output}`);12  parts.push(`Input: ${opts.input}\nOutput:`);13  return parts.join("\n\n");14}
14 linesUTF-8TypeScript
parts = [opts.task]
Rules first.
opts.schema
States the output format, which later gets checked by the guard.
Input: ... Output:
The real input goes last, in the same shape as the examples.

Tested by npm run test:modules.

Fitting context into a budget

Retrieval usually returns more than fits. This keeps the best sentences, in the order they were written.

context-budget.tsTypeScript
1export interface Sentence {2  text: string;3  score: number;4}5 6/** Keeps the highest-scoring sentences that fit the budget, then restores reading order. */7export function fitContext(sentences: Sentence[], budget: number, count: (s: string) => number): string[] {8  const ranked = sentences.map((s, i) => ({ ...s, i })).sort((a, b) => b.score - a.score);9  const kept: { text: string; i: number }[] = [];10  let used = 0;11  for (const s of ranked) {12    const cost = count(s.text);13    if (used + cost > budget) continue;14    used += cost;15    kept.push(s);16  }17  return kept.sort((a, b) => a.i - b.i).map((s) => s.text);18}
18 linesUTF-8TypeScript
sort((a, b) => b.score - a.score)
Picks by relevance.
continue
Skips one that doesn't fit and tries smaller ones, instead of stopping.
sort((a, b) => a.i - b.i)
Restores reading order, because shuffled sentences confuse a model.

Tested by npm run test:modules.

Never trust model output

Treat a model's reply as untrusted input. Validate it, and re-ask with the exact errors.

output-guard.tsTypeScript
1export type Field = { type: "string" | "number" | "boolean"; required?: boolean; oneOf?: string[]; min?: number; max?: number };2export type Schema = Record<string, Field>;3 4export function validate(value: unknown, schema: Schema): string[] {5  if (typeof value !== "object" || value === null) return ["expected a JSON object"];6  const obj = value as Record<string, unknown>;7  const errors: string[] = [];8  for (const [key, f] of Object.entries(schema)) {9    const v = obj[key];10    if (v === undefined) {11      if (f.required) errors.push(`${key} is required`);12      continue;13    }14    if (typeof v !== f.type) errors.push(`${key} must be a ${f.type}`);15    else if (f.oneOf && !f.oneOf.includes(v as string)) errors.push(`${key} must be one of ${f.oneOf.join(", ")}`);16    else if (typeof v === "number" && ((f.min !== undefined && v < f.min) || (f.max !== undefined && v > f.max))) errors.push(`${key} is out of range`);17  }18  return errors;19}20 21/** Validate, then re-ask with the exact errors. Never pass unvalidated output downstream. */22export async function guardedCall<T>(prompt: string, llm: (p: string) => Promise<string>, schema: Schema, maxRetries = 2) {23  let ask = prompt;24  let errors: string[] = [];25  for (let attempt = 1; attempt <= maxRetries + 1; attempt++) {26    const raw = await llm(ask);27    try {28      const parsed = JSON.parse(raw) as T;29      errors = validate(parsed, schema);30      if (errors.length === 0) return { ok: true as const, value: parsed, attempts: attempt };31    } catch {32      errors = ["reply was not valid JSON"];33    }34    ask = `${prompt}\n\nYour last reply failed validation:\n- ${errors.join("\n- ")}\nReply with corrected JSON only.`;35  }36  return { ok: false as const, errors, attempts: maxRetries + 1 };37}
37 linesUTF-8TypeScript
validate(parsed, schema)
Checks type, range and allowed values, including for tool arguments.
ask = `${prompt}...`
Sends the exact errors back, which fixes most replies in one retry.
ok: false
After the retry limit, return a failure. Never pass bad output downstream.

Tested by npm run test:modules.

3.6Client-side concerns

Mobile apps

  • Never embed provider keys; call your gateway with the user’s session.
  • Stream with SSE, render in batches, allow cancel and retry.
  • Cache results in SQLite; show something useful offline.

Web

  • Same gateway; stream with fetch and ReadableStream.
  • Disable submit while streaming to avoid duplicate calls.

Privacy

  • Redact personal data before logging; set provider data-retention options.
  • Region-pin traffic where residency rules apply.
Part 4 of 5

Q&A

The questions from part 2, now with model answers. Try each one out loud before opening it.

4.1Interview questions

Why put a gateway in front of LLM providers?
Central keys, routing, retries, fallbacks, quotas, metering and policy. Teams ship features faster and the company can switch models without touching every service.
How do you get reliable structured data from a model?
Schema in the request using provider features where available, validation on the response, one repair attempt, then a fallback.
How do you add an LLM to a mobile app safely?
The app calls your backend, never the provider. Stream responses, support cancel, cache results, and design a useful experience when the AI path fails.
Part 5 of 5

Extra depth

Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.

5.1Building blocks

Show detailsHide details
NeedOptionsWhen to pick
GatewaysIn-house service · open-source LLM proxies · cloud AI gatewaysIn-house when you need custom policy; a proxy to start quickly.
SchemasJSON Schema · Zod · PydanticDefine once, use for prompts, validation and types.
ObservabilityOpenTelemetry · LLM tracing tools (e.g. Langfuse, LangSmith)Trace prompts, outputs, tokens and cost per feature.
Cost controlsBatch APIs · prompt caching · routing · output capsCombine; each alone is partial.

5.2Operational metrics

Show detailsHide details
MetricWhy it matters
Error rate and fallback rate per providerProvider health and resilience.
Schema validation failure ratePrompt or model regressions.
Cost per feature per dayBudget control.
p95 latency per featureUser experience.

5.3Production failures

Show detailsHide details
CaseHow it is handled
Provider outageCircuit breaker, fallback provider, then a non-AI path.
Model update changes behaviourPin model versions; re-run evaluations before switching.
Runaway cost from a loopPer-feature budgets and per-request token caps.
Personal data in logsRedaction at the gateway; restricted access.
AI Engineer · Integrating LLMs into products