/Tracks

Two-week programme · Day 11 of 14

When a duplicate costs real money

Study with AI

Part 1Basics

A customer taps Pay, and the network drops before an answer arrives. The money may or may not have moved. The phone can't know. If it retries, it might charge twice. If it doesn't, the order might be lost.

So there are two rules. Every attempt carries a key, so the server can spot a repeat. And the server, not the phone, decides what happened. The phone only displays the answer.

Words you need today

Idempotency key
The op_id idea, applied to money: one key per checkout, sent with every retry.
PSP
Payment service provider, the company that actually moves the money, such as Stripe.
State machine
A fixed list of stages an order can be in, and which moves between them are allowed.
Webhook
A message the provider sends your server when something happens, such as a payment succeeding.
Reconciliation
A regular comparison of your records against the provider's, to catch anything missed.
Keystore and Keychain
Hardware-backed secure storage on Android and iOS, used for tokens and keys.

The story

A customer taps Pay on a train. The request leaves the phone, the train enters a tunnel, and no response arrives. Was she charged? The phone honestly can't know. Retry and you may charge twice. Don't retry and you may lose the order.

You met the answer on Day 2: a key made before the first send. Each checkout gets one idempotency key, the server stores it under a unique constraint and returns the stored result for repeats, and it forwards the same key to the payment provider.

What changes is who decides. For Priya, the phone could accept an edit and sort it out later. Here the phone only shows state. The order lives on the server, and the provider's webhook settles the outcome, even if it arrives minutes later and the app is gone.

Part 2Today's question

What changes when you can't let the app decide, and can't get it wrong?

By the end of this lesson, you should be able to:

  • Design end-to-end idempotency
  • Handle unknown payment outcomes
  • Say where offline-first stops

Part 3In detail

Step by step

  1. Create the key. When checkout starts, the phone creates a key and stores it, so a crash can't change it.
  2. Create the order. The server stores the key under a unique constraint and creates an order in the payment pending state.
  3. Pay. The provider's SDK collects the card details, so your server never sees them. The same key goes to the provider.
  4. Wait for the truth. The screen says confirming. The provider's webhook updates the order on the server.
  5. Show the result. The phone fetches the order's state and displays it, whenever it next can.

Offline-first has a limit

The local-first instinct stops at money. You can cache the cart and queue the order intent, but you can't confirm a payment offline. Saying where the pattern stops is a strong principal signal.

Unknown is a real state

Besides success and failure, design for unknown. The screen says we're confirming your payment, the server polls the provider, and a daily reconciliation catches anything missed. Never show failed when you don't know.

Assume the phone is hostile

Prices and permissions get checked on the server. Tokens live in hardware-backed storage, and many 401s trigger one single-flight refresh. That is the same single-flight idea as the sync engine's lock.

A retry with no second charge

retry-no-second.httpHTTP
1POST /v1/orders2Idempotency-Key: 0192e0c6-a04d3{ "cart_id": "k-19", "amount": 4999, "currency": "INR" }4 5→ 202 { "order_id": "o-881", "state": "payment_pending" }6 7# Network dropped, so the app retries with the SAME key.8POST /v1/orders9Idempotency-Key: 0192e0c6-a04d10 11→ 202 { "order_id": "o-881", "state": "payment_pending" }   # stored reply, nothing new
11 linesUTF-8HTTP

This is op_id once more. The stakes changed, the mechanism did not.

The code, explained

This is the real code, and our tests run it. Read each block, then the notes under it.

The moves an order can make

List the allowed moves, and impossible states stop being possible.

Checkout.ktKotlin
1enum class OrderState { PAYMENT_PENDING, PAID, FAILED, CANCELLED, REFUND_DUE }2 3/** The only moves an order can make. Anything else is ignored rather than guessed at. */4private val ALLOWED = mapOf(5  OrderState.PAYMENT_PENDING to setOf(OrderState.PAID, OrderState.FAILED, OrderState.CANCELLED),6  OrderState.CANCELLED to setOf(OrderState.REFUND_DUE),   // paid after cancelling: give it back7)8 9data class Order(val id: String, val amount: Long, val state: OrderState)
9 linesUTF-8Kotlin
ALLOWED
Anything not listed gets ignored. A late failed webhook can't undo a paid order.
OrderState.CANCELLED to setOf(OrderState.REFUND_DUE)
Money arrived after the customer cancelled. It's a named case, not an exception.

Compiled and tested by npm run test:mobile.

Taking payment exactly once

The phone can retry as much as it likes. These methods make that harmless.

Checkout.ktKotlin
1class OrderService(private val priceOf: (cartId: String) -> Long) {2  private val byKey = HashMap<String, Order>()3  private val byId = HashMap<String, String>()4  private val seenEvents = HashSet<String>()5  private var nextOrder = 8806 7  /** A repeat with the same key gets the stored order back. The phone's price is never read. */8  @Synchronized9  fun create(idempotencyKey: String, cartId: String): Order =10    byKey.getOrPut(idempotencyKey) {11      nextOrder += 112      val order = Order("o-$nextOrder", priceOf(cartId), OrderState.PAYMENT_PENDING)13      byId[order.id] = idempotencyKey14      order15    }16 17  /** Providers resend webhooks, so each event ID counts once. */18  @Synchronized19  fun onWebhook(eventId: String, orderId: String, succeeded: Boolean): Order {20    val key = byId.getValue(orderId)21    val order = byKey.getValue(key)22    if (!seenEvents.add(eventId)) return order23    val wanted = when {24      !succeeded -> OrderState.FAILED25      order.state == OrderState.CANCELLED -> OrderState.REFUND_DUE26      else -> OrderState.PAID27    }28    if (wanted !in ALLOWED[order.state].orEmpty()) return order29    return order.copy(state = wanted).also { byKey[key] = it }30  }31 32  @Synchronized33  fun cancel(orderId: String): Order {34    val key = byId.getValue(orderId)35    val order = byKey.getValue(key)36    if (OrderState.CANCELLED !in ALLOWED[order.state].orEmpty()) return order37    return order.copy(state = OrderState.CANCELLED).also { byKey[key] = it }38  }39 40  /** Daily: find what the provider charged that we still think is waiting. */41  @Synchronized42  fun reconcile(providerPaidIds: Set<String>): List<String> =43    byKey.values.filter { it.state == OrderState.PAYMENT_PENDING && it.id in providerPaidIds }.map { it.id }44}
44 linesUTF-8Kotlin
byKey.getOrPut(idempotencyKey)
A repeat returns the stored order. In production, a unique constraint does this.
priceOf(cartId)
The server works out the price. Whatever the phone claims is never read.
seenEvents.add(eventId)
Providers resend webhooks, so each event counts once.
reconcile(...)
Runs daily, and finds charges whose webhook never arrived.

Compiled and tested by npm run test:mobile.

What the customer sees

The phone never decides what happened. It just shows the last answer it got.

Checkout.ktKotlin
1sealed interface CheckoutUi {2  data object Confirming : CheckoutUi3  data class Done(val orderId: String) : CheckoutUi4  data object TryAnotherMethod : CheckoutUi5  data object Refunding : CheckoutUi6}7 8/** No network answer means Confirming. The phone never claims failure when it does not know. */9fun checkoutUi(lastKnown: Order?): CheckoutUi = when (lastKnown?.state) {10  null, OrderState.PAYMENT_PENDING -> CheckoutUi.Confirming11  OrderState.PAID -> CheckoutUi.Done(lastKnown.id)12  OrderState.FAILED, OrderState.CANCELLED -> CheckoutUi.TryAnotherMethod13  OrderState.REFUND_DUE -> CheckoutUi.Refunding14}
14 linesUTF-8Kotlin
null, OrderState.PAYMENT_PENDING -> CheckoutUi.Confirming
No answer means confirming, never failed. Telling someone a payment failed when it succeeded makes them pay twice.

Compiled and tested by npm run test:mobile.

Refreshing a token only once

When a token expires, ten requests can fail together. If each one refreshes, rotation treats the repeats as theft and logs the user out.

TokenRefresher.ktKotlin
1data class Tokens(val access: String, val refresh: String)2 3/** Ten requests can get a 401 at once. Only the first refreshes, and the rest wait for its result. */4class TokenRefresher(5  private var tokens: Tokens,6  private val callRefresh: suspend (refreshToken: String) -> Tokens,7) {8  private val lock = Mutex()9  private var inFlight: CompletableDeferred<Tokens>? = null10 11  val current: Tokens get() = tokens12 13  /** [stale] is the access token that just failed. */14  suspend fun refreshed(stale: String): Tokens {15    val mine = CompletableDeferred<Tokens>()16    val joined = lock.withLock {17      if (tokens.access != stale) return tokens       // someone already replaced it18      inFlight?.also { return@withLock it }19      inFlight = mine20      null21    }22    if (joined != null) return joined.await()23    try {24      val fresh = callRefresh(tokens.refresh)         // rotates the refresh token too25      lock.withLock {26        tokens = fresh27        inFlight = null28      }29      mine.complete(fresh)30      return fresh31    } catch (e: Exception) {32      lock.withLock { inFlight = null }33      mine.completeExceptionally(e)34      throw e35    }36  }37}
37 linesUTF-8Kotlin
if (tokens.access != stale) return tokens
Someone already replaced this token, so use the new one.
inFlight?.also { return@withLock it }
A refresh is running, so join it rather than starting another.
callRefresh(tokens.refresh)
Runs outside the lock, so waiting requests never block on the network while holding it.
completeExceptionally(e)
If the refresh fails, everyone waiting finds out, and the old tokens stay.

Compiled and tested by npm run test:mobile.

What can go wrong

A retry after a lost reply
Same key, so the server returns the stored result. No second charge.
The app dies during payment
The webhook still arrives, and the order completes on the server.
The webhook goes missing
The server polls the provider, and daily reconciliation catches the rest.
The user double-taps
Disabling the button is cosmetic. The key is the real protection.
Someone edits the price on their phone
The server computes the price itself and ignores the phone's number.

Decisions to defend

  • Who decides the outcome? Server state machine plus provider webhook, not the app, based on the SDK's callback. The app can die mid-payment. The webhook always arrives, eventually.
  • Same as Week 1: Key generated before the first send, not disable the button. A disabled button stops a double tap but not a network retry.

Part 4Questions and answers

Answer out loud first, then open each one.

Provider webhook says paid, but the order was cancelled. What now?
Keep it on the state machine as a named case, refund automatically, and record it in the ledger. Reconciliation must catch this, so don't let it be an exception path nobody tests.
Where should the idempotency key be generated?
On the phone, when checkout starts, and stored so it survives a crash. A key created at retry time would look like a new order.

Part 5Going further

The three questions, for today

Where does it live?
The order lives on the server. The key lives on the phone.
Who runs the next step?
The provider's webhook, backed by daily reconciliation.
What if it runs twice?
The same key works at the app, the order service and the provider.

Practise

Optional reference pages

You don’t need these for the lesson. They hold more detail if you want it later.

Payments and chat were about writes. Tomorrow nearly everything is a read, on a mid-range phone, at 60 frames a second.

← Day 10: Now the edit is a message

AI Engineer · Day 11 of 14