Two-week programme · Day 11 of 14
When a duplicate costs real money
Part 1Basics
A customer taps Pay, and the network drops before an answer arrives. The money may or may not have moved. The phone can't know. If it retries, it might charge twice. If it doesn't, the order might be lost.
So there are two rules. Every attempt carries a key, so the server can spot a repeat. And the server, not the phone, decides what happened. The phone only displays the answer.
Words you need today
- Idempotency key
- The op_id idea, applied to money: one key per checkout, sent with every retry.
- PSP
- Payment service provider, the company that actually moves the money, such as Stripe.
- State machine
- A fixed list of stages an order can be in, and which moves between them are allowed.
- Webhook
- A message the provider sends your server when something happens, such as a payment succeeding.
- Reconciliation
- A regular comparison of your records against the provider's, to catch anything missed.
- Keystore and Keychain
- Hardware-backed secure storage on Android and iOS, used for tokens and keys.
The story
A customer taps Pay on a train. The request leaves the phone, the train enters a tunnel, and no response arrives. Was she charged? The phone honestly can't know. Retry and you may charge twice. Don't retry and you may lose the order.
You met the answer on Day 2: a key made before the first send. Each checkout gets one idempotency key, the server stores it under a unique constraint and returns the stored result for repeats, and it forwards the same key to the payment provider.
What changes is who decides. For Priya, the phone could accept an edit and sort it out later. Here the phone only shows state. The order lives on the server, and the provider's webhook settles the outcome, even if it arrives minutes later and the app is gone.
Part 2Today's question
What changes when you can't let the app decide, and can't get it wrong?
By the end of this lesson, you should be able to:
- Design end-to-end idempotency
- Handle unknown payment outcomes
- Say where offline-first stops
Part 3In detail
Step by step
- Create the key. When checkout starts, the phone creates a key and stores it, so a crash can't change it.
- Create the order. The server stores the key under a unique constraint and creates an order in the payment pending state.
- Pay. The provider's SDK collects the card details, so your server never sees them. The same key goes to the provider.
- Wait for the truth. The screen says confirming. The provider's webhook updates the order on the server.
- Show the result. The phone fetches the order's state and displays it, whenever it next can.
Offline-first has a limit
The local-first instinct stops at money. You can cache the cart and queue the order intent, but you can't confirm a payment offline. Saying where the pattern stops is a strong principal signal.
Unknown is a real state
Besides success and failure, design for unknown. The screen says we're confirming your payment, the server polls the provider, and a daily reconciliation catches anything missed. Never show failed when you don't know.
Assume the phone is hostile
Prices and permissions get checked on the server. Tokens live in hardware-backed storage, and many 401s trigger one single-flight refresh. That is the same single-flight idea as the sync engine's lock.
A retry with no second charge
1POST /v1/orders2Idempotency-Key: 0192e0c6-a04d3{ "cart_id": "k-19", "amount": 4999, "currency": "INR" }4 5→ 202 { "order_id": "o-881", "state": "payment_pending" }6 7# Network dropped, so the app retries with the SAME key.8POST /v1/orders9Idempotency-Key: 0192e0c6-a04d10 11→ 202 { "order_id": "o-881", "state": "payment_pending" } # stored reply, nothing newThis is op_id once more. The stakes changed, the mechanism did not.
The code, explained
This is the real code, and our tests run it. Read each block, then the notes under it.
The moves an order can make
List the allowed moves, and impossible states stop being possible.
1enum class OrderState { PAYMENT_PENDING, PAID, FAILED, CANCELLED, REFUND_DUE }2 3/** The only moves an order can make. Anything else is ignored rather than guessed at. */4private val ALLOWED = mapOf(5 OrderState.PAYMENT_PENDING to setOf(OrderState.PAID, OrderState.FAILED, OrderState.CANCELLED),6 OrderState.CANCELLED to setOf(OrderState.REFUND_DUE), // paid after cancelling: give it back7)8 9data class Order(val id: String, val amount: Long, val state: OrderState)ALLOWED- Anything not listed gets ignored. A late failed webhook can't undo a paid order.
OrderState.CANCELLED to setOf(OrderState.REFUND_DUE)- Money arrived after the customer cancelled. It's a named case, not an exception.
Compiled and tested by npm run test:mobile.
Taking payment exactly once
The phone can retry as much as it likes. These methods make that harmless.
1class OrderService(private val priceOf: (cartId: String) -> Long) {2 private val byKey = HashMap<String, Order>()3 private val byId = HashMap<String, String>()4 private val seenEvents = HashSet<String>()5 private var nextOrder = 8806 7 /** A repeat with the same key gets the stored order back. The phone's price is never read. */8 @Synchronized9 fun create(idempotencyKey: String, cartId: String): Order =10 byKey.getOrPut(idempotencyKey) {11 nextOrder += 112 val order = Order("o-$nextOrder", priceOf(cartId), OrderState.PAYMENT_PENDING)13 byId[order.id] = idempotencyKey14 order15 }16 17 /** Providers resend webhooks, so each event ID counts once. */18 @Synchronized19 fun onWebhook(eventId: String, orderId: String, succeeded: Boolean): Order {20 val key = byId.getValue(orderId)21 val order = byKey.getValue(key)22 if (!seenEvents.add(eventId)) return order23 val wanted = when {24 !succeeded -> OrderState.FAILED25 order.state == OrderState.CANCELLED -> OrderState.REFUND_DUE26 else -> OrderState.PAID27 }28 if (wanted !in ALLOWED[order.state].orEmpty()) return order29 return order.copy(state = wanted).also { byKey[key] = it }30 }31 32 @Synchronized33 fun cancel(orderId: String): Order {34 val key = byId.getValue(orderId)35 val order = byKey.getValue(key)36 if (OrderState.CANCELLED !in ALLOWED[order.state].orEmpty()) return order37 return order.copy(state = OrderState.CANCELLED).also { byKey[key] = it }38 }39 40 /** Daily: find what the provider charged that we still think is waiting. */41 @Synchronized42 fun reconcile(providerPaidIds: Set<String>): List<String> =43 byKey.values.filter { it.state == OrderState.PAYMENT_PENDING && it.id in providerPaidIds }.map { it.id }44}byKey.getOrPut(idempotencyKey)- A repeat returns the stored order. In production, a unique constraint does this.
priceOf(cartId)- The server works out the price. Whatever the phone claims is never read.
seenEvents.add(eventId)- Providers resend webhooks, so each event counts once.
reconcile(...)- Runs daily, and finds charges whose webhook never arrived.
Compiled and tested by npm run test:mobile.
What the customer sees
The phone never decides what happened. It just shows the last answer it got.
1sealed interface CheckoutUi {2 data object Confirming : CheckoutUi3 data class Done(val orderId: String) : CheckoutUi4 data object TryAnotherMethod : CheckoutUi5 data object Refunding : CheckoutUi6}7 8/** No network answer means Confirming. The phone never claims failure when it does not know. */9fun checkoutUi(lastKnown: Order?): CheckoutUi = when (lastKnown?.state) {10 null, OrderState.PAYMENT_PENDING -> CheckoutUi.Confirming11 OrderState.PAID -> CheckoutUi.Done(lastKnown.id)12 OrderState.FAILED, OrderState.CANCELLED -> CheckoutUi.TryAnotherMethod13 OrderState.REFUND_DUE -> CheckoutUi.Refunding14}null, OrderState.PAYMENT_PENDING -> CheckoutUi.Confirming- No answer means confirming, never failed. Telling someone a payment failed when it succeeded makes them pay twice.
Compiled and tested by npm run test:mobile.
Refreshing a token only once
When a token expires, ten requests can fail together. If each one refreshes, rotation treats the repeats as theft and logs the user out.
1data class Tokens(val access: String, val refresh: String)2 3/** Ten requests can get a 401 at once. Only the first refreshes, and the rest wait for its result. */4class TokenRefresher(5 private var tokens: Tokens,6 private val callRefresh: suspend (refreshToken: String) -> Tokens,7) {8 private val lock = Mutex()9 private var inFlight: CompletableDeferred<Tokens>? = null10 11 val current: Tokens get() = tokens12 13 /** [stale] is the access token that just failed. */14 suspend fun refreshed(stale: String): Tokens {15 val mine = CompletableDeferred<Tokens>()16 val joined = lock.withLock {17 if (tokens.access != stale) return tokens // someone already replaced it18 inFlight?.also { return@withLock it }19 inFlight = mine20 null21 }22 if (joined != null) return joined.await()23 try {24 val fresh = callRefresh(tokens.refresh) // rotates the refresh token too25 lock.withLock {26 tokens = fresh27 inFlight = null28 }29 mine.complete(fresh)30 return fresh31 } catch (e: Exception) {32 lock.withLock { inFlight = null }33 mine.completeExceptionally(e)34 throw e35 }36 }37}if (tokens.access != stale) return tokens- Someone already replaced this token, so use the new one.
inFlight?.also { return@withLock it }- A refresh is running, so join it rather than starting another.
callRefresh(tokens.refresh)- Runs outside the lock, so waiting requests never block on the network while holding it.
completeExceptionally(e)- If the refresh fails, everyone waiting finds out, and the old tokens stay.
Compiled and tested by npm run test:mobile.
What can go wrong
- A retry after a lost reply
- Same key, so the server returns the stored result. No second charge.
- The app dies during payment
- The webhook still arrives, and the order completes on the server.
- The webhook goes missing
- The server polls the provider, and daily reconciliation catches the rest.
- The user double-taps
- Disabling the button is cosmetic. The key is the real protection.
- Someone edits the price on their phone
- The server computes the price itself and ignores the phone's number.
Decisions to defend
- Who decides the outcome? Server state machine plus provider webhook, not the app, based on the SDK's callback. The app can die mid-payment. The webhook always arrives, eventually.
- Same as Week 1: Key generated before the first send, not disable the button. A disabled button stops a double tap but not a network retry.
Part 4Questions and answers
Answer out loud first, then open each one.
Provider webhook says paid, but the order was cancelled. What now?
Where should the idempotency key be generated?
Part 5Going further
The three questions, for today
- Where does it live?
- The order lives on the server. The key lives on the phone.
- Who runs the next step?
- The provider's webhook, backed by daily reconciliation.
- What if it runs twice?
- The same key works at the app, the order service and the provider.
Practise
- Mock45 minCheckout and payments that never double-charge
Say the unknown case aloud first. It decides most of the design.
- Stretch30 minHealthcare appointment booking without double-booking
Optional. Popular slots add a new twist: one conditional update, instead of a lock.
Optional reference pages
You don’t need these for the lesson. They hold more detail if you want it later.
Payments and chat were about writes. Tomorrow nearly everything is a read, on a mid-range phone, at 60 frames a second.