Architecture Autopsies
Case #01Silent Data Corruption6 min read

Autopsy #01: Anatomy of a Double Payment in Production

Production Symptom

Rapid double-click or 3G network blip charges the user twice despite a disabled={isSubmitting} React button.

The Incident & The False Culprit

On Friday at 14:10 UTC, a two-line PR replaced a native <form> submit with a custom onClick handler on the checkout button to support Apple Pay telemetry. QA verified the happy path on desktop Chrome over office fiber and approved the release.

Within 40 minutes of the deploy, finance alerts flagged duplicate settlements on mobile Safari and throttled Android devices.

  1. T+00:00Deploy `checkout-web@4.18.2` reaches 100% of edge PoPs.
  2. T+08:00Support receives the first ticket reporting two \$99.00 charges for a single `cart_id`.
  3. T+19:00Payment gateway logs show 0.42% of `POST /v1/charges` requests arriving in pairs spaced 80 ms to 3,400 ms apart, both returning `200 OK` with distinct `charge_id` values.
  4. T+42:00Refund webhooks lag 48 hours behind support volume; checkout traffic is rolled back to `4.18.1`.
REJECTED PATCH

The Red Herring — Why the First Reflex Failed

The on-call engineer's first Slack reflex was to add disabled={isSubmitting} on the <button> element and guard the handler with if (isSubmitting) return;. The patch passed every local Playwright test on localhost with zero duplicate requests.

In production, the patch failed to reduce the 0.42% duplicate rate for two reasons:

  • Async render lag: setIsSubmitting(true) schedules a React render for the next frame; button.disabled stays false when a second click enters the call stack 12 ms later.
  • Premature timeout reset: A 3-second client timeout in catch reset isSubmitting to false on slow 3G connections while the first POST socket remained open in flight.

Causal Chain Traversed

Step-by-step propagation across atomic concepts. Hover or click any step to inspect the underlying mechanism.

  1. 01DOM & Events
    DOM Event Propagation (Capture, Target, Bubble)

    Two native click events fire synchronously in the same frame before React commits disabled={true} to the DOM.

  2. 02State & Architecture
    Debounce, Throttle & rAF Coalescing

    Uncoalesced click handlers invoke submitPayment() twice in consecutive tasks instead of locking on the leading edge.

  3. 03State & Architecture
    Finite State Machines for UI

    Boolean flags (isLoading, hasError) permit re-entry after a timeout while the first request remains in flight.

  4. 04JS Runtime
    AbortController and Cooperative Cancellation

    Missing AbortSignal leaves orphaned POST streams active on the network socket after UI reset or retry.

  5. 05State & Architecture
    Outbox Queues, Idempotency Keys & Causal Ordering

    Absent Idempotency-Key headers cause the payment gateway to treat retried TCP packets as two distinct charges.

The duplicate charge requires five consecutive layer failures. A synchronous DOM event outruns asynchronous React state, uncoalesced callbacks fire two network requests, boolean flags allow re-entry after a timeout, uncancelled sockets finish in the background, and a stateless server commits both payloads.

Failure Reproduction Lab

Step through the 5-stage timeline below to inspect how two clicks inside one 16 ms frame become two settled charges, then advance to Step 5 to verify how a synchronous finite state machine and Idempotency-Key collapse the race into a single charge.

Interactive Simulator

Double-Submit Race: Unbounded Clicks vs FSM + Idempotency Key

Trace how two pointer clicks inside one 16 ms frame bypass React's disabled prop and settle as two distinct Stripe charges.

DOM Click x2
isSubmitting: false
POST /v1/charges (0)
Ledger (0)
Step 1 / 5: 1. Synchronous DOM Capture & Bubble (t = 0 ms)

Two rapid pointerup/click events dispatch synchronously on the <button> before React schedules a commit. Both handlers read isSubmitting === false from the current closure.

button.addEventListener('click', onPay);
// Click #1 & #2 both enter handler
Internal State / Memory
frameMs
0.4
reactDisabledProp
false
inFlightRequests
0
ledgerCharges
0

Layer-by-Layer Dissection

01 → 02

Synchronous DOM Dispatch Outruns Async React Commit

In DOM Event Propagation (Capture, Target, Bubble), Blink dispatches click events synchronously from the native input queue into the JavaScript call stack. When onClick invokes setIsSubmitting(true), React enqueues a state update for the next render pass while button.disabled remains false during the current frame.

A second tap arriving 12 ms later hits the same active DOM node. Because Debounce, Throttle & rAF Coalescing is absent on the leading edge, both closures read isSubmitting === false and enter submitPayment().

02 → 03

Uncoalesced Handlers Feed Re-Entrant Boolean State

Even when a leading-edge lock blocks sub-second double-clicks, cellular handoffs introduce a 3.2-second TLS or gateway stall. Under Finite State Machines for UI, a SUBMITTING state cannot transition back to IDLE on a client-side timer while the HTTP socket remains open.

Boolean flags (isSubmitting, hasTimedOut) violate that invariant. A catch block executes setIsSubmitting(false) at t = 3,000 ms, re-enabling the button while Request #1 is still traversing the carrier network.

03 → 04

Re-Enabled UI Leaves Orphaned Sockets on the Wire

When the user taps "Pay" a second time, Request #2 enters the network stack alongside Request #1. Without AbortController and Cooperative Cancellation, the client never signals intent to drop Request #1, and an aborted fetch() only closes the local socket after the server may have parsed the TCP payload.

  • Client view: Marks Request #1 failed at t = 3,000 ms and dispatches Request #2.
  • Server view: Receives Request #1 at t = 3,150 ms and Request #2 at t = 3,420 ms.
04 → 05

Stateless Retries Commit Twice Without Idempotency Keys

Both HTTP requests reach the payment endpoint as independent mutations. As documented in Outbox Queues, Idempotency Keys & Causal Ordering, HTTP POST is non-idempotent by specification; only an explicit Idempotency-Key header scoped to the checkout intent lets the gateway deduplicate Request #2 and replay Request #1's receipt.

Architectural Fix & Invariants

TS
type PayState = 'idle' | 'submitting' | 'settled' | 'failed';
export function createPaymentSubmitter(intentUuid: string) {
  let state: PayState = 'idle';
  const controller = new AbortController();
  return async function submit(payload: unknown): Promise<Response | null> {
    if (state === 'submitting' || state === 'settled') return null;
    state = 'submitting';
    try {
      const res = await fetch('/v1/charges', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json', 'Idempotency-Key': intentUuid },
        body: JSON.stringify(payload),
        signal: controller.signal,
      });
      state = res.ok ? 'settled' : 'failed';
      return res;
    } catch (err) { state = 'failed'; throw err; }
  };
}

Architectural Invariants:

  • Synchronous transition guard: Mutate idle → submitting synchronously before any await or React render commit to block same-frame double-clicks.
  • Stable intent key: Generate intentUuid once per checkout form mount and reuse it across all retries so the gateway deduplicates re-submissions.
  • Terminal settled state: Treat settled as final for a given intentUuid; starting another payment requires a fresh mount and UUID.
  • Cooperative teardown: Call controller.abort() on unmount so stale responses never mutate unmounted UI state.

Golden Rule: Never guard a financial mutation with React's asynchronous render state; lock synchronously in memory and deduplicate idempotently on the server.

Primary Sources

Last verified: 2026-11-16