Architecture Autopsies
Case #08Silent Data Corruption8 min read

Autopsy #08: Surviving a 4-Hour Tunnel: Local-First Sync & Conflicts

Production Symptom

A user edits a document offline on a train and silently overwrites a teammate's merged changes upon reconnection.

The Incident & The False Culprit

A two-line PR replaced our incremental patch endpoint with a "simpler" PUT /api/docs/:id full-document upload triggered on window online events. It passed QA on office fiber in 40 ms.

  1. T+00:00Deployed PR #1412. A product manager boarded a 4-hour train and edited a 14-page specification across intermittent tunnel LTE.
  2. T+02:15Three office teammates merged 24 consecutive revisions (`rev: 95` → `rev: 118`) into the same document.
  3. T+04:08The train exited the final tunnel. Two open browser tabs received `online` simultaneously and flushed `PUT /api/docs/42` with `rev: 94` and `updatedAt: 1763318529000`.
  4. T+04:09Gateway logs recorded `200 OK` from `client_a`, silently overwriting `rev: 118` and erasing 3 hours of teammate commits.
REJECTED PATCH

The Red Herring — Why the First Reflex Failed

In #inc-storage, the on-call engineer blamed missing optimistic locking and added an If-Match ETag check on PUT /api/docs/:id. Unit tests verified that stale writes returned 412 Precondition Failed.

In production, that patch inverted the data loss. When the next offline laptop reconnected, the server rejected the PUT with 412, and the client's catch handler purged the local IndexedDB outbox, destroying the traveler's edits instead.

Causal Chain Traversed

Step-by-step propagation across atomic concepts. Hover or click any step to inspect the underlying mechanism.

  1. 01Web APIs
    IndexedDB Transactions and Storage Quotas

    Offline edits persist in IndexedDB, where unawaited transaction aborts or quota limits drop local deltas.

  2. 02State & Architecture
    Local-First Sync & CRDTs

    Naive full-document PUT with wall-clock timestamps replaces CRDT delta merging, causing Last-Write-Wins clobbering.

  3. 03Web APIs
    BroadcastChannel and Web Locks API

    Two open tabs wake up simultaneously on reconnect and race to read and mutate the same IndexedDB outbox.

  4. 04State & Architecture
    Outbox Queues, Idempotency Keys & Causal Ordering

    Flaky cellular 4G retries replay outbox mutations without idempotency keys, duplicating or reordering operations.

  5. 05State & Architecture
    Multi-Tab State Synchronization & Leader Election

    Without a single elected sync leader, competing tabs open duplicate WebSockets and apply out-of-order server snapshots.

Five storage and concurrency boundaries fail in sequence when a partitioned client reconnects. Local transaction lifetimes, delta representation, cross-tab locking, retry idempotency, and leader election must all hold together or one side's commits are lost.

Failure Reproduction Lab

Step through the 4-hour tunnel scenario below to trace how IndexedDB transactions, CRDT state vectors, Web Locks, and idempotent outbox flushing converge without data loss.

Interactive Simulator

4-Hour Tunnel Sync: LWW Overwrite vs CRDT + Outbox Convergence

Step through how an offline edit on a train collides with a teammate's online commit across IndexedDB, Web Locks, and idempotent sync.

IndexedDB Tx
CRDT Delta
Web Lock
Outbox Queue
Step 1 / 4: 1. Offline Write Hits IndexedDB Quota & Transaction

Client A enters a tunnel and edits section 2. The write commits to IndexedDB inside an explicit readwrite transaction.

const tx = db.transaction(['ops'], 'readwrite');
tx.objectStore('ops').put(delta);
Internal State / Memory
network
offline
clientAVersion
v42+local1
teammateServerVersion
v42
indexedDbCommitted
true

Layer-by-Layer Dissection

01 → 02

Unawaited IndexedDB Transactions Drop Local Deltas

Offline keystrokes persist in IndexedDB Transactions and Storage Quotas, where awaiting fetch() or setTimeout() inside an active IDBTransaction auto-commits the transaction and throws TransactionInactiveError. Dropped writes leave local state behind user expectations before the network even returns.

Even when deltas survive locally, serializing a full JSON snapshot instead of causal operations bypasses Local-First Sync & CRDTs. A wall-clock PUT overwrites concurrent server revisions instead of merging state vectors.

02 → 03

Simultaneous Reconnects Trigger Multi-Tab Races

When LTE returns, Blink fires online in every open tab on the same origin within the same millisecond. Without BroadcastChannel and Web Locks API coordination, Tab 1 and Tab 2 read the identical pending CRDT delta from IndexedDB and race to advance the synced_seq cursor.

03 → 04

Flaky Cell Handoffs Replay Non-Idempotent Mutations

As the train switches cell towers, the first POST /sync times out on the client after the server already committed the batch. Without Outbox Queues, Idempotency Keys & Causal Ordering, automatic retries either fail sequence validation or append duplicate comments.

04 → 05

Competing WebSockets Apply Out-of-Order Snapshots

If both tabs maintain independent WebSocket streams during reconnect, they interleave server broadcasts with their own unacknowledged local outbox writes. Enforcing Multi-Tab State Synchronization & Leader Election via navigator.locks elects one tab to drain the outbox and broadcast converged state to followers.

Architectural Fix & Invariants

TS
// sync-leader.ts
export async function syncWithLeaderLock(docId: string, ydoc: YDoc): Promise<void> {
  await navigator.locks.request(`sync:${docId}`, { ifAvailable: true }, async (lock) => {
    if (!lock) return;
    for (const op of await readOutboxDeltas(docId)) {
      const res = await fetch(`/api/docs/${docId}/sync`, {
        method: 'POST',
        headers: { 'Content-Type': 'application/octet-stream', 'Idempotency-Key': op.id },
        body: op.update,
      });
      if (!res.ok) throw new Error(`Sync paused at ${op.id}: ${res.status}`);
      await markOutboxAcked(docId, op.id);
    }
    new BroadcastChannel(`doc:${docId}`).postMessage({ type: 'SYNCED', sv: ydoc.stateVector() });
  });
}

Architectural Invariants

  • Zero wall-clock overwrites: Remote merges use CRDT state vectors or causal clocks; full-document PUT snapshots are forbidden on collaborative records.
  • Synchronous IndexedDB chains: Every IDBTransaction executes reads and writes synchronously on IDBRequest callbacks with zero await fetch gaps.
  • Single-leader outbox drain: Only the tab holding navigator.locks.request('sync:<id>') flushes network deltas, and every request carries a deterministic Idempotency-Key.

Golden Rule: Never resolve offline divergence with wall-clock timestamps or HTTP status codes; persist causal deltas, elect one tab to flush them, and deduplicate every retry by key.

Primary Sources

Last verified: 2026-11-16