Autopsy #08: Surviving a 4-Hour Tunnel: Local-First Sync & Conflicts
A user edits a document offline on a train and silently overwrites a teammate's merged changes upon reconnection.
The Incident & The False Culprit
A two-line PR replaced our incremental patch endpoint with a "simpler" PUT /api/docs/:id full-document upload triggered on window online events. It passed QA on office fiber in 40 ms.
- T+00:00Deployed PR #1412. A product manager boarded a 4-hour train and edited a 14-page specification across intermittent tunnel LTE.
- T+02:15Three office teammates merged 24 consecutive revisions (`rev: 95` → `rev: 118`) into the same document.
- T+04:08The train exited the final tunnel. Two open browser tabs received `online` simultaneously and flushed `PUT /api/docs/42` with `rev: 94` and `updatedAt: 1763318529000`.
- T+04:09Gateway logs recorded `200 OK` from `client_a`, silently overwriting `rev: 118` and erasing 3 hours of teammate commits.
The Red Herring — Why the First Reflex Failed
In #inc-storage, the on-call engineer blamed missing optimistic locking and added an If-Match ETag check on PUT /api/docs/:id. Unit tests verified that stale writes returned 412 Precondition Failed.
In production, that patch inverted the data loss. When the next offline laptop reconnected, the server rejected the PUT with 412, and the client's catch handler purged the local IndexedDB outbox, destroying the traveler's edits instead.
Causal Chain Traversed
Step-by-step propagation across atomic concepts. Hover or click any step to inspect the underlying mechanism.
- 01Web APIsIndexedDB Transactions and Storage Quotas
Offline edits persist in IndexedDB, where unawaited transaction aborts or quota limits drop local deltas.
- 02State & ArchitectureLocal-First Sync & CRDTs
Naive full-document PUT with wall-clock timestamps replaces CRDT delta merging, causing Last-Write-Wins clobbering.
- 03Web APIsBroadcastChannel and Web Locks API
Two open tabs wake up simultaneously on reconnect and race to read and mutate the same IndexedDB outbox.
- 04State & ArchitectureOutbox Queues, Idempotency Keys & Causal Ordering
Flaky cellular 4G retries replay outbox mutations without idempotency keys, duplicating or reordering operations.
- 05State & ArchitectureMulti-Tab State Synchronization & Leader Election
Without a single elected sync leader, competing tabs open duplicate WebSockets and apply out-of-order server snapshots.
Five storage and concurrency boundaries fail in sequence when a partitioned client reconnects. Local transaction lifetimes, delta representation, cross-tab locking, retry idempotency, and leader election must all hold together or one side's commits are lost.
Failure Reproduction Lab
Step through the 4-hour tunnel scenario below to trace how IndexedDB transactions, CRDT state vectors, Web Locks, and idempotent outbox flushing converge without data loss.
4-Hour Tunnel Sync: LWW Overwrite vs CRDT + Outbox Convergence
Step through how an offline edit on a train collides with a teammate's online commit across IndexedDB, Web Locks, and idempotent sync.
Client A enters a tunnel and edits section 2. The write commits to IndexedDB inside an explicit readwrite transaction.
const tx = db.transaction(['ops'], 'readwrite');
tx.objectStore('ops').put(delta);- network
- offline
- clientAVersion
- v42+local1
- teammateServerVersion
- v42
- indexedDbCommitted
- true
Layer-by-Layer Dissection
Unawaited IndexedDB Transactions Drop Local Deltas
Offline keystrokes persist in IndexedDB Transactions and Storage Quotas, where awaiting fetch() or setTimeout() inside an active IDBTransaction auto-commits the transaction and throws TransactionInactiveError. Dropped writes leave local state behind user expectations before the network even returns.
Even when deltas survive locally, serializing a full JSON snapshot instead of causal operations bypasses Local-First Sync & CRDTs. A wall-clock PUT overwrites concurrent server revisions instead of merging state vectors.
Simultaneous Reconnects Trigger Multi-Tab Races
When LTE returns, Blink fires online in every open tab on the same origin within the same millisecond. Without BroadcastChannel and Web Locks API coordination, Tab 1 and Tab 2 read the identical pending CRDT delta from IndexedDB and race to advance the synced_seq cursor.
Flaky Cell Handoffs Replay Non-Idempotent Mutations
As the train switches cell towers, the first POST /sync times out on the client after the server already committed the batch. Without Outbox Queues, Idempotency Keys & Causal Ordering, automatic retries either fail sequence validation or append duplicate comments.
Competing WebSockets Apply Out-of-Order Snapshots
If both tabs maintain independent WebSocket streams during reconnect, they interleave server broadcasts with their own unacknowledged local outbox writes. Enforcing Multi-Tab State Synchronization & Leader Election via navigator.locks elects one tab to drain the outbox and broadcast converged state to followers.
Architectural Fix & Invariants
// sync-leader.ts
export async function syncWithLeaderLock(docId: string, ydoc: YDoc): Promise<void> {
await navigator.locks.request(`sync:${docId}`, { ifAvailable: true }, async (lock) => {
if (!lock) return;
for (const op of await readOutboxDeltas(docId)) {
const res = await fetch(`/api/docs/${docId}/sync`, {
method: 'POST',
headers: { 'Content-Type': 'application/octet-stream', 'Idempotency-Key': op.id },
body: op.update,
});
if (!res.ok) throw new Error(`Sync paused at ${op.id}: ${res.status}`);
await markOutboxAcked(docId, op.id);
}
new BroadcastChannel(`doc:${docId}`).postMessage({ type: 'SYNCED', sv: ydoc.stateVector() });
});
}Architectural Invariants
- Zero wall-clock overwrites: Remote merges use CRDT state vectors or causal clocks; full-document
PUTsnapshots are forbidden on collaborative records. - Synchronous IndexedDB chains: Every
IDBTransactionexecutes reads and writes synchronously onIDBRequestcallbacks with zeroawait fetchgaps. - Single-leader outbox drain: Only the tab holding
navigator.locks.request('sync:<id>')flushes network deltas, and every request carries a deterministicIdempotency-Key.
Golden Rule: Never resolve offline divergence with wall-clock timestamps or HTTP status codes; persist causal deltas, elect one tab to flush them, and deduplicate every retry by key.
Primary Sources
Last verified: 2026-11-16