Architecture Autopsies
Case #10Cascading Latency7 min read

Autopsy #10: The Multi-Tab WebSocket Storm & Backpressure Collapse

Production Symptom

Users opening 8 tabs exhaust realtime gateway sockets and freeze the main thread during 400 msg/s bursts.

The Incident & The False Culprit

A dashboard feature PR enabled per-widget live ticker subscriptions inside useRealtimeFeed(). In single-tab local dev against a 5 msg/s mock stream, CPU usage stayed under 3%.

  1. T+00:00Market open triggered a 400 msg/s upstream price burst across 10,000 active browser sessions.
  2. T+02:00Edge telemetry recorded 80,000 concurrent WebSocket connections because power users kept 6 to 10 tabs open on the same origin.
  3. T+04:30RUM p95 Interaction to Next Paint (INP) jumped from 38 ms to 1,840 ms as each browser executed 3,200 `JSON.parse()` calls per second across 8 main threads.
  4. T+07:00A gateway node recycled, dropping 80,000 sockets and triggering a 14-minute `503` storm at the TLS terminator.
REJECTED PATCH

The Red Herring — Why the First Reflex Failed

On Slack, the team shipped a quick patch: setTimeout(() => reconnect(), 1000) paired with a 100 ms React setState throttle. It lowered render counts in local DevTools.

In production, it amplified the outage:

  • Socket multiplication stayed intact: 8 tabs still opened 8 TLS/WebSocket connections per user.
  • Main-thread parsing continued: Throttling setState still executed JSON.parse() for all 400 msg/s on the UI thread before discarding updates.
  • Fixed-delay retries synchronized failures: All 80,000 dropped sockets reconnected in lockstep at t + 1000 ms, pinning the TLS edge in a 503 loop.

Causal Chain Traversed

Step-by-step propagation across atomic concepts. Hover or click any step to inspect the underlying mechanism.

  1. 01Web APIs
    BroadcastChannel and Web Locks API

    Coordinates cross-tab socket ownership using Web Locks and fans out frames over BroadcastChannel.

  2. 02Web APIs
    Dedicated Web Workers and Transferables

    Moves JSON parsing and frame normalization off the main thread into a dedicated worker.

  3. 03Web APIs
    Streams API and Backpressure Mechanics

    Applies TransformStream queuing and desiredSize thresholds so slow tabs drop or coalesce stale frames.

  4. 04State & Architecture
    Multi-Tab State Synchronization & Leader Election

    Elects a single active tab to hold the upstream WebSocket and transfers ownership on tab close.

  5. 05Network & Performance
    WebSockets, SSE, Reconnection Jitter & Consumer Backpressure

    Enforces full-jitter exponential backoff on reconnects so gateway restarts do not trigger a thundering herd.

Five concurrency and backpressure failures compound across tabs:

  • Tab fan-out: Every tab opens its own WebSocket without cross-tab coordination.
  • Main-thread JSON storm: JSON.parse() runs on the UI thread for every frame.
  • Missing backpressure: Unbounded postMessage queues balloon memory in slow tabs.
  • Unguarded failover & synchronized retries: Tab closes drop feeds and fixed timers trigger reconnect storms.

Failure Reproduction Lab

Use the step machine below to trace how 8 open tabs amplify a 400 msg/s burst into main-thread stalls and a synchronized reconnect storm, then step into the Web Locks + Worker Stream + Full Jitter architecture.

Interactive Simulator

Multi-Tab WebSocket Storm vs Leader-Elected Backpressure

Trace how 8 open tabs amplify a 400 msg/s burst into main-thread stalls and reconnect storms, and how Web Locks + Worker Streams + Full Jitter isolate it.

RT Gateway (80k)
8 Tabs (8 Sockets)
Main Thread
Step 1 / 4: 1. Naive 8-Tab Fan-Out

Each tab opens its own WebSocket. At 10k active users with 8 tabs each, the gateway holds 80,000 concurrent sockets.

const ws = new WebSocket('wss://rt.example.com/feed');
// x8 per browser
Internal State / Memory
openTabs
8
socketsPerUser
8
gatewayConnections
80000
mainThreadBlockedMs
14

Layer-by-Layer Dissection

01 → 02

Cross-Tab Ownership Eliminates Duplicate Sockets and UI Parsing

Without BroadcastChannel and Web Locks API, eight same-origin tabs instantiate eight independent WebSocket connections. Acquiring navigator.locks.request('ws-leader', ...) restricts upstream socket ownership to one tab while BroadcastChannel fans out normalized deltas to followers:

  • Leader tab: Holds the single rt-socket-leader lock and broadcasts parsed deltas.
  • Follower tabs: Subscribe to BroadcastChannel and wait in the Web Locks queue for automatic failover.

Moving socket ownership to one tab still leaves JSON.parse on that leader's UI thread. Delegating socket I/O and frame decoding into Dedicated Web Workers and Transferables isolates the main thread from 400 msg/s allocation spikes and GC sweeps.

02 → 03

Worker Parsing Feeds Stream Backpressure

A dedicated worker protects the UI thread, yet an unbounded postMessage loop still floods backgrounded or slow follower tabs. Routing inbound frames through a TransformStream enforces Streams API and Backpressure Mechanics using controller.desiredSize and a CountQueuingStrategy:

  • desiredSize > 0: Forwards normalized ticker deltas downstream to BroadcastChannel.
  • desiredSize <= 0: Coalesces or drops superseded updates instead of queuing unbounded objects in memory.
03 → 04

Backpressure Requires Atomic Leader Failover

When the user closes the leader tab, follower tabs lose their upstream feed unless failover is automatic. Under Multi-Tab State Synchronization & Leader Election, navigator.locks handles handoff atomically:

  • Tab termination: Blink releases the held rt-socket-leader lock immediately on unload.
  • Next queued tab: Resolves its pending lock promise and resumes from the last sequence ID.
04 → 05

Leader Failover Connects Through Full-Jitter Backoff

Even with one socket per browser, a gateway restart drops 10,000 leader sockets simultaneously. Applying WebSockets, SSE, Reconnection Jitter & Consumer Backpressure replaces fixed timers with full-jitter exponential backoff (Math.random() * Math.min(30_000, 500 * 2 ** attempt)).

Randomized delays spread reconnect handshakes evenly across the backoff window instead of slamming the TLS edge in one wave.

Architectural Fix & Invariants

TS
// worker.ts: Leader-locked socket + full-jitter reconnect
let attempt = 0;

function runLeaderSocket(channel: BroadcastChannel) {
  const ws = new WebSocket('wss://rt.example.com/feed');
  ws.onmessage = (ev) => channel.postMessage({ type: 'DELTA', payload: JSON.parse(ev.data) });
  ws.onclose = () => {
    const delay = Math.random() * Math.min(30_000, 500 * 2 ** attempt++);
    setTimeout(() => runLeaderSocket(channel), delay);
  };
}

navigator.locks.request('rt-socket-leader', () => {
  runLeaderSocket(new BroadcastChannel('rt-feed'));
  return new Promise(() => {});
});

Architectural Invariants

  • Single socket per origin: Only the holder of the rt-socket-leader Web Lock opens an upstream WebSocket connection.
  • Zero main-thread frame parsing: All JSON.parse and payload normalization executes inside a Dedicated Worker.
  • Bounded queue depth: Stream consumers inspect desiredSize and coalesce stale frames when backpressure activates.
  • Full-jitter reconnects: Every reconnect timer samples uniformly from [0, min(cap, base * 2^attempt)] and resets attempt only after a stable open window.

Golden Rule: One browser equals one upstream socket; parse frames in a worker, shed load with stream backpressure, and jitter every reconnect.

Primary Sources

Last verified: 2026-11-16