Autopsy #02: Anatomy of a Memory Leak on an 8-Hour Dashboard
SPA operations dashboard climbs from 120 MB to 2.4 GB RAM over an 8-hour shift and crashes with Aw, Snap!
The Incident & The False Culprit
A routine PR added an auto-resizing MutationObserver to the NOC telemetry table widget so column widths adjusted when live alerts streamed in. Every 10-minute QA smoke test passed with flat memory graphs.
Six hours into the Monday shift, dedicated wallboard monitors running the dispatch SPA began crashing with Chrome's Aw, Snap! (Out of Memory) error.
- T+00:00Initial JS heap settles at 120 MB after dashboard boot.
- T+02:00Operators switch between regional tabs; heap climbs at 280 MB/hour to 680 MB.
- T+05:30V8 Mark-Compact GC pauses hit 900 ms per frame; DevTools heap snapshots show 18,400 `Detached HTMLDivElement` nodes and 28 retained `FiberNode` trees.
- T+07:45Renderer process crosses 2.4 GB and terminates with `Aw, Snap!`.
The Red Herring — Why the First Reflex Failed
The first Slack reflex was to add telemetryRows = null and host = null inside the useEffect cleanup function. Local testing showed a brief drop in the DevTools memory timeline after manual GC, so the team shipped the hotfix.
Production crashes continued on the exact same 6-to-8-hour cadence. Nulling local bindings inside a cleanup closure does not sever the native C++ retainer edge held by an active MutationObserver or an uncancelled fetch stream.
Causal Chain Traversed
Step-by-step propagation across atomic concepts. Hover or click any step to inspect the underlying mechanism.
- 01JS RuntimeJavaScript Generational Garbage Collection
Unreclaimed objects survive Scavenge and promote to V8 Old Space, forcing frequent Mark-Compact pauses.
- 02JS RuntimeMemory Leaks, Retainers, and Detached DOM Trees
Live retainer edges from global registries pin entire detached DOM subtrees and Fiber nodes in memory.
- 03State & ArchitectureStale Closures in Hooks & Callbacks
Long-lived observer and timer callbacks capture stale React state arrays and DOM refs across route swaps.
- 04DOM & EventsMutationObserver and Batched Microtask Delivery
A MutationObserver registered without disconnect() holds a native C++ root edge to its JS callback.
- 05JS RuntimeAbortController and Cooperative Cancellation
Missing AbortController teardown prevents unified cancellation of observers, listeners, and polling streams.
The leak compounds across five runtime layers. Uncollected allocations promote into V8 Old Space, native DOM observers act as immortal GC roots, stale hook closures pull entire component trees into those roots, un-disconnected MutationObserver instances persist across route unmounts, and absent AbortController wiring leaves no unified teardown path.
Failure Reproduction Lab
Step through the 8-hour shift timeline below to inspect how V8 Old Space grows from 120 MB to 2.41 GB across route swaps, then advance to Step 5 to verify how a single AbortController severs every retainer edge on unmount.
8-Hour Shift Heap Climb: Retainer Chain vs AbortController Teardown
Step through how an un-disconnected MutationObserver and stale closure pin detached DOM subtrees in V8 Old Space until tab OOM.
Operator switches between Queue and Telemetry views. Component unmounts, but MutationObserver.disconnect() is never called.
observer.observe(widgetRoot, { childList: true, subtree: true });
// no cleanup- shiftHours
- 0.25
- heapMb
- 145
- detachedNodes
- 420
- gcMajorCollections
- 3
Layer-by-Layer Dissection
Surviving Allocations Promote to Immortal Retainers
Under JavaScript Generational Garbage Collection, short-lived objects die in V8's Young Generation during fast Scavenge cycles. Allocations that remain reachable across two scavenges promote into Old Space, where reclamation requires expensive Mark-Compact sweeps.
As shown in Memory Leaks, Retainers, and Detached DOM Trees, a single reachable JS reference to an unmounted <div class="chart-host"> pins the entire detached DOM subtree in Old Space.
Retainer Paths Originate in Stale Hook Closures
The retainer edge rarely comes from a global window.leak = node assignment. Instead, Stale Closures in Hooks & Callbacks capture the lexical environment of the render in which an effect was created:
- Captured state: A 40 MB
telemetryRowsarray fromt = 0h 15mstays bound inside the callback scope. - Captured ref:
widgetRef.currentpoints to the detached DOM container after the user navigates to another route.
Native MutationObserver Keeps the Closure Reachable
V8 fails to collect the stale closure after unmount because MutationObserver and Batched Microtask Delivery registers the callback inside Blink's C++ DOM mutation agent.
- Native GC root: Until
observer.disconnect()runs, Blink holds a strong edge to the JS callback. - Retained graph: Every variable and DOM node inside that callback closure remains reachable across route swaps.
Absent AbortController Prevents Unified Teardown
In the incident codebase, the effect also opened a fetch stream and attached a window resize listener. Without AbortController and Cooperative Cancellation, early returns skipped observer.disconnect() whenever host changed mid-flight, leaving partial subscriptions alive.
Architectural Fix & Invariants
useEffect(() => {
const host = widgetRef.current;
if (!host) return;
const controller = new AbortController();
const { signal } = controller;
const observer = new MutationObserver(() => syncLayout(host));
observer.observe(host, { childList: true, subtree: true });
signal.addEventListener('abort', () => observer.disconnect(), { once: true });
window.addEventListener('resize', () => syncLayout(host), { signal });
fetch('/api/telemetry', { signal })
.then((r) => r.json())
.then((rows) => renderChart(host, rows))
.catch((err) => { if (err.name !== 'AbortError') reportError(err); });
return () => controller.abort();
}, []);Architectural Invariants:
- Single-token teardown: Every
useEffectthat allocates a DOM observer,windowlistener, or network stream creates oneAbortControllerand returns() => controller.abort(). - Paired observer disconnect: Every
observer.observe()call registers{ once: true }onsignalto invokeobserver.disconnect()atomically. - Thin callback scopes: Observers read only the DOM node passed to their callback or a mutable
ref, never large historical state arrays from outer renders. - Soak-test verification: Navigation stress tests swap routes 50 times in Playwright and assert
Detachednode count returns to baseline aftergc().
Golden Rule: Nulling variables never collects memory held by native C++ DOM roots; bind every observer, listener, and fetch to one AbortSignal and abort on unmount.
Primary Sources
Last verified: 2026-11-16