Architecture Autopsies
Case #02Memory Exhaustion7 min read

Autopsy #02: Anatomy of a Memory Leak on an 8-Hour Dashboard

Production Symptom

SPA operations dashboard climbs from 120 MB to 2.4 GB RAM over an 8-hour shift and crashes with Aw, Snap!

The Incident & The False Culprit

A routine PR added an auto-resizing MutationObserver to the NOC telemetry table widget so column widths adjusted when live alerts streamed in. Every 10-minute QA smoke test passed with flat memory graphs.

Six hours into the Monday shift, dedicated wallboard monitors running the dispatch SPA began crashing with Chrome's Aw, Snap! (Out of Memory) error.

  1. T+00:00Initial JS heap settles at 120 MB after dashboard boot.
  2. T+02:00Operators switch between regional tabs; heap climbs at 280 MB/hour to 680 MB.
  3. T+05:30V8 Mark-Compact GC pauses hit 900 ms per frame; DevTools heap snapshots show 18,400 `Detached HTMLDivElement` nodes and 28 retained `FiberNode` trees.
  4. T+07:45Renderer process crosses 2.4 GB and terminates with `Aw, Snap!`.
REJECTED PATCH

The Red Herring — Why the First Reflex Failed

The first Slack reflex was to add telemetryRows = null and host = null inside the useEffect cleanup function. Local testing showed a brief drop in the DevTools memory timeline after manual GC, so the team shipped the hotfix.

Production crashes continued on the exact same 6-to-8-hour cadence. Nulling local bindings inside a cleanup closure does not sever the native C++ retainer edge held by an active MutationObserver or an uncancelled fetch stream.

Causal Chain Traversed

Step-by-step propagation across atomic concepts. Hover or click any step to inspect the underlying mechanism.

  1. 01JS Runtime
    JavaScript Generational Garbage Collection

    Unreclaimed objects survive Scavenge and promote to V8 Old Space, forcing frequent Mark-Compact pauses.

  2. 02JS Runtime
    Memory Leaks, Retainers, and Detached DOM Trees

    Live retainer edges from global registries pin entire detached DOM subtrees and Fiber nodes in memory.

  3. 03State & Architecture
    Stale Closures in Hooks & Callbacks

    Long-lived observer and timer callbacks capture stale React state arrays and DOM refs across route swaps.

  4. 04DOM & Events
    MutationObserver and Batched Microtask Delivery

    A MutationObserver registered without disconnect() holds a native C++ root edge to its JS callback.

  5. 05JS Runtime
    AbortController and Cooperative Cancellation

    Missing AbortController teardown prevents unified cancellation of observers, listeners, and polling streams.

The leak compounds across five runtime layers. Uncollected allocations promote into V8 Old Space, native DOM observers act as immortal GC roots, stale hook closures pull entire component trees into those roots, un-disconnected MutationObserver instances persist across route unmounts, and absent AbortController wiring leaves no unified teardown path.

Failure Reproduction Lab

Step through the 8-hour shift timeline below to inspect how V8 Old Space grows from 120 MB to 2.41 GB across route swaps, then advance to Step 5 to verify how a single AbortController severs every retainer edge on unmount.

Interactive Simulator

8-Hour Shift Heap Climb: Retainer Chain vs AbortController Teardown

Step through how an un-disconnected MutationObserver and stale closure pin detached DOM subtrees in V8 Old Space until tab OOM.

V8 Young Gen
Observer Root
Stale Closure
Heap: 145 MB
Step 1 / 5: 1. Route Swap Without Observer Teardown (t = 0h 15m)

Operator switches between Queue and Telemetry views. Component unmounts, but MutationObserver.disconnect() is never called.

observer.observe(widgetRoot, { childList: true, subtree: true });
// no cleanup
Internal State / Memory
shiftHours
0.25
heapMb
145
detachedNodes
420
gcMajorCollections
3

Layer-by-Layer Dissection

01 → 02

Surviving Allocations Promote to Immortal Retainers

Under JavaScript Generational Garbage Collection, short-lived objects die in V8's Young Generation during fast Scavenge cycles. Allocations that remain reachable across two scavenges promote into Old Space, where reclamation requires expensive Mark-Compact sweeps.

As shown in Memory Leaks, Retainers, and Detached DOM Trees, a single reachable JS reference to an unmounted <div class="chart-host"> pins the entire detached DOM subtree in Old Space.

02 → 03

Retainer Paths Originate in Stale Hook Closures

The retainer edge rarely comes from a global window.leak = node assignment. Instead, Stale Closures in Hooks & Callbacks capture the lexical environment of the render in which an effect was created:

  • Captured state: A 40 MB telemetryRows array from t = 0h 15m stays bound inside the callback scope.
  • Captured ref: widgetRef.current points to the detached DOM container after the user navigates to another route.
03 → 04

Native MutationObserver Keeps the Closure Reachable

V8 fails to collect the stale closure after unmount because MutationObserver and Batched Microtask Delivery registers the callback inside Blink's C++ DOM mutation agent.

  • Native GC root: Until observer.disconnect() runs, Blink holds a strong edge to the JS callback.
  • Retained graph: Every variable and DOM node inside that callback closure remains reachable across route swaps.
04 → 05

Absent AbortController Prevents Unified Teardown

In the incident codebase, the effect also opened a fetch stream and attached a window resize listener. Without AbortController and Cooperative Cancellation, early returns skipped observer.disconnect() whenever host changed mid-flight, leaving partial subscriptions alive.

Architectural Fix & Invariants

TS
useEffect(() => {
  const host = widgetRef.current;
  if (!host) return;
  const controller = new AbortController();
  const { signal } = controller;

  const observer = new MutationObserver(() => syncLayout(host));
  observer.observe(host, { childList: true, subtree: true });
  signal.addEventListener('abort', () => observer.disconnect(), { once: true });

  window.addEventListener('resize', () => syncLayout(host), { signal });
  fetch('/api/telemetry', { signal })
    .then((r) => r.json())
    .then((rows) => renderChart(host, rows))
    .catch((err) => { if (err.name !== 'AbortError') reportError(err); });

  return () => controller.abort();
}, []);

Architectural Invariants:

  • Single-token teardown: Every useEffect that allocates a DOM observer, window listener, or network stream creates one AbortController and returns () => controller.abort().
  • Paired observer disconnect: Every observer.observe() call registers { once: true } on signal to invoke observer.disconnect() atomically.
  • Thin callback scopes: Observers read only the DOM node passed to their callback or a mutable ref, never large historical state arrays from outer renders.
  • Soak-test verification: Navigation stress tests swap routes 50 times in Playwright and assert Detached node count returns to baseline after gc().

Golden Rule: Nulling variables never collects memory held by native C++ DOM roots; bind every observer, listener, and fetch to one AbortSignal and abort on unmount.

Primary Sources

Last verified: 2026-11-16