Web APIsBaseline · Widely

Streams API and Backpressure Mechanics

Mental Model

The WHATWG Streams API processes data incrementally through ReadableStream, TransformStream, and WritableStream pipelines. When a downstream consumer slows down, desiredSize drops to <= 0 and pipeTo() pauses upstream pull() calls so memory stays bounded.

Interactive Simulator

Stream Queue Depth & Backpressure Simulator

Adjust producer chunk size, consumer drain rate, and highWaterMark to observe desiredSize backpressure.

Parameter Profile
highWaterMark64
chunkSize16
respectBackpressureOn
highWaterMark
64
chunkSize
16
respectBackpressure
On
Parameters evaluated in real time.

Under the Hood

A stream pipeline works like a factory conveyor belt with a weight sensor at the packing station: when boxes pile up past highWaterMark, the sensor flips desiredSize <= 0 and halts the upstream feeder.

  1. Queuing Strategy: ByteLengthQueuingStrategy({ highWaterMark }) measures queued bytes instead of chunk counts, preventing 1-chunk-equals-100-MB spikes.
  2. Pull-Based Flow: The engine invokes source.pull(controller) only while controller.desiredSize > 0, coordinating chunk scheduling across Event Loop microtasks.
  3. Zero-Buffer Piping: readable.pipeThrough(transform).pipeTo(writable) propagates backpressure and abort signals end-to-end, ideal for writing large downloads into OPFS.
JS
const response = await fetch('/assets/archive.ndjson.gz');
const decompressed = response.body.pipeThrough(new DecompressionStream('gzip'));
const textStream = decompressed.pipeThrough(new TextDecoderStream());

const reader = textStream.getReader();
while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  processChunk(value);
}

Problem Genealogy

  • 2011 (response.blob() / arrayBuffer()): Forced buffering the entire HTTP payload in RAM before user code could inspect byte 0.
  • 2016 (WHATWG Streams): Introduced ReadableStream on fetch().body with standardized highWaterMark backpressure.
  • 2022+ (CompressionStream / TextDecoderStream): Added native C++-backed TransformStream primitives directly in the browser.

Production Pitfalls

Production Pitfall
  • Ignoring desiredSize in push sources: A WebSocket or EventSource wrapper that calls controller.enqueue() unconditionally still buffers unboundedly in RAM; pause or drop frames when desiredSize <= 0.
  • Locked stream errors: Calling response.json() after inspecting response.body.getReader() throws TypeError: ReadableStream is locked; call response.clone() first.
When Not to Use
  • When not to use Streams: Small JSON payloads (< 1 MB) are faster and simpler with await response.json() than setting up reader loops.

Connections & Logical Next Steps

Seen in Architecture Autopsies