Streams API and Backpressure Mechanics
The WHATWG Streams API processes data incrementally through ReadableStream, TransformStream, and WritableStream pipelines. When a downstream consumer slows down, desiredSize drops to <= 0 and pipeTo() pauses upstream pull() calls so memory stays bounded.
Stream Queue Depth & Backpressure Simulator
Adjust producer chunk size, consumer drain rate, and highWaterMark to observe desiredSize backpressure.
Under the Hood
A stream pipeline works like a factory conveyor belt with a weight sensor at the packing station: when boxes pile up past highWaterMark, the sensor flips desiredSize <= 0 and halts the upstream feeder.
- Queuing Strategy:
ByteLengthQueuingStrategy({ highWaterMark })measures queued bytes instead of chunk counts, preventing 1-chunk-equals-100-MB spikes. - Pull-Based Flow: The engine invokes
source.pull(controller)only whilecontroller.desiredSize > 0, coordinating chunk scheduling across Event Loop microtasks. - Zero-Buffer Piping:
readable.pipeThrough(transform).pipeTo(writable)propagates backpressure and abort signals end-to-end, ideal for writing large downloads into OPFS.
const response = await fetch('/assets/archive.ndjson.gz');
const decompressed = response.body.pipeThrough(new DecompressionStream('gzip'));
const textStream = decompressed.pipeThrough(new TextDecoderStream());
const reader = textStream.getReader();
while (true) {
const { value, done } = await reader.read();
if (done) break;
processChunk(value);
}Problem Genealogy
- 2011 (
response.blob()/arrayBuffer()): Forced buffering the entire HTTP payload in RAM before user code could inspect byte 0. - 2016 (WHATWG Streams): Introduced
ReadableStreamonfetch().bodywith standardizedhighWaterMarkbackpressure. - 2022+ (
CompressionStream/TextDecoderStream): Added native C++-backedTransformStreamprimitives directly in the browser.
Production Pitfalls
- Ignoring
desiredSizein push sources: AWebSocketorEventSourcewrapper that callscontroller.enqueue()unconditionally still buffers unboundedly in RAM; pause or drop frames whendesiredSize <= 0. - Locked stream errors: Calling
response.json()after inspectingresponse.body.getReader()throwsTypeError: ReadableStream is locked; callresponse.clone()first.
- When not to use Streams: Small JSON payloads (< 1 MB) are faster and simpler with
await response.json()than setting up reader loops.
Connections & Logical Next Steps
Network-layer mocks must preserve ReadableStream chunking and backpressure for streaming responses.
ReadableStream desiredSize signals allow slow consumers to exert TCP-level backpressure on WebSocketStream.
Streaming SSR writes HTML chunks through a ReadableStream that respects socket backpressure.
Stream pull(), write(), and pipeTo() promises settle via microtasks between chunk transfers.
Streaming fetch bodies directly into FileSystemWritableFileStream avoids buffering gigabytes in RAM.