Node.js Streams & Pipelines

Streaming Large Datasets

Streams process data chunk-by-chunk sequentially without loading the entire dataset into RAM at once.

Types of Streams in Node.js

1. Readable: Source from which data is read (e.g. fs.createReadStream, req in Express).
2. Writable: Destination where data is written (e.g. fs.createWriteStream, res in Express).
3. Duplex: Stream that is both Readable and Writable (e.g. TCP Socket).
4. Transform: Duplex stream that modifies or transforms data as it is read/written (e.g. zlib.createGzip()).

Piping Streams

const fs = require("node:fs");
const zlib = require("node:zlib");
const { pipeline } = require("node:stream/promises");

async function compressFile() {
  try {
    await pipeline(
      fs.createReadStream("large_log.txt"),
      zlib.createGzip(),
      fs.createWriteStream("large_log.txt.gz")
    );
    console.log("File compression completed!");
  } catch (err) {
    console.error("Stream Pipeline failed:", err);
  }
}
compressFile();

TL;DR

• Streams save memory by handling data in chunks (Buffers).
• Use stream/promises pipeline() to pipe streams safely with automatic error handling.