Module 11: Deployment, DevOps & Production Readiness

Process Management with PM2

In non-containerized production environments (bare-metal, VMs, VPS), a Node.js process crashing on an unhandled exception will terminate the entire service. PM2 is an enterprise-grade Production Process Manager that provides automatic restarts, built-in cluster mode without code modifications, log rotation, zero-downtime reloads, and system startup hooks.

Ecosystem Configuration (ecosystem.config.cjs)

Avoid running raw CLI commands in production. Declare execution topology via an ecosystem manifest:

JavaScript

module.exports = {
  apps: [
    {
      name: 'api-service',
      script: './dist/server.js',
      instances: 'max',              // Scales across all available CPU cores automatically
      exec_mode: 'cluster',          // Utilizes Node.js native cluster module under the hood
      watch: false,                  // Never enable watch in production
      max_memory_restart: '1G',      // Kills and re-spawns worker if it leaks over 1GB RAM
      autorestart: true,
      exp_backoff_restart_delay: 100,// Prevents restart thrashing if crashing on boot
      env_production: {
        NODE_ENV: 'production',
        PORT: 3000
      },
      // Centralized PM2 Log aggregation
      error_file: './logs/pm2-err.log',
      out_file: './logs/pm2-out.log',
      merge_logs: true,
      time: true
    }
  ]
};

Core PM2 Commands

Bash

# Start or reload via ecosystem configuration
pm2 start ecosystem.config.cjs --env production

# Zero-downtime rolling reload (workers reboot sequentially)
pm2 reload api-service

# Real-time resource dashboard (CPU, Memory, Event Loop latency)
pm2 monit

# Save current running process list to survive OS reboots
pm2 save
pm2 startup systemd # Generates and registers systemd boot script

Containerization: Production Docker Patterns

Running Node.js securely in Docker requires mitigating common vulnerabilities: running as root, leaking build secrets, bloating image sizes with dev dependencies, and failing to handle OS signals (SIGINT/SIGTERM).

1. Multi-Stage Production Dockerfile

This multi-stage Dockerfile uses a lean Alpine base, isolates native build dependencies, strips development tooling, and drops root privileges.

Dockerfile

# ---------------------------------------------------------------------
# Stage 1: Build Dependencies & Source Compilation
# ---------------------------------------------------------------------
FROM node:22-alpine AS builder

WORKDIR /usr/src/app

# Install native dependencies required for node-gyp, bcrypt, or canvas
RUN apk add --no-cache python3 make g++

# Copy package manifests first to leverage Docker layer caching
COPY package*.json ./

# Install ALL dependencies (including devDependencies like TypeScript)
RUN npm ci

# Copy source and build artifacts
COPY . .
RUN npm run build

# Prune devDependencies to keep only production packages
RUN npm prune --omit=dev

# ---------------------------------------------------------------------
# Stage 2: Minimal Production Runtime
# ---------------------------------------------------------------------
FROM node:22-alpine AS runner

# Tini acts as PID 1 to handle signal forwarding (SIGTERM) and reap zombie processes
RUN apk add --no-cache tini

WORKDIR /usr/src/app

ENV NODE_ENV=production
ENV PORT=3000

# Security: Never run containers as root
# 'node' user is built directly into official Node Alpine images (UID/GID 1000)
USER node

# Copy production dependencies and compiled code with non-root ownership
COPY --chown=node:node package*.json ./
COPY --chown=node:node --from=builder /usr/src/app/node_modules ./node_modules
COPY --chown=node:node --from=builder /usr/src/app/dist ./dist

EXPOSE 3000

# Wrap execution with tini init process
ENTRYPOINT ["/sbin/tini", "--"]

CMD ["node", "dist/server.js"]

2. Multi-Container Orchestration (docker-compose.yml)

A self-contained production-mirror environment linking the Node app, a persistent PostgreSQL database, and a Redis caching layer with health dependencies and resource constraints.

YAML

services:
  app:
    build:
      context: .
      target: runner
    container_name: node_api
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - PORT=3000
      - DATABASE_URL=postgresql://app_user:strong_password@postgres:5432/app_db?schema=public
      - REDIS_URL=redis://redis:6379
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    deploy:
      resources:
        limits:
          cpus: '1.5'
          memory: 1024M
    networks:
      - internal_network

  postgres:
    image: postgres:16-alpine
    container_name: postgres_db
    restart: unless-stopped
    environment:
      POSTGRES_USER: app_user
      POSTGRES_PASSWORD: strong_password
      POSTGRES_DB: app_db
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app_user -d app_db"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - internal_network

  redis:
    image: redis:7-alpine
    container_name: redis_cache
    restart: unless-stopped
    command: ["redis-server", "--appendonly", "yes", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"]
    volumes:
      - redisdata:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 3
    networks:
      - internal_network

volumes:
  pgdata:
  redisdata:

networks:
  internal_network:
    driver: bridge

Logging & Observability

1. Structured Logging with Pino

Standard console.log() executes synchronously when writing to stdout in terminal configurations, which can block the event loop. Pino is an asynchronous, high-throughput JSON logger designed to produce machine-parseable log lines for aggregation tools like Datadog, Grafana Loki, or AWS CloudWatch.

Bash

npm install pino pino-http

JavaScript

// src/utils/logger.js
import pino from 'pino';

export const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  // Redact sensitive payload properties automatically to protect PII
  redact: {
    paths: ['req.headers.authorization', 'req.headers.cookie', 'password', 'token', '*.creditCard'],
    censor: '[REDACTED]'
  },
  base: {
    env: process.env.NODE_ENV,
    pid: process.pid
  },
  timestamp: pino.stdTimeFunctions.isoTime
});

JavaScript

// Register HTTP Request Logging Middleware (src/app.js)
import express from 'express';
import { pinoHttp } from 'pino-http';
import crypto from 'node:crypto';
import { logger } from './utils/logger.js';

const app = express();

app.use(
  pinoHttp({
    logger,
    genReqId: (req) => req.headers['x-request-id'] || crypto.randomUUID(),
    customLogLevel: (req, res, err) => {
      if (res.statusCode >= 500 || err) return 'error';
      if (res.statusCode >= 400) return 'warn';
      return 'info';
    }
  })
);

2. Health-Check Endpoints: Liveness vs. Readiness

Container orchestrators (Kubernetes, AWS ECS) use probes to evaluate container state:

  • Liveness Probe (/health/live): Is the Node process running? If this fails, the orchestrator immediately kills and restarts the container.
  • Readiness Probe (/health/ready): Is the application able to handle incoming traffic? Checks database connections, Redis caches, and message brokers. If this fails, traffic routing pauses without restarting the container.

JavaScript

// src/routes/health.routes.js
import { Router } from 'express';
import { prisma } from '../config/db.js';
import { redis } from '../config/redis.js';

const router = Router();

// Liveness Probe
router.get('/health/live', (req, res) => {
  res.status(200).json({ status: 'live', uptime: process.uptime() });
});

// Readiness Probe
router.get('/health/ready', async (req, res) => {
  const checks = {
    database: false,
    redis: false
  };

  try {
    // 1. Verify DB connection
    await prisma.$queryRaw`SELECT 1`;
    checks.database = true;

    // 2. Verify Redis cache connection
    const redisPing = await redis.ping();
    checks.redis = (redisPing === 'PONG');

    const isHealthy = Object.values(checks).every(Boolean);

    res.status(isHealthy ? 200 : 503).json({
      status: isHealthy ? 'ready' : 'unhealthy',
      timestamp: new Date().toISOString(),
      dependencies: checks
    });
  } catch (err) {
    req.log.error({ err }, 'Readiness check failed');
    res.status(503).json({
      status: 'unhealthy',
      dependencies: checks
    });
  }
});

export default router;

3. Graceful Shutdown Signal Handling

When an orchestrator scales down or deploys a new container revision, it sends a SIGTERM signal. The server must close active TCP listeners, finish in-flight requests, close database connection pools, and exit cleanly.

JavaScript

// src/server.js
import { app } from './app.js';
import { prisma } from './config/db.js';
import { redis } from './config/redis.js';
import { logger } from './utils/logger.js';

const PORT = process.env.PORT || 3000;
const server = app.listen(PORT, () => {
  logger.info(`Server operational on port ${PORT}`);
});

function gracefulShutdown(signal) {
  logger.info(`Received ${signal}. Starting graceful shutdown sequence...`);

  // 1. Stop accepting new connections
  server.close(async () => {
    logger.info('HTTP server closed. Disconnecting persistent client pools...');

    try {
      // 2. Drain and disconnect database and Redis clients
      await prisma.$disconnect();
      await redis.quit();
      logger.info('Database & Cache connections closed safely.');

      process.exit(0);
    } catch (err) {
      logger.error({ err }, 'Error during resource teardown');
      process.exit(1);
    }
  });

  // Force kill if requests do not clear within a 15-second grace period
  setTimeout(() => {
    logger.error('Forced shutdown: Timed out waiting for lingering connections.');
    process.exit(1);
  }, 15000).unref();
}

process.on('SIGTERM', () => gracefulShutdown('SIGTERM'));
process.on('SIGINT', () => gracefulShutdown('SIGINT'));

Deployment Targets & Architectures

Deployment TargetParadigmInfrastructure ModelBest Suited ForTradeoffs
AWS ECS (Fargate)ContainerizedServerless ContainersProduction microservices, sustained trafficRequires VPC, ALB, IAM setup
AWS LambdaServerlessFunction-as-a-ServiceSporadic APIs, webhooks, cron jobsCold starts, 15m run limit, connection limits
AWS EC2Virtual MachineRaw Compute (IaaS)Custom OS needs, legacy monolithsManual OS patching, scaling, security groups
DigitalOcean (App Platform/Droplets)PaaS / VPSManaged runtime or VMSmall/Mid teams, cost-efficient scalingLimited custom low-level networking
Render / RailwayPaaSGit-driven DeploymentsStartups, staging, fast prototypingBandwidth/compute costs scale steeper at volume

CI/CD Pipeline: GitHub Actions

A production CI/CD pipeline validates code quality, runs test suites with real services, builds an optimized Docker image, and triggers an automated rollout.

Create .github/workflows/deploy.yml:

YAML

name: Continuous Integration & Deployment

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  validate-and-test:
    name: Code Quality & Automated Tests
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_USER: test_user
          POSTGRES_PASSWORD: test_password
          POSTGRES_DB: test_db
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

      redis:
        image: redis:7-alpine
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - name: Checkout Source Code
        uses: actions/checkout@v4

      - name: Setup Node.js Runtime
        uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: 'npm'

      - name: Install Dependencies
        run: npm ci

      - name: Static Code Analysis (Linter & Type Check)
        run: |
          npm run lint --if-present
          npm run typecheck --if-present

      - name: Execute Automated Test Suites
        env:
          DATABASE_URL: postgresql://test_user:test_password@localhost:5432/test_db
          REDIS_URL: redis://localhost:6379
          JWT_ACCESS_SECRET: github_actions_dummy_secret_key_32bytes_long
        run: npm run test:coverage

  build-and-ship:
    name: Build Multi-Stage Docker Image
    needs: validate-and-test
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Source Code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Authenticate to Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and Push Docker Image
        uses: docker/build-push-action@v5
        with:
          context: .
          target: runner
          push: true
          tags: |
            ghcr.io/${{ github.repository }}:latest
            ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      # Deployment dispatch step (e.g., Render, Railway, AWS ECS task update)
      - name: Trigger Remote Deployment Webhook
        env:
          DEPLOY_WEBHOOK_URL: ${{ secrets.RENDER_DEPLOY_HOOK }}
        run: |
          if [ -n "$DEPLOY_WEBHOOK_URL" ]; then
            curl -f -X POST "$DEPLOY_WEBHOOK_URL"
          fi

Recommended Portfolio Projects

  1. REST API with Full Auth & RBAC: Multi-tenant blog or e-commerce API with JWTs, refresh tokens, role checks, and pagination.
  2. Real-Time Collaborative App: Chat application or collaborative document editor with Socket.io and Redis adapters.
  3. High-Throughput File Processor: Video/image transcoding queue or CSV batch importer using streams and background worker queues.

Module 10: Performance Optimization & Concurrency

Multi-Processing & Concurrency Models

Because Node.js runs JavaScript on a single thread, heavy synchronous operations block the event loop, freezing I/O and network processing across the entire application. Scaling across hardware cores and executing parallel compute tasks relies on three distinct built-in primitives:

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚                              Concurrency Primitives                                    โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ Primitive          โ”‚ Memory Model                      โ”‚ Primary Use Case              โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ cluster            โ”‚ Separate OS processes (Isolated)  โ”‚ HTTP server throughput / scaleโ”‚
โ”‚ child_process      โ”‚ Separate OS processes (Isolated)  โ”‚ External CLIs, Python/C tools โ”‚
โ”‚ worker_threads     โ”‚ Single process, Shared V8 Memory  โ”‚ In-process CPU-bound workloadsโ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

1. Cluster Module: Utilizing Multi-Core CPUs

The node:cluster module forks multiple child processes (workers) that share the same server port. The primary process binds to the TCP port and delegates incoming socket connections across workers using a Round-Robin algorithm (on Linux/macOS) or via the OS network stack.

  • Each worker runs its own V8 instance, event loop, and memory heap (~30MB+ base RAM per worker).
  • Memory is not shared between processes. Inter-process communication (IPC) uses serialized message passing.

JavaScript

// server-cluster.js
import cluster from 'node:cluster';
import http from 'node:http';
import os from 'node:os';
import process from 'node:process';

const numCPUs = os.availableParallelism ? os.availableParallelism() : os.cpus().length;

if (cluster.isPrimary) {
  console.log(`Primary master process ${process.pid} is running.`);

  // Fork a worker process for each available physical/logical CPU core
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork();
  }

  // Graceful restart: Replace any worker process that crashes unexpectedly
  cluster.on('exit', (worker, code, signal) => {
    console.warn(`Worker ${worker.process.pid} terminated (code: ${code}, signal: ${signal}). Spawning replacement...`);
    cluster.fork();
  });
} else {
  // Workers share the underlying TCP socket connection on port 8000
  http.createServer((req, res) => {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ status: 'ok', workerPid: process.pid }));
  }).listen(8000);

  console.log(`Worker process ${process.pid} started and listening.`);
}

2. Child Process: Executing External System Tasks

The node:child_process module spawns independent OS processes to execute external shell utilities, system-level binaries, or separate runtime scripts (e.g., Python, Rust, Bash).

Node provides three primary spawning functions:

MethodSpawns Shell?I/O TransmissionMemory Buffer LimitBest Used For
spawn()No (by default)Streaming (Async streams)Unlimited (Streamed)Large payloads, long-running processes, video transcoding
exec()YesBuffered (Callback)Fixed (Default: 1MB)Quick shell commands (ls, git rev-parse)
fork()NoIPC Channel (Messaging)Node V8 instance overheadRunning standalone Node.js scripts with built-in process.send()
A. spawn() โ€” Streaming Large Datasets

spawn streams raw stdout/stderr chunks directly through event streams, keeping memory consumption constant:

JavaScript

import { spawn } from 'node:child_process';

// Convert a large video file via FFmpeg without buffering the entire output in RAM
const ffmpeg = spawn('ffmpeg', ['-i', 'input.mov', '-vcodec', 'h264', 'output.mp4']);

ffmpeg.stdout.on('data', (chunk) => {
  console.log(`Transcoding progress: ${chunk.length} bytes processed`);
});

ffmpeg.stderr.on('data', (errChunk) => {
  console.error(`FFmpeg log: ${errChunk.toString()}`);
});

ffmpeg.on('close', (exitCode) => {
  console.log(`FFmpeg process finished with code ${exitCode}`);
});
B. exec() โ€” Command Execution with Buffer Capping

exec passes the command string directly to the system shell (/bin/sh or cmd.exe). Never pass unvalidated user input to exec to avoid command injection vulnerabilities.

JavaScript

import { exec } from 'node:child_process';
import { promisify } from 'node:util';

const execAsync = promisify(exec);

async function getSystemUptime() {
  try {
    // maxBuffer protects against out-of-memory crashes if stdout exceeds size
    const { stdout, stderr } = await execAsync('uptime', { maxBuffer: 1024 * 1024 });
    return stdout.trim();
  } catch (err) {
    console.error('Command failed to execute:', err.message);
  }
}
C. fork() โ€” Node.js-to-Node.js IPC Channel

fork is a specialized variation of spawn designed specifically for Node.js modules. It establishes an internal, bidirectional JSON message channel between parent and child processes:

JavaScript

// parent.js
import { fork } from 'node:child_process';

const child = fork('./heavy-computation.js');

// Send payload to child
child.send({ task: 'ANALYZE_DATASET', items: [10, 20, 30] });

// Receive message from child
child.on('message', (result) => {
  console.log('Result received from child process:', result);
  child.kill(); // Terminate worker once finished
});

// heavy-computation.js
process.on('message', (message) => {
  if (message.task === 'ANALYZE_DATASET') {
    const computed = message.items.reduce((acc, curr) => acc + curr * 2, 0);
    // Send response back to parent
    process.send({ status: 'done', output: computed });
  }
});

3. Worker Threads: In-Process CPU Workloads

While cluster and child_process spin up new operating system processes, node:worker_threads runs multiple threads inside the same process.

                Node.js Single OS Process
 โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
 โ”‚                      Main Thread                       โ”‚
 โ”‚      (Event Loop + Libuv + HTTP Network Sockets)       โ”‚
 โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                โ”‚ MessageChannel / IPC     โ”‚ SharedArrayBuffer
                โ–ผ                          โ–ผ
 โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
 โ”‚     Worker Thread #1         โ”‚ โ”‚   Worker Thread #2    โ”‚
 โ”‚ (Isolated V8 Isolate Context)โ”‚ โ”‚  (Isolated V8 Isolate)โ”‚
 โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  • V8 Isolates: Each worker thread possesses its own distinct V8 Isolate (independent microtask queue, call stack, and garbage collector).
  • Zero-Copy Memory: Threads can share the exact same physical memory using SharedArrayBuffer and thread-safe synchronization via Atomics, avoiding serialization overhead.
Worker Thread Implementation

JavaScript

// main.js
import { Worker } from 'node:worker_threads';
import path from 'node:path';

export function runCpuIntensiveTask(workerData) {
  return new Promise((resolve, reject) => {
    const worker = new Worker(path.resolve('./worker.js'), {
      workerData // Data is cloned via the structured clone algorithm
    });

    worker.on('message', resolve);
    worker.on('error', reject);
    worker.on('exit', (code) => {
      if (code !== 0) {
        reject(new Error(`Worker stopped with exit code ${code}`));
      }
    });
  });
}

// worker.js
import { parentPort, workerData } from 'node:worker_threads';

function calculateHeavyPrimes(iterations) {
  let count = 0;
  for (let i = 2; i < iterations; i++) {
    let isPrime = true;
    for (let j = 2; j * j <= i; j++) {
      if (i % j === 0) { isPrime = false; break; }
    }
    if (isPrime) count++;
  }
  return count;
}

// Perform calculation without blocking main thread's event loop
const result = calculateHeavyPrimes(workerData.maxNumber);

// Return result to main thread
parentPort.postMessage({ result });

Memory Management & Profiling

1. V8 Garbage Collection Architecture

The V8 engine divides the JavaScript Memory Heap into two primary generations based on the Weak Generational Hypothesis (most allocated objects die young):

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚                                 V8 Heap Space                                โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚            New Space (Young)            โ”‚          Old Space (Tenured)       โ”‚
โ”‚  โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚  โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
โ”‚  โ”‚   From-Space     โ”‚    To-Space     โ”‚ โ”‚  โ”‚ Old Pointer & Data Space      โ”‚ โ”‚
โ”‚  โ”‚  (Allocations)   โ”‚   (Evacuation)  โ”‚ โ”‚  โ”‚ (Surviving Long-Lived Data)   โ”‚ โ”‚
โ”‚  โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚  โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
โ”‚    Scavenger Minor GC (Stop-the-world)  โ”‚    Major GC: Mark-Sweep-Compact    โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  • New Space (Nursery): Where allocations occur (const x = {}). Memory is managed via the Scavenger Cycle (Minor GC):
    • Divides space into active From-Space and inactive To-Space.
    • Alive objects are evacuated to To-Space; dead objects are discarded.
    • Roles flip. Objects that survive two consecutive Scavenger cycles are promoted to the Old Space.
  • Old Space: Contains long-lived entities (singletons, caches, modules, long-standing closures). Managed via Major GC (Mark-Sweep-Compact):
    • Marking: Traverses memory starting from root pointers (Call Stack, Global namespace) and marks reachable objects.
    • Sweeping: Frees unreferenced memory addresses back to the allocation free-list.
    • Compacting: Defragments physical memory blocks to prevent memory fragmentation.
Monitoring Heap Metrics Programmatically

JavaScript

import v8 from 'node:v8';

function logMemoryHealth() {
  const memory = process.memoryUsage();
  const heapStats = v8.getHeapStatistics();

  console.log({
    rss: `${(memory.rss / 1024 / 1024).toFixed(2)} MB`,             // Resident Set Size (Total RAM allocated to process)
    heapTotal: `${(memory.heapTotal / 1024 / 1024).toFixed(2)} MB`, // Allocated V8 memory
    heapUsed: `${(memory.heapUsed / 1024 / 1024).toFixed(2)} MB`,   // Actively occupied by JS objects
    external: `${(memory.external / 1024 / 1024).toFixed(2)} MB`,   // C++ bindings & Buffer objects
    heapLimit: `${(heapStats.heap_size_limit / 1024 / 1024).toFixed(2)} MB` // Crash ceiling before OOM
  });
}

2. Identifying and Fixing Memory Leaks

A memory leak occurs when unneeded references remain attached to root objects, preventing V8’s garbage collector from sweeping them.

Anti-Pattern A: Unbounded Global Caches

JavaScript

// LEAK: Array/Map grows indefinitely with no Time-To-Live (TTL) or eviction strategy
const requestCache = new Map();

export function handleRequest(req, res) {
  requestCache.set(req.id, { headers: req.headers, body: req.body }); // Memory permanently retained
  res.end('OK');
}

// FIX: Use an LRU (Least Recently Used) cache or WeakMap for auto-eviction
import { LRUCache } from 'lru-cache';
const safeCache = new LRUCache({ max: 500, ttl: 1000 * 60 * 5 });
Anti-Pattern B: Forgotten Event Listeners & Closures

JavaScript

// LEAK: Adding listeners inside route handlers without cleanup
app.get('/events', (req, res) => {
  const onUpdate = (data) => res.write(JSON.stringify(data));
  
  // Every incoming HTTP request registers a listener that holds onto the `res` stream
  globalEmitter.on('metric:update', onUpdate);

  // FIX: Unbind listener when the client closes the connection
  req.on('close', () => {
    globalEmitter.off('metric:update', onUpdate);
  });
});

3. Profiling Tools & Diagnostics

A. Profiling with Chrome DevTools

You can attach Google Chrome’s dedicated profiler directly to any live Node.js process:

Bash

# Start your script with the V8 inspector enabled
node --inspect=0.0.0.0:9229 server.js

# Or pause execution on the first line to diagnose startup allocations:
node --inspect-brk server.js
  1. Open Google Chrome and navigate to: chrome://inspect.
  2. Locate the Node process under Remote Target and click Inspect.
  3. Use the Memory tab to:
    • Take Heap Snapshot: Capture point-in-time heap states. Compare snapshots before and after stress tests to identify objects whose Delta count never decreases.
    • Allocation Instrumentation on Timeline: Observe live allocations over time to pinpoint functions creating memory spikes.
B. Capturing Programmatic Heap Snapshots on Demand

To diagnose memory leaks directly in production without stopping the service:

JavaScript

import v8 from 'node:v8';
import fs from 'node:fs';

export function dumpHeapSnapshot(destinationPath) {
  const snapshotStream = v8.getHeapSnapshot();
  const fileStream = fs.createWriteStream(destinationPath);
  snapshotStream.pipe(fileStream);
  console.log(`Heap snapshot written to ${destinationPath}`);
}

// Trigger snapshot when memory exceeds 85% of allowed heap
const STATS = v8.getHeapStatistics();
if (process.memoryUsage().heapUsed > STATS.heap_size_limit * 0.85) {
  dumpHeapSnapshot(`./heap-${Date.now()}.heapsnapshot`);
}
C. Clinic.js Diagnostics

Clinic.js is an industry-standard diagnostic tool that isolates performance bottlenecks, event loop lag, and memory leaks.

Bash

npm install -g clinic
  • Clinic Doctor: High-level health evaluation. Checks CPU spikes, Event Loop delay, and memory saturation:Bashclinic doctor --on-port 'autocannon http://localhost:3000' -- node server.js
  • Clinic Flame: Generates actionable CPU flame graphs to identify synchronous methods blocking the Call Stack:Bashclinic flame --on-port 'autocannon http://localhost:3000' -- node server.js
  • Clinic Bubbleprof: Profiles latency across asynchronous operations, libuv boundaries, and database queries.

Module 9: Testing & Quality Assurance

The Testing Pyramid in Node.js

A resilient testing strategy strikes a deliberate balance across three layers:

        /  E2E Tests  \       โ–ฒ  Slow, Expensive, High Confidence
       / Integration   \      โ”‚
      /   API Tests     \     โ”‚
     /โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\    โ”‚
    /     Unit Tests      \   โ–ผ  Fast, Cheap, Isolated
  • Unit Tests: Verify individual, pure functions or isolated business logic classes in complete isolation. All side effects (I/O, network, database) are mocked or stubbed.
  • Integration & API Tests: Validate that multiple components (routers, middleware, controllers, database models, caching layers) work together correctly using real or ephemeral test databases.
  • End-to-End (E2E) Tests: Exercise the full deployed system from an external client’s perspective.

Unit & Integration Testing: Jest vs. Vitest

Both Jest and Vitest share nearly identical matchers and assertion APIs (describe, it, expect, beforeEach).

  • Jest: The mature industry standard. However, it requires extra toolchains (babel-jest or ts-jest) to execute modern TypeScript or native ES Modules.
  • Vitest: Built on Vite. It natively understands ES Modules, TypeScript, and JSX with zero extra configuration, boots with multi-threaded worker pools, and shares the same configuration pipelines.

Setting Up Vitest for Node.js

Bash

npm install -D vitest

JavaScript

// vitest.config.js
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    globals: true,               // Exposes describe, it, expect globally
    environment: 'node',         // Uses Node runtime environment (not browser/jsdom)
    coverage: {
      provider: 'v8',            // Native V8 code coverage
      reporter: ['text', 'json', 'html'],
      exclude: ['node_modules/', 'dist/', 'tests/']
    },
    testTimeout: 10000           // 10s timeout for async operations
  }
});

Add test scripts to package.json:

JSON

{
  "scripts": {
    "test": "vitest run",
    "test:watch": "vitest",
    "test:coverage": "vitest run --coverage"
  }
}

1. Unit Testing: Business Logic Isolation

Unit tests should test edge cases, parameter validations, and deterministic calculations without any network or database activity.

JavaScript

// src/utils/pricing.js
export function calculateOrderDiscount(subtotal, userTier, couponRate = 0) {
  if (subtotal < 0) {
    throw new Error('Subtotal cannot be negative');
  }

  let tierDiscount = 0;
  if (userTier === 'GOLD') tierDiscount = 0.20;
  else if (userTier === 'SILVER') tierDiscount = 0.10;

  const totalDiscountRate = Math.min(tierDiscount + couponRate, 0.50); // Cap discount at 50%
  const finalAmount = subtotal * (1 - totalDiscountRate);

  return Math.round(finalAmount * 100) / 100;
}

JavaScript

// tests/unit/pricing.test.js
import { describe, it, expect } from 'vitest';
import { calculateOrderDiscount } from '../../src/utils/pricing.js';

describe('Pricing Utility: calculateOrderDiscount', () => {
  it('should return the original amount if no tier or coupon is applied', () => {
    const result = calculateOrderDiscount(100, 'STANDARD');
    expect(result).toBe(100);
  });

  it('should apply the correct discount for GOLD tier users', () => {
    const result = calculateOrderDiscount(200, 'GOLD');
    expect(result).toBe(160); // 20% off of 200
  });

  it('should combine tier and coupon discounts accurately', () => {
    const result = calculateOrderDiscount(100, 'SILVER', 0.15);
    expect(result).toBe(75); // 10% + 15% = 25% off
  });

  it('should cap cumulative discounts at exactly 50%', () => {
    const result = calculateOrderDiscount(100, 'GOLD', 0.40); // 20% + 40% = 60% -> capped at 50%
    expect(result).toBe(50);
  });

  it('should throw an explicit error for negative amounts', () => {
    expect(() => calculateOrderDiscount(-10, 'GOLD')).toThrow('Subtotal cannot be negative');
  });
});

2. Integration Testing: Controller & Database Layer

Integration tests execute operations across the database without mocking the query engine. A common best practice is spinning up an ephemeral test database (such as an in-memory database or a dedicated Docker test instance) and wiping state in lifecycle hooks.

JavaScript

// tests/integration/userRepository.test.js
import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest';
import { prisma } from '../../src/config/db.js';
import { userRepository } from '../../src/repositories/user.repository.js';

describe('Integration: User Repository with Database', () => {
  beforeAll(async () => {
    await prisma.$connect();
  });

  afterAll(async () => {
    await prisma.$disconnect();
  });

  beforeEach(async () => {
    // Truncate tables between tests to guarantee test isolation
    await prisma.order.deleteMany();
    await prisma.user.deleteMany();
  });

  it('should persist a new user and retrieve with relationships', async () => {
    const createdUser = await userRepository.create({
      email: 'test@domain.com',
      name: 'Integration Tester'
    });

    expect(createdUser).toHaveProperty('id');
    expect(createdUser.email).toBe('test@domain.com');

    // Verify record directly via DB client
    const fetched = await prisma.user.findUnique({
      where: { id: createdUser.id }
    });

    expect(fetched).not.toBeNull();
    expect(fetched?.name).toBe('Integration Tester');
  });

  it('should enforce unique constraint errors at database boundary', async () => {
    await userRepository.create({ email: 'duplicate@domain.com', name: 'User 1' });

    await expect(
      userRepository.create({ email: 'duplicate@domain.com', name: 'User 2' })
    ).rejects.toThrow();
  });
});

API Testing: Testing Endpoints with Supertest

Supertest binds to your Express app object and exercises real HTTP operations over an ephemeral local network port without needing you to manually boot the server on a live host port using app.listen().

Bash

npm install -D supertest

Structuring Express for Testability

Never invoke app.listen() directly inside your route definition file (app.js). Export the un-listened app instance, and create a separate entry point (server.js) that boots the listener:

src/
โ”œโ”€โ”€ app.js      โ”€โ”€▶ Configures middleware & routes (Exported for Supertest)
โ””โ”€โ”€ server.js   โ”€โ”€▶ Imports app and calls app.listen(PORT) (Bootstraps production)

Writing End-to-End API Specs

JavaScript

// tests/api/auth.api.test.js
import { describe, it, expect, beforeEach } from 'vitest';
import request from 'supertest';
import { app } from '../../src/app.js';
import { prisma } from '../../src/config/db.js';

describe('POST /api/v1/auth/register', () => {
  beforeEach(async () => {
    await prisma.user.deleteMany();
  });

  it('should return 201 Created and return a sanitized user entity', async () => {
    const payload = {
      email: 'alex@example.com',
      password: 'StrongPassword123!',
      name: 'Alex Rivera'
    };

    const res = await request(app)
      .post('/api/v1/auth/register')
      .send(payload)
      .set('Accept', 'application/json');

    expect(res.status).toBe(201);
    expect(res.headers['content-type']).toMatch(/json/);
    expect(res.body.status).toBe('success');
    expect(res.body.data.user).toMatchObject({
      email: 'alex@example.com',
      name: 'Alex Rivera'
    });
    // Critical security check: Sensitive credentials must never be in responses
    expect(res.body.data.user).not.toHaveProperty('password');
    expect(res.body.data.user).not.toHaveProperty('passwordHash');
  });

  it('should return 422 Unprocessable Entity when validation fails', async () => {
    const invalidPayload = {
      email: 'not-an-email',
      password: 'short'
    };

    const res = await request(app)
      .post('/api/v1/auth/register')
      .send(invalidPayload);

    expect(res.status).toBe(422);
    expect(res.body.status).toBe('fail');
    expect(res.body.errors).toBeInstanceOf(Array);
    expect(res.body.errors[0]).toHaveProperty('field');
  });

  it('should return 401 Unauthorized when accessing protected route without Bearer token', async () => {
    const res = await request(app)
      .get('/api/v1/users/profile');

    expect(res.status).toBe(401);
    expect(res.body.message).toContain('Authentication required');
  });

  it('should set an httpOnly Secure cookie when logging in', async () => {
    // Seed user
    await request(app).post('/api/v1/auth/register').send({
      email: 'login@test.com',
      password: 'Password123!',
      name: 'Tester'
    });

    const res = await request(app)
      .post('/api/v1/auth/login')
      .send({ email: 'login@test.com', password: 'Password123!' });

    expect(res.status).toBe(200);
    expect(res.body).toHaveProperty('accessToken');

    // Inspect Set-Cookie header
    const cookies = res.headers['set-cookie'];
    expect(cookies).toBeDefined();
    expect(cookies[0]).toMatch(/refreshToken=/);
    expect(cookies[0]).toMatch(/HttpOnly/i);
  });
});

Mocking & Spying

Mocking replaces dependencies that are slow, nondeterministic, costly, or that produce external side effects (credit card processing, third-party emails, AWS buckets).

  • Spy (vi.spyOn): Observes calls, arguments, and return values of a real method while optionally leaving original behavior intact.
  • Stub / Mock (vi.fn): Replaces the target implementation with a fake function returning pre-configured data.
  • Module Mock (vi.mock): Replaces an entire file or npm module with a mocked interface.

1. Mocking Database Queries

When writing unit tests for your service layer, mock the repository or ORM client to avoid requiring a real database connection.

JavaScript

// tests/unit/userService.test.js
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { userService } from '../../src/services/user.service.js';
import { userRepository } from '../../src/repositories/user.repository.js';
import { AppError } from '../../src/errors/AppError.js';

// Automatically mock all methods on userRepository
vi.mock('../../src/repositories/user.repository.js');

describe('User Service Unit Tests', () => {
  beforeEach(() => {
    vi.clearAllMocks(); // Clear call counts and return values between tests
  });

  it('should throw an AppError if the user already exists', async () => {
    // Arrange: Configure findByEmail to return an existing user
    userRepository.findByEmail.mockResolvedValue({
      id: 'usr_1',
      email: 'exists@domain.com'
    });

    // Act & Assert
    await expect(
      userService.registerUser({ email: 'exists@domain.com', password: 'password123' })
    ).rejects.toThrow(AppError);

    // Verify create was never triggered
    expect(userRepository.create).not.toHaveBeenCalled();
    expect(userRepository.findByEmail).toHaveBeenCalledTimes(1);
    expect(userRepository.findByEmail).toHaveBeenCalledWith('exists@domain.com');
  });

  it('should successfully hash the password and persist a new user', async () => {
    // Arrange
    userRepository.findByEmail.mockResolvedValue(null);
    userRepository.create.mockImplementation(async (data) => ({
      id: 'usr_new',
      ...data,
      createdAt: new Date()
    }));

    // Act
    const result = await userService.registerUser({
      email: 'new@domain.com',
      password: 'PlainPassword',
      name: 'New User'
    });

    // Assert
    expect(result).toHaveProperty('id', 'usr_new');
    expect(result.email).toBe('new@domain.com');
    // Ensure repository received a hashed string, not the plain password
    expect(userRepository.create).toHaveBeenCalledWith(
      expect.objectContaining({
        email: 'new@domain.com',
        password: expect.not.stringMatching('PlainPassword')
      })
    );
  });
});

2. Mocking Network Calls (MSW & Fetch/Axios Mocks)

Third-party APIs should never be called during test execution. Mock Service Worker (MSW) intercepts outgoing HTTP requests at the network transport level, making it library-agnostic (works with native fetch, axios, and got).

Bash

npm install -D msw

JavaScript

// tests/mocks/handlers.js
import { http, HttpResponse } from 'msw';

export const handlers = [
  // Intercept outgoing POST requests to third-party payment gateway
  http.post('https://api.stripe.com/v1/payment_intents', async ({ request }) => {
    const auth = request.headers.get('Authorization');
    
    if (!auth || !auth.includes('sk_test_valid')) {
      return new HttpResponse(null, { status: 401 });
    }

    return HttpResponse.json({
      id: 'pi_test_12345',
      status: 'succeeded',
      amount: 5000,
      currency: 'usd'
    });
  }),

  // Intercept Currency Conversion API
  http.get('https://api.exchangerates.io/v1/latest', () => {
    return HttpResponse.json({
      rates: { EUR: 0.92, GBP: 0.78 }
    });
  })
];

JavaScript

// tests/setupTests.js (Registered in vitest.config.js setupFiles)
import { beforeAll, afterAll, afterEach } from 'vitest';
import { setupServer } from 'msw/node';
import { handlers } from './mocks/handlers.js';

export const server = setupServer(...handlers);

beforeAll(() => server.listen({ onUnhandledRequest: 'error' })); // Throw if code calls an unmocked external API
afterEach(() => server.resetHandlers()); // Reset runtime overrides
afterAll(() => server.close());

3. Spying & Fake Timers

Spying on Methods and Built-ins

Use vi.spyOn to assert that internal utility methods or logging agents were notified with the expected metadata without altering their execution:

JavaScript

import { logger } from '../../src/utils/logger.js';
import { metrics } from '../../src/utils/metrics.js';

it('should increment metrics and log warning when cache misses occur', async () => {
  const logSpy = vi.spyOn(logger, 'warn');
  const metricSpy = vi.spyOn(metrics, 'increment').mockImplementation(() => {});

  await getCachedData('key_missing');

  expect(metricSpy).toHaveBeenCalledWith('cache.miss', 1);
  expect(logSpy).toHaveBeenCalledWith(expect.stringContaining('Cache miss encountered'));

  // Restore original non-mocked implementation
  metricSpy.mockRestore();
});
Controlling Time with Fake Timers

When testing rate-limiters, TTL expirations, or recurring background queues, never use real delays like setTimeout in test assertions. Fake timers fast-forward system time synchronously.

JavaScript

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { SessionManager } from '../../src/services/session.manager.js';

describe('Session TTL Expiration', () => {
  beforeEach(() => {
    vi.useFakeTimers(); // Intercept system Date, setTimeout, setInterval
  });

  afterEach(() => {
    vi.useRealTimers(); // Restore real system clock
  });

  it('should invalidate active sessions after 15 minutes of inactivity', () => {
    const sessionManager = new SessionManager();
    sessionManager.createSession('session_1');

    expect(sessionManager.isValid('session_1')).toBe(true);

    // Fast-forward virtual system clock by 14 minutes
    vi.advanceTimersByTime(14 * 60 * 1000);
    expect(sessionManager.isValid('session_1')).toBe(true);

    // Advance 2 more minutes (16 minutes total -> TTL expired)
    vi.advanceTimersByTime(2 * 60 * 1000);
    expect(sessionManager.isValid('session_1')).toBe(false);
  });
});

Test Suite Execution Summary Matrix

ToolPrimary PurposeScopeKey Methods / Mechanics
Vitest / JestTest runner & assertion harnessUnit, Integrationdescribe(), it(), expect(), toBe(), toMatchObject()
SupertestHTTP assertion agentREST API Endpointsrequest(app).get(), .send(), .set(), .expect(200)
MSW (Mock Service Worker)Outgoing HTTP interceptionExternal Serviceshttp.get(), http.post(), HttpResponse.json()
vi.fn() / vi.mock()Dependency replacementStubs & FakesmockResolvedValue(), mockImplementation(), toHaveBeenCalledWith()
vi.useFakeTimers()Clock manipulationTimeouts & Expirationsvi.advanceTimersByTime(), vi.setSystemTime()

Module 8: Real-Time Communication & Background Tasks

WebSockets & Real-Time Communication (Socket.io)

HTTP is a stateless, request-response protocol initiated solely by the client. For use cases requiring bidirectional, low-latency updates (chats, live metrics, collaboration tools), the WebSocket protocol (ws://, wss://) establishes a persistent, full-duplex TCP connection.

Socket.io builds on top of WebSockets by providing automatic reconnection, connection fallback (HTTP long-polling), packet buffering, acknowledgments, and horizontal scaling abstractions (rooms and namespaces).

Client โ”€โ”€▶ HTTP Upgrade Request โ”€โ”€▶ Server Handshake โ”€โ”€▶ Persistent Full-Duplex TCP Socket

Bash

npm install socket.io

Core Concepts: Namespaces, Rooms, and Broadcasting

  • Namespaces (io.of('/custom')): Multiplexes a single shared TCP connection into isolated logical channels with independent middleware, event handlers, and rooms.
  • Rooms (socket.join('room-name')): Server-side-only subdivisions within a namespace. Sockets can join/leave rooms to receive targeted broadcasts without other clients receiving that traffic.
  • Broadcasting: Emitting an event to subsets of connected clients.

JavaScript

// server.js
import http from 'node:http';
import express from 'express';
import { Server } from 'socket.io';
import { verifyAccessToken } from './utils/jwt.js';

const app = express();
const server = http.createServer(app);

// Initialize Socket.io with strict CORS
const io = new Server(server, {
  cors: {
    origin: 'https://app.domain.com',
    methods: ['GET', 'POST'],
    credentials: true
  },
  pingTimeout: 20000,
  pingInterval: 25000
});

// Authentication Middleware for WebSockets
io.use((socket, next) => {
  const token = socket.handshake.auth?.token || socket.handshake.headers?.authorization?.split(' ')[1];
  
  if (!token) {
    return next(new Error('Authentication error: Missing token'));
  }

  try {
    const user = verifyAccessToken(token);
    socket.data.user = user; // Attach authenticated identity to socket instance
    next();
  } catch (err) {
    next(new Error('Authentication error: Invalid or expired token'));
  }
});

// Connection lifecycle & event dispatching
io.on('connection', (socket) => {
  const userId = socket.data.user.sub;
  console.log(`Socket connected: ${socket.id} (User: ${userId})`);

  // Auto-join personal room for private notifications
  socket.join(`user:${userId}`);

  // Joining a group room
  socket.on('room:join', (roomId) => {
    socket.join(roomId);
    // Broadcast to everyone ELSE in the room
    socket.to(roomId).emit('user:joined', { userId, timestamp: Date.now() });
  });

  // Bidirectional messaging with Acknowledgments
  socket.on('message:send', async (payload, callback) => {
    try {
      const messageRecord = {
        senderId: userId,
        text: payload.text,
        roomId: payload.roomId,
        createdAt: new Date().toISOString()
      };

      // Broadcast message to everyone in the target room (including the sender)
      io.in(payload.roomId).emit('message:received', messageRecord);

      // Fire callback acknowledgment to the original sender
      callback({ status: 'delivered', id: 'msg_abc123' });
    } catch (err) {
      callback({ status: 'error', message: err.message });
    }
  });

  // Leaving a room
  socket.on('room:leave', (roomId) => {
    socket.leave(roomId);
    socket.to(roomId).emit('user:left', { userId });
  });

  // Handle client disconnection
  socket.on('disconnect', (reason) => {
    console.log(`Socket ${socket.id} disconnected: ${reason}`);
  });
});

server.listen(3000, () => console.log('Realtime server running on :3000'));

Event Emission Cheat Sheet

JavaScript

// Send to the current socket only
socket.emit('event', data);

// Send to all connected clients EXCEPT the sender
socket.broadcast.emit('event', data);

// Send to all clients in 'roomA' EXCEPT the sender
socket.to('roomA').emit('event', data);

// Send to all clients in 'roomA' INCLUDING the sender
io.in('roomA').emit('event', data);

// Send to a single targeted user by their personal room
io.to(`user:${targetUserId}`).emit('notification', { alert: 'New message' });

// Send to everyone connected to the entire server
io.emit('system:alert', { message: 'Server restarting in 5m' });

Horizontal Scaling with Redis Streams Adapter

Because WebSockets are stateful, running multiple Node.js instances behind a load balancer will isolate sockets to individual processes. Use the @socket.io/redis-adapter to publish and synchronize events across instances through Redis:

Bash

npm install @socket.io/redis-adapter ioredis

JavaScript

import { createAdapter } from '@socket.io/redis-adapter';
import Redis from 'ioredis';

const pubClient = new Redis(process.env.REDIS_URL);
const subClient = pubClient.duplicate();

// Distribute socket events across any number of clustered Node nodes
io.adapter(createAdapter(pubClient, subClient));

Background Jobs & Message Queues

Running long-running or CPU-heavy tasks (video transcoding, PDF generation, CSV imports, batch emails) synchronously inside an HTTP handler ties up the main thread and causes client timeouts.

Offload these operations to background worker queues.

HTTP POST /reports โ”€โ”€▶ [ Producer ] โ”€โ”€▶ BullMQ Job pushed to Redis โ”€โ”€▶ 202 Accepted
                                                                               โ”‚
[ Background Worker Process ] ◀โ”€โ”€ Picks up job from Redis Queue ◀โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

1. BullMQ (Redis-Backed Distributed Queue)

BullMQ is the NodeJS standard for message queues, handling concurrency, exponential backoff retries, delayed jobs, parent-child flows, and metrics.

Bash

npm install bullmq ioredis
Step 1: Redis Connection & Queue Producer

JavaScript

// queues/email.queue.js
import { Queue } from 'bullmq';

const connection = {
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: Number(process.env.REDIS_PORT) || 6379
};

export const emailQueue = new Queue('email-delivery', {
  connection,
  defaultJobOptions: {
    attempts: 4, // Retry up to 4 times on failure
    backoff: {
      type: 'exponential',
      delay: 2000 // 2s, 4s, 8s, 16s...
    },
    removeOnComplete: 1000, // Keep last 1,000 completed jobs in Redis for audit
    removeOnFail: 5000      // Keep failed jobs for inspection
  }
});

// Adding jobs to the queue from an Express controller
export async function scheduleWelcomeEmail(user) {
  return await emailQueue.add(
    'send-welcome',
    { userId: user.id, email: user.email, name: user.name },
    {
      priority: 2, // 1 is highest priority
      delay: 5000  // Deliver 5 seconds after invocation
    }
  );
}
Step 2: Dedicated Worker Processor

Run workers in a separate process or container so heavy compute operations never interfere with HTTP traffic.

JavaScript

// workers/email.worker.js
import { Worker } from 'bullmq';

const connection = {
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: Number(process.env.REDIS_PORT) || 6379
};

export const emailWorker = new Worker(
  'email-delivery',
  async (job) => {
    console.log(`Processing job ${job.id} of type ${job.name}...`);

    switch (job.name) {
      case 'send-welcome': {
        const { email, name } = job.data;
        // Update job progress (0% - 100%) for UI trackers
        await job.updateProgress(25);
        
        // Execute the external operation
        await sendEmailViaProvider({ to: email, template: 'welcome', context: { name } });
        
        await job.updateProgress(100);
        return { delivered: true, sentAt: new Date().toISOString() };
      }

      default:
        throw new Error(`Unhandled job type: ${job.name}`);
    }
  },
  {
    connection,
    concurrency: 5 // Process up to 5 jobs concurrently per worker instance
  }
);

emailWorker.on('completed', (job, returnvalue) => {
  console.log(`Job ${job.id} completed. Result:`, returnvalue);
});

emailWorker.on('failed', (job, err) => {
  console.error(`Job ${job.id} failed after ${job.attemptsMade} attempts: ${err.message}`);
});

2. In-Process Cron Jobs (node-cron)

For simple, single-instance recurring scheduled tasks (e.g., nightly cache purges, daily report aggregation), node-cron provides a straightforward cron syntax parser.

Bash

npm install node-cron

JavaScript

// jobs/scheduler.js
import cron from 'node-cron';

// Cron Syntax: [second] [minute] [hour] [day-of-month] [month] [day-of-week]
// Runs every night at exactly 02:00 AM (server local time)
cron.schedule('0 0 2 * * *', async () => {
  console.log('Running daily database backup and orphan file cleanup...');
  try {
    await purgeExpiredSessions();
    console.log('Maintenance finished successfully');
  } catch (err) {
    console.error('Scheduled task failed:', err);
  }
}, {
  scheduled: true,
  timezone: 'UTC' // Always lock cron execution to UTC
});

Warning on Clustered Deployments: If running multiple replicas of your app behind a load balancer, standard node-cron schedules execute on every single running instance, causing duplicate executions. For multi-replica setups, use BullMQ Repeatable Jobs, which use Redis to enforce single execution across clusters:

JavaScript

await queue.add('nightly-sync', {}, { repeat: { pattern: '0 2 * * *' } });

Email & Transactional Notifications

Transactional emails (password resets, invoice PDFs, magic links) are critical delivery pathways. Node.js applications use SMTP via Nodemailer or modern HTTP-based REST APIs (Resend, SendGrid, Postmark).

Method 1: Nodemailer (SMTP Engine)

Nodemailer connects directly to an arbitrary SMTP server (AWS SES, Mailgun, or local Postfix).

Bash

npm install nodemailer

JavaScript

// services/nodemailer.service.js
import nodemailer from 'nodemailer';

// Create persistent transporter pool
const transporter = nodemailer.createTransport({
  host: process.env.SMTP_HOST,
  port: Number(process.env.SMTP_PORT) || 587,
  secure: false, // true for 465, false for 587 (STARTTLS)
  auth: {
    user: process.env.SMTP_USER,
    pass: process.env.SMTP_PASS
  },
  pool: true,     // Keep connections alive to reuse TCP sockets
  maxConnections: 5
});

export async function sendPasswordResetEmail({ to, token }) {
  const resetLink = `https://app.domain.com/reset-password?token=${token}`;

  const mailOptions = {
    from: '"Security Team" <security@domain.com>',
    to,
    subject: 'Action Required: Reset Your Password',
    text: `Reset your password by opening this URL: ${resetLink}`,
    html: `
      <div style="font-family: sans-serif; padding: 20px;">
        <h2>Password Reset Request</h2>
        <p>Click the link below to set a new password. This link expires in 15 minutes.</p>
        <a href="${resetLink}" style="display:inline-block;padding:10px 20px;background:#0070f3;color:#fff;text-decoration:none;border-radius:5px;">Reset Password</a>
      </div>
    `
  };

  const info = await transporter.sendMail(mailOptions);
  return { messageId: info.messageId };
}

Method 2: Modern HTTP APIs (Resend SDK)

Direct SMTP can be slow, vulnerable to firewall blocking, and requires maintaining persistent connections. Modern platforms use transactional HTTP APIs with lower overhead and built-in React template rendering.

Bash

npm install resend

JavaScript

// services/resend.service.js
import { Resend } from 'resend';

const resend = new Resend(process.env.RESEND_API_KEY);

export async function sendInvoiceEmail({ to, invoiceId, pdfBuffer }) {
  const { data, error } = await resend.emails.send({
    from: 'billing@domain.com',
    to: [to],
    subject: `Invoice #${invoiceId} is available`,
    html: '<p>Attached is your monthly billing statement. Thank you for your business!</p>',
    attachments: [
      {
        filename: `invoice-${invoiceId}.pdf`,
        content: pdfBuffer // Pass raw binary Buffer directly without writing to disk
      }
    ],
    tags: [
      { name: 'category', value: 'invoicing' }
    ]
  });

  if (error) {
    throw new Error(`Email delivery failed: ${error.message}`);
  }

  return data; // { id: '49af6ee4-4806-44c6-8d8d-29406085ff87' }
}

Architectural Pattern: Combining WebSockets, Queues, and Email

1. Client POST /api/reports/export
   โ”‚
   โ”œโ”€โ”€▶ Controller pushes 'generate-pdf' task to BullMQ
   โ””โ”€โ”€▶ Immediately replies: 202 Accepted { jobId: 'job_456' }

2. BullMQ Worker picks up 'generate-pdf'
   โ”‚
   โ”œโ”€โ”€▶ Compiles heavy dynamic report
   โ”œโ”€โ”€▶ Sends email with file attachment via Resend
   โ””โ”€โ”€▶ Emits completion to Redis PubSub

3. Socket.io Cluster receives Redis PubSub
   โ”‚
   โ””โ”€โ”€▶ Dispatches socket event to target user room:
        io.to(`user:${userId}`).emit('report:ready', { downloadUrl: '...' });

Module 7: Authentication, Authorization & Security

Authentication Strategies

Authentication verifies the identity of an incoming request. Secure production systems never store plain text passwords, prefer modern cryptographic key derivation functions, and pick stateful or stateless session models based on scalability and revocation needs.

Credential Verification:
Client Password โ”€โ”€▶ Argon2 / Bcrypt Hash Check โ”€โ”€▶ Issued Auth Artifact
                                                          โ”‚
          โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
          โ–ผ                                                                               โ–ผ
Stateless (JWT Pair)                                                            Stateful (Session Cookie)
- Access Token (Short-lived: 15m)                                               - Opaque Session ID
- Refresh Token (Long-lived: 7d, rotated)                                       - In-memory/Redis lookup
- Encoded claims signed with private key                                        - Revocation is instant

1. Password Hashing: bcrypt vs. argon2

General-purpose hash algorithms (MD5, SHA-256, SHA-512) are built for throughput and are vulnerable to brute-force attacks on GPUs/ASICs. Password hashing functions must be deliberately slow and resource-heavy.

  • bcrypt: A Battle-tested key derivation function based on the Blowfish cipher. It is primarily CPU-hard (controlled via work factor rounds, typically 10 to 12).
  • argon2: The winner of the Password Hashing Competition (PHC). It is both CPU-hard and memory-hard, providing resistance against custom hardware and GPU-assisted cracking. Use the argon2id variant, which combines data-independent and data-dependent memory access.

Bash

npm install argon2 bcrypt

JavaScript

import argon2 from 'argon2';
import bcrypt from 'bcrypt';

// Argon2id Implementation (Recommended modern default)
export async function hashPasswordArgon(plainPassword) {
  return await argon2.hash(plainPassword, {
    type: argon2.argon2id,
    memoryCost: 2 ** 16, // 64 MB
    timeCost: 3,         // 3 iterations
    parallelism: 1
  });
}

export async function verifyPasswordArgon(hash, plainPassword) {
  // Constant-time comparison handled internally to prevent timing attacks
  return await argon2.verify(hash, plainPassword);
}

// bcrypt Alternative
const SALT_ROUNDS = 12;

export async function hashPasswordBcrypt(plainPassword) {
  return await bcrypt.hash(plainPassword, SALT_ROUNDS);
}

export async function verifyPasswordBcrypt(plainPassword, hash) {
  return await bcrypt.compare(plainPassword, hash);
}

2. Stateless Token Authentication: JWT (Access & Refresh Tokens)

A JSON Web Token (JWT) contains three base64url-encoded parts separated by dots: Header.Payload.Signature.

  • Signature: Created by hashing the header and payload with a private secret or asymmetric key (RS256 / EdDSA).
  • Storage Threat Model: Never store tokens in browser localStorage or sessionStorage (they are directly accessible via Cross-Site Scripting / XSS). Keep the Access Token in memory (or private module variable) and the Refresh Token in an httpOnly, Secure cookie.

Bash

npm install jsonwebtoken

JavaScript

// utils/jwt.js
import jwt from 'jsonwebtoken';

const ACCESS_SECRET = process.env.JWT_ACCESS_SECRET;
const REFRESH_SECRET = process.env.JWT_REFRESH_SECRET;

export function generateTokenPair(user) {
  const payload = {
    sub: user.id,
    role: user.role
  };

  const accessToken = jwt.sign(payload, ACCESS_SECRET, {
    expiresIn: '15m',
    issuer: 'api.domain.com',
    audience: 'app.domain.com'
  });

  const refreshToken = jwt.sign({ sub: user.id }, REFRESH_SECRET, {
    expiresIn: '7d',
    issuer: 'api.domain.com'
  });

  return { accessToken, refreshToken };
}

export function verifyAccessToken(token) {
  return jwt.verify(token, ACCESS_SECRET, {
    issuer: 'api.domain.com',
    audience: 'app.domain.com'
  });
}

export function verifyRefreshToken(token) {
  return jwt.verify(token, REFRESH_SECRET, {
    issuer: 'api.domain.com'
  });
}
Refresh Token Rotation & Invalidation Flow

To mitigate token theft, issue a new Refresh Token on every refresh call and track token families in Redis:

JavaScript

// controllers/auth.controller.js
export async function refresh(req, res) {
  const incomingRefreshToken = req.cookies.refreshToken;
  if (!incomingRefreshToken) {
    return res.status(401).json({ error: 'Refresh token missing' });
  }

  try {
    const decoded = verifyRefreshToken(incomingRefreshToken);
    
    // Check if token was already used or blacklisted in Redis
    const isRevoked = await redis.get(`bl:${incomingRefreshToken}`);
    if (isRevoked) {
      // Possible token theft: invalidate all sessions for this user family
      await redis.del(`user_session:${decoded.sub}`);
      return res.status(403).json({ error: 'Token reuse detected. Please log in again.' });
    }

    // Blacklist previous refresh token until its expiration
    await redis.set(`bl:${incomingRefreshToken}`, '1', 'EX', 7 * 86400);

    // Issue brand-new pair
    const user = await db.user.findUnique({ where: { id: decoded.sub } });
    const { accessToken, refreshToken: newRefreshToken } = generateTokenPair(user);

    // Send new refresh token in secured cookie
    res.cookie('refreshToken', newRefreshToken, {
      httpOnly: true,
      secure: process.env.NODE_ENV === 'production',
      sameSite: 'strict',
      path: '/api/v1/auth/refresh',
      maxAge: 7 * 24 * 60 * 60 * 1000
    });

    res.status(200).json({ accessToken });
  } catch (err) {
    return res.status(401).json({ error: 'Invalid or expired refresh token' });
  }
}

3. Stateful Session-Based Auth with Hardened Cookies

Stateful auth uses an opaque session identifier stored in client cookies. The server stores user state in Redis or a database.

JavaScript

import session from 'express-session';
import { RedisStore } from 'connect-redis';
import { redis } from '../config/redis.js';

export function configureSession(app) {
  app.use(
    session({
      store: new RedisStore({ client: redis, prefix: 'sess:' }),
      name: '__Host-sid', // __Host- prefix guarantees Secure, path=/, and no subdomains
      secret: process.env.SESSION_SECRET,
      resave: false,
      saveUninitialized: false,
      cookie: {
        httpOnly: true, // Inaccessible to document.cookie (Defends against XSS exfiltration)
        secure: process.env.NODE_ENV === 'production', // Transmitted strictly over HTTPS
        sameSite: 'lax', // Defends against Cross-Site Request Forgery (CSRF)
        path: '/',
        maxAge: 1000 * 60 * 60 * 8 // 8 hours
      }
    })
  );
}
  • httpOnly: true: JavaScript cannot read the cookie string, preventing stolen sessions via injected XSS scripts.
  • SameSite: 'strict' | 'lax':
    • Strict: The cookie is never sent in cross-site requests (e.g., following a link from an external site).
    • Lax: Default safe option. The cookie is withheld on cross-site subrequests (images, iframes), but sent when a user navigates to the origin site directly (top-level GET).
  • Secure: true: The browser sends the cookie only when the connection uses TLS/HTTPS.

4. OAuth 2.0 with Passport.js (Google Strategy)

OAuth 2.0 delegates authentication to an Identity Provider (IdP) without exposing user passwords to the consuming application.

Bash

npm install passport passport-google-oauth20

JavaScript

// config/passport.js
import passport from 'passport';
import { Strategy as GoogleStrategy } from 'passport-google-oauth20';

passport.use(
  new GoogleStrategy(
    {
      clientID: process.env.GOOGLE_CLIENT_ID,
      clientSecret: process.env.GOOGLE_CLIENT_SECRET,
      callbackURL: 'https://api.domain.com/api/v1/auth/google/callback',
      scope: ['profile', 'email']
    },
    async (accessToken, refreshToken, profile, done) => {
      try {
        const email = profile.emails?.[0]?.value;
        const googleId = profile.id;

        // Upsert user into database
        let user = await db.user.findFirst({
          where: { OR: [{ googleId }, { email }] }
        });

        if (!user) {
          user = await db.user.create({
            data: {
              email,
              name: profile.displayName,
              googleId,
              avatar: profile.photos?.[0]?.value
            }
          });
        }

        return done(null, user);
      } catch (err) {
        return done(err, null);
      }
    }
  )
);

// Routes
// 1. Redirect to Google Consent Screen
app.get('/api/v1/auth/google', passport.authenticate('google', { session: false }));

// 2. Callback endpoint
app.get(
  '/api/v1/auth/google/callback',
  passport.authenticate('google', { session: false, failureRedirect: '/login' }),
  (req, res) => {
    // Generate internal JWTs or sessions for the authenticated user
    const { accessToken, refreshToken } = generateTokenPair(req.user);
    res.cookie('refreshToken', refreshToken, { httpOnly: true, secure: true, sameSite: 'strict' });
    res.redirect(`https://app.domain.com/oauth/success?token=${accessToken}`);
  }
);

Authorization: Role-Based Access Control (RBAC)

Authentication proves who you are; Authorization determines what you are allowed to do.

Request โ”€โ”€▶ [ verifyToken ] โ”€โ”€▶ req.user attached โ”€โ”€▶ [ checkRole / checkPermission ] โ”€โ”€▶ Next()

Middleware Implementation

JavaScript

// middleware/auth.middleware.js
import { verifyAccessToken } from '../utils/jwt.js';

export function authenticate(req, res, next) {
  const authHeader = req.headers.authorization;
  if (!authHeader?.startsWith('Bearer ')) {
    return res.status(401).json({ status: 'fail', message: 'Authentication required' });
  }

  const token = authHeader.split(' ')[1];
  try {
    const payload = verifyAccessToken(token);
    req.user = payload; // { sub: 'usr_123', role: 'editor' }
    next();
  } catch (err) {
    return res.status(401).json({ status: 'fail', message: 'Invalid or expired token' });
  }
}

// Hierarchical Role Authorization Middleware
export function authorizeRoles(...allowedRoles) {
  return (req, res, next) => {
    if (!req.user || !allowedRoles.includes(req.user.role)) {
      return res.status(403).json({
        status: 'fail',
        message: 'Forbidden: Insufficient privileges'
      });
    }
    next();
  };
}

// Granular Permission Matrix Middleware
const ROLE_PERMISSIONS = {
  viewer: ['read:articles'],
  editor: ['read:articles', 'create:articles', 'update:articles'],
  admin:  ['read:articles', 'create:articles', 'update:articles', 'delete:articles', 'manage:users']
};

export function requirePermission(permission) {
  return (req, res, next) => {
    const userRole = req.user?.role;
    const permissions = ROLE_PERMISSIONS[userRole] || [];

    if (!permissions.includes(permission)) {
      return res.status(403).json({
        status: 'fail',
        message: `Forbidden: Missing required permission [${permission}]`
      });
    }
    next();
  };
}

JavaScript

// routes/article.routes.js
import { Router } from 'express';
import { authenticate, requirePermission } from '../middleware/auth.middleware.js';

const router = Router();

router.delete(
  '/articles/:id',
  authenticate,
  requirePermission('delete:articles'),
  (req, res) => res.json({ message: 'Article deleted successfully' })
);

Backend Security Hardening

1. Setting HTTP Security Headers with Helmet

Helmet adds secure HTTP response headers to defend against clickjacking, sniffed MIME types, and cross-site injections.

Bash

npm install helmet

JavaScript

import helmet from 'helmet';

app.use(
  helmet({
    // Content Security Policy: Whitelist origins that can execute scripts / load styles
    contentSecurityPolicy: {
      directives: {
        defaultSrc: ["'self'"],
        scriptSrc: ["'self'"],
        styleSrc: ["'self'", "'unsafe-inline'"],
        imgSrc: ["'self'", 'data:', 'https://cdn.domain.com'],
        connectSrc: ["'self'", 'https://api.domain.com'],
        frameAncestors: ["'none'"] // Blocks site from being embedded inside an iframe (Clickjacking defense)
      }
    },
    crossOriginEmbedderPolicy: true,
    crossOriginOpenerPolicy: { policy: 'same-origin' },
    crossOriginResourcePolicy: { policy: 'same-origin' },
    dnsPrefetchControl: { allow: false },
    frameguard: { action: 'deny' }, // Older X-Frame-Options equivalent
    hidePoweredBy: true,           // Strips 'X-Powered-By: Express' (prevents fingerprinting)
    hsts: {                        // Enforces HTTPS for 1 year including subdomains
      maxAge: 31536000,
      includeSubDomains: true,
      preload: true
    },
    noSniff: true,                 // Sets X-Content-Type-Options: nosniff
    xssFilter: true
  })
);

2. Defending Against Injection & Web Attacks

Attack Vector       Vulnerable Pattern                       Defense
โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
SQL Injection       String concatenation in raw SQL          Parameterized queries ($1, ?), ORMs
NoSQL Injection     Passing unvalidated objects ($ne, $gt)   Input schema casting (Zod), mongo-sanitize
XSS                 Trusting untrusted HTML or reflections   Contextual encoding, Strict CSP, httpOnly cookies
CSRF                Unchecked cross-origin mutations         SameSite cookies, CSRF Double-Submit Tokens
Timing Attacks      Standard equality checks (===)           crypto.timingSafeEqual()
SQL Injection
  • Vulnerable: db.query("SELECT * FROM users WHERE email = '" + req.body.email + "'")
  • Defended: Use parameterized inputs where the SQL engine compiles the query template before binding values:

JavaScript

// Parameterized query using pg
await pool.query('SELECT id, name FROM users WHERE email = $1', [req.body.email]);
NoSQL Injection (MongoDB)
  • Vulnerable: Sending { "username": "admin", "password": { "$ne": null } } bypasses standard equality logic.
  • Defended: Enforce rigid input validation using Zod schemas and sanitize raw query keys:

Bash

npm install express-mongo-sanitize

JavaScript

import mongoSanitize from 'express-mongo-sanitize';

// Strips out any keys starting with '$' or containing '.'
app.use(mongoSanitize());
Cross-Site Scripting (XSS)

Never trust user inputs rendered directly back into HTML interfaces. Clean output using sanitizers:

Bash

npm install sanitize-html

JavaScript

import sanitizeHtml from 'sanitize-html';

const cleanBody = sanitizeHtml(req.body.untrustedContent, {
  allowedTags: ['b', 'i', 'em', 'strong', 'a'],
  allowedAttributes: { a: ['href'] }
});
Cross-Site Request Forgery (CSRF)

If using cookie-based auth, a rogue site can force an authenticated browser to fire requests to your backend (e.g., via an image tag or hidden form).

  • Use SameSite=Lax or SameSite=Strict cookies.
  • For legacy browsers or cross-origin setups, use the Double Submit Cookie pattern or custom headers (x-requested-with or custom API tokens). Since modern SPA/API architectures rely heavily on the Authorization: Bearer <token> header, CSRF is naturally mitigated because browsers do not automatically attach headers cross-origin without prior CORS approval.
Timing Attacks

Using standard string equality operators (=== or ==) checks character by character and exits on the first mismatch. Attackers can measure response time variations to infer secret tokens or signatures byte-by-byte.

JavaScript

import crypto from 'node:crypto';

export function timingSafeCheck(userInput, actualSecret) {
  const userBuffer = Buffer.from(userInput);
  const secretBuffer = Buffer.from(actualSecret);

  // If byte lengths do not match, comparison fails
  if (userBuffer.length !== secretBuffer.length) {
    return false;
  }

  // Executes in constant time regardless of where mismatches occur
  return crypto.timingSafeEqual(userBuffer, secretBuffer);
}

3. Rate-Limiting & DDoS Mitigation

Defend backend services against volumetric floods, denial-of-service, and brute-force credential stuffing by placing a rate-limiting tier in front of endpoints.

Bash

npm install express-rate-limit rate-limit-redis

JavaScript

// middleware/rateLimiter.js
import { rateLimit } from 'express-rate-limit';
import { RedisStore } from 'rate-limit-redis';
import { redis } from '../config/redis.js';

// Global API Rate Limiter
export const apiLimiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  limit: 100,               // Limit each IP to 100 requests per window
  standardHeaders: 'draft-7', // Return standard RateLimit-* headers
  legacyHeaders: false,
  store: new RedisStore({
    sendCommand: (...args) => redis.call(...args)
  }),
  message: {
    status: 429,
    message: 'Too many requests from this IP, please try again after 15 minutes'
  }
});

// Strict Limiter for Sensitive Endpoints (Login, Password Reset)
export const authLimiter = rateLimit({
  windowMs: 60 * 60 * 1000, // 1 hour
  limit: 5,                 // 5 failed attempts per hour
  skipSuccessfulRequests: true, // Only count 4xx/5xx failures
  store: new RedisStore({
    sendCommand: (...args) => redis.call(...args),
    prefix: 'rl:auth:'
  }),
  message: {
    status: 429,
    message: 'Too many failed login attempts. Account temporarily locked for 1 hour.'
  }
});

JavaScript

// Mount limiters across the Express stack
app.use('/api/', apiLimiter);
app.use('/api/v1/auth/login', authLimiter);

Module 6: Database Integration

NoSQL: MongoDB & Mongoose

MongoDB is a document-oriented database that persists data in BSON format (Binary JSON). Unlike relational rows with fixed columns, MongoDB documents are dynamic objects capable of housing nested subdocuments and arrays.

Relational Database               MongoDB Equivalent
โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
Database                   โ”€โ”€โ”€▶   Database
Table                      โ”€โ”€โ”€▶   Collection
Row                        โ”€โ”€โ”€▶   Document (BSON)
Column                     โ”€โ”€โ”€▶   Field
Primary Key (id)           โ”€โ”€โ”€▶   _id (ObjectId: 12-byte hex)
JOIN                       โ”€โ”€โ”€▶   $lookup / Mongoose populate

MongoDB Document Structures & Indexing

  1. Embedding vs. Referencing:
    • Embedding (Denormalization): Embed subdocuments when data is accessed together, has a 1-to-few relationship, and rarely grows indefinitely (e.g., street address on a user profile).
    • Referencing (Normalization): Store an array of ObjectId references when data has 1-to-many or many-to-many relationships, is updated independently, or risks breaching the 16MB document size limit (e.g., an author with 100,000 blog comments).
  2. Indexing: Without indexes, MongoDB executes a Collection Scan (COLLSCAN), checking every document sequentially. An index creates a sorted B-Tree data structure enabling an Index Scan (IXSCAN).
    • Single Field Index: Efficient for direct equality or range lookups ({ email: 1 }).
    • Compound Index: Orders data across multiple keys ({ status: 1, createdAt: -1 }). Equality, Sort, Range (ESR rule) dictates property ordering.
    • Unique Index: Prevents duplicate keys at the storage engine level ({ sparse: true, unique: true }).
    • TTL Index: Automatically deletes documents after an elapsed timeframe (e.g., password reset tokens).

Object Data Modeling (ODM) with Mongoose

Mongoose enforces type casting, validation, query building, and lifecycle hooks on top of the MongoDB native driver.

Schemas, Models & Native Validations

JavaScript

import mongoose from 'mongoose';

const postSchema = new mongoose.Schema(
  {
    title: {
      type: String,
      required: [true, 'Post title is required'],
      trim: true,
      minlength: [5, 'Title must be at least 5 characters'],
      maxlength: [120, 'Title cannot exceed 120 characters']
    },
    slug: {
      type: String,
      unique: true,
      lowercase: true
    },
    content: {
      type: String,
      required: true
    },
    author: {
      type: mongoose.Schema.Types.ObjectId,
      ref: 'User', // Reference to the User model
      required: true,
      index: true
    },
    tags: [{ type: String, trim: true }],
    isPublished: {
      type: Boolean,
      default: false
    }
  },
  {
    timestamps: true, // Auto-generates createdAt and updatedAt
    toJSON: { virtuals: true },
    toObject: { virtuals: true }
  }
);

// Compound Index: Optimizes queries fetching published posts sorted by date
postSchema.index({ isPublished: 1, createdAt: -1 });

export const Post = mongoose.model('Post', postSchema);
Middleware (Hooks)

Mongoose hooks intercept execution at the document or query level.

  • Document Middleware: this refers to the document instance. Supported actions: init, validate, save, deleteOne.
  • Query Middleware: this refers to the Query object, not the document. Supported actions: find, findOne, findOneAndUpdate, updateMany.

JavaScript

// Pre-save hook: Generate URL slug before document persistence
postSchema.pre('save', function (next) {
  if (this.isModified('title')) {
    this.slug = this.title
      .toLowerCase()
      .replace(/[^\w ]+/g, '')
      .replace(/ +/g, '-');
  }
  next();
});

// Query Middleware: Automatically filter out soft-deleted items
postSchema.pre(/^find/, function (next) {
  // 'this' is the active query object
  this.where({ isDeleted: { $ne: true } });
  next();
});
Population

Population replaces specified paths in a document with real documents from another collection (similar to an application-level foreign key join).

JavaScript

// Retrieve post and populate author profile, omitting sensitive credentials
const post = await Post.findById(postId)
  .populate('author', 'name email avatar')
  .lean(); // .lean() skips hydrating Mongoose Documents, returning raw plain JS objects (4-5x faster)

Relational Databases: PostgreSQL & MySQL

Relational Database Management Systems (RDBMS) organize data into structured, tabular schemas governed by ACID transactions and referential integrity constraints.

       Table: users                           Table: orders
โ”Œโ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”   โ”Œโ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ id โ”‚ email (UQ)    โ”‚ name         โ”‚   โ”‚ id โ”‚ user_id โ”‚ total  โ”‚ status     โ”‚
โ”œโ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค   โ”œโ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ 1  โ”‚ alex@work.com โ”‚ Alex Rivera  โ”‚◀โ”€โ”€โ”ผโ”€ 1 โ”‚    1    โ”‚ 129.50 โ”‚ DELIVERED  โ”‚
โ”‚ 2  โ”‚ sam@dev.io    โ”‚ Sam Taylor   โ”‚   โ”‚ 2  โ”‚    1    โ”‚  45.00 โ”‚ PROCESSING โ”‚
โ””โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜   โ””โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                                        Foreign Key: orders.user_id โ”€โ”€▶ users.id

Relational Schema Design & Foreign Keys

  • Normalization:
    • 1NF: Atomicity of data (no arrays as scalar values).
    • 2NF: Elimination of partial dependency (all non-key attributes rely on the whole primary key).
    • 3NF: Elimination of transitive dependency (non-key attributes depend only on the primary key).
  • Foreign Keys (FK): Maintain relational integrity across tables.
    • ON DELETE CASCADE: Deleting a parent row automatically purges associated children.
    • ON DELETE RESTRICT / NO ACTION: Rejects parent deletion if child references exist.
    • ON DELETE SET NULL: Sets child FK columns to NULL on parent removal.

Transactions & ACID Guarantees

Transactions guarantee that multiple SQL operations succeed or fail as an indivisible unit:

  • Atomicity: All operations succeed, or the entire batch rolls back.
  • Consistency: Constraints (uniques, checks, FKs) are verified before committing.
  • Isolation: Uncommitted state changes remain invisible to concurrent operations (controlled by isolation levels: Read Uncommitted, Read Committed, Repeatable Read, Serializable).
  • Durability: Committed transactions persist through sudden power failures or crashes (Write-Ahead Logging / WAL).

Modern ORMs & Query Builders

While TypeORM uses traditional OOP decorators with Active Record / Data Mapper patterns, modern Node.js environments favor Prisma (type-safe, generated client) and Drizzle ORM (zero-overhead, SQL-like query builder).

Option 1: Prisma ORM

Prisma uses a custom declarative schema file (schema.prisma), which generates an end-to-end type-safe TypeScript/JavaScript client.

Code snippet

// prisma/schema.prisma
datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

generator client {
  provider = "prisma-client-js"
}

enum OrderStatus {
  PENDING
  PAID
  CANCELLED
}

model User {
  id        Int      @id @default(autoincrement())
  email     String   @unique
  name      String?
  orders    Order[]
  createdAt DateTime @default(now())

  @@map("users")
}

model Order {
  id        Int         @id @default(autoincrement())
  total     Decimal     @db.Decimal(10, 2)
  status    OrderStatus @default(PENDING)
  userId    Int
  user      User        @relation(fields: [userId], references: [id], onDelete: Cascade)
  createdAt DateTime    @default(now())

  @@index([userId])
  @@map("orders")
}

JavaScript

// Querying & ACID Transactions via Prisma Client
import { PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();

// Interactive Transaction: Transfer Funds
async function transferFunds(fromUserId, toUserId, amount) {
  return await prisma.$transaction(async (tx) => {
    // 1. Decrement sender balance
    const sender = await tx.account.update({
      where: { userId: fromUserId },
      data: { balance: { decrement: amount } }
    });

    if (sender.balance < 0) {
      throw new Error('Insufficient funds. Transaction rolled back.');
    }

    // 2. Increment receiver balance
    const receiver = await tx.account.update({
      where: { userId: toUserId },
      data: { balance: { increment: amount } }
    });

    return { sender, receiver };
  });
}
Option 2: Drizzle ORM

Drizzle executes close to pure SQL with zero abstraction overhead. Schemas are defined in native JavaScript/TypeScript.

JavaScript

// schema.js
import { pgTable, serial, text, integer, timestamp, numeric, pgEnum } from 'drizzle-orm/pg-core';
import { relations } from 'drizzle-orm';

export const statusEnum = pgEnum('status', ['PENDING', 'PAID', 'CANCELLED']);

export const users = pgTable('users', {
  id: serial('id').primaryKey(),
  email: text('email').notNull().unique(),
  name: text('name')
});

export const orders = pgTable('orders', {
  id: serial('id').primaryKey(),
  total: numeric('total', { precision: 10, scale: 2 }).notNull(),
  status: statusEnum('status').default('PENDING'),
  userId: integer('user_id').references(() => users.id, { onDelete: 'cascade' }),
  createdAt: timestamp('created_at').defaultNow()
});

// Relational Definitions
export const usersRelations = relations(users, ({ many }) => ({
  orders: many(orders)
}));

JavaScript

// Queries with Drizzle
import { db } from './db.js';
import { users, orders } from './schema.js';
import { eq, desc } from 'drizzle-orm';

// Performant Left Join
const userWithOrders = await db
  .select({
    userId: users.id,
    userEmail: users.email,
    orderId: orders.id,
    orderTotal: orders.total
  })
  .from(users)
  .leftJoin(orders, eq(users.id, orders.userId))
  .where(eq(users.id, 1))
  .orderBy(desc(orders.createdAt));

In-Memory Caching: Redis

Redis (Remote Dictionary Server) is an in-memory, single-threaded, key-value data structure store used to lower database load and maintain ephemeral state.

Client โ”€โ”€▶ Express โ”€โ”€▶ Cache Hit (1-2ms)  โ”€โ”€▶ Return Cached Response
                 โ”‚
                 โ””โ”€โ”€โ”€▶ Cache Miss โ”€โ”€▶ DB Query (50-200ms) โ”€โ”€▶ Store in Redis (TTL)

Bash

npm install ioredis

Redis Client Setup

JavaScript

import Redis from 'ioredis';

export const redis = new Redis({
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: Number(process.env.REDIS_PORT) || 6379,
  password: process.env.REDIS_PASSWORD || undefined,
  retryStrategy(times) {
    const delay = Math.min(times * 50, 2000);
    return delay;
  }
});

redis.on('connect', () => console.log('Connected to Redis'));
redis.on('error', (err) => console.error('Redis connection error:', err));

1. Key-Value Caching (Cache-Aside Pattern)

The Cache-Aside strategy queries the cache first. If a key is absent, it queries the persistent database, writes the result to Redis with a Time-To-Live (TTL) expiration, and returns the response.

JavaScript

export async function getProductById(productId) {
  const cacheKey = `cache:products:${productId}`;

  // 1. Check Redis Cache
  const cachedData = await redis.get(cacheKey);
  if (cachedData) {
    return JSON.parse(cachedData); // Cache Hit
  }

  // 2. Cache Miss: Query Database (Postgres / Mongo)
  const product = await db.product.findUnique({ where: { id: productId } });
  if (!product) return null;

  // 3. Populate Redis with a 1-hour expiration (3600 seconds)
  // EX sets TTL in seconds; prevents stale data from persisting indefinitely
  await redis.set(cacheKey, JSON.stringify(product), 'EX', 3600);

  return product;
}

// Cache Invalidation: Clear cache immediately on data mutation
export async function updateProduct(productId, updates) {
  const updatedProduct = await db.product.update({
    where: { id: productId },
    data: updates
  });

  // Invalidate stale cache
  await redis.del(`cache:products:${productId}`);
  return updatedProduct;
}

2. Session Storage

Centralizing user sessions inside Redis decouples state from specific Node.js instances, allowing the backend application to scale horizontally across multi-core clusters or multiple servers without losing session state.

JavaScript

import session from 'express-session';
import { RedisStore } from 'connect-redis';

// Configure session middleware backed by Redis store
export function setupSessionMiddleware(app) {
  const store = new RedisStore({
    client: redis,
    prefix: 'sess:' // Redis keys will look like "sess:session_id"
  });

  app.use(
    session({
      store,
      secret: process.env.SESSION_SECRET || 'insecure-secret-key',
      resave: false,
      saveUninitialized: false,
      cookie: {
        secure: process.env.NODE_ENV === 'production', // Transmit over HTTPS only
        httpOnly: true,                                // Block client-side XSS access
        maxAge: 1000 * 60 * 60 * 24                   // 24 hours
      }
    })
  );
}

3. Rate Limiting (Sliding Window Algorithm)

Rate limiters protect API resources against brute-force attacks and abuse. While basic rate limiters use fixed buckets (which are vulnerable to traffic spikes at window boundaries), Redis Sorted Sets (ZSET) enable an accurate, high-performance sliding-window counter.

Sorted Set: ratelimit:user_42
Score / Value: [ 1773082910000, 1773082910000 ] (Timestamps in ms)
Operation:
1. ZREMRANGEBYSCORE: Remove elements older than (now - windowSize)
2. ZADD: Add current timestamp
3. ZCARD: Count total items in the remaining window
4. If count > limit โ”€โ”€▶ Reject (429)

JavaScript

// Express Middleware: Sliding Window Rate Limiter via Redis Sorted Sets
export function rateLimiter({ windowMs, maxRequests }) {
  return async (req, res, next) => {
    const ip = req.ip || req.headers['x-forwarded-for'] || 'unknown';
    const key = `ratelimit:${ip}`;
    const now = Date.now();
    const windowStart = now - windowMs;

    try {
      // Execute operations atomically using a Redis pipeline
      const pipeline = redis.pipeline();

      // 1. Purge requests that fall outside the active sliding window
      pipeline.zremrangebyscore(key, 0, windowStart);

      // 2. Add current request timestamp
      pipeline.zadd(key, now, `${now}:${Math.random()}`);

      // 3. Retrieve total count of requests within the active window
      pipeline.zcard(key);

      // 4. Set expiry to auto-clean keys after window expires
      pipeline.pexpire(key, windowMs);

      const results = await pipeline.exec();
      // results[2] contains output of zcard: [err, count]
      const totalRequests = results[2][1];

      res.setHeader('X-RateLimit-Limit', maxRequests);
      res.setHeader('X-RateLimit-Remaining', Math.max(0, maxRequests - totalRequests));

      if (totalRequests > maxRequests) {
        return res.status(429).json({
          status: 'fail',
          message: 'Too many requests. Please slow down.'
        });
      }

      next();
    } catch (err) {
      console.error('Rate limiter failure:', err);
      // Fail open: Allow request through if Redis fails to prevent taking down the API
      next();
    }
  };
}

Database Architecture Decision Matrix

DimensionMongoDBPostgreSQL / MySQLRedis
Primary ParadigmDocument Store (NoSQL)Relational (RDBMS)In-Memory Key-Value Store
Data SchemaDynamic / PolymorphicRigid / Normal FormPrimitives, Hashes, Sets, Lists
Ideal Use CasesCatalogs, event streams, content managementFinancial ledgers, multi-table orders, analyticsCaching, session store, leaderboards, locks
Scaling ModelNative sharding (horizontal)Read replicas + vertical sizingCluster sharding, Sentinel replication
Query MechanismAggregation pipelines, MongooseStructured Query Language (SQL)Atomic commands (GET, ZADD, HSET)
LatencyMedium (5-25ms)Medium (5-30ms)Sub-millisecond (<1-2ms)

Module 5: Web Frameworks & RESTful APIs (Express.js)

Express Basics

Express is an unopinionated, minimalist routing and middleware framework that abstracts raw node:http servers into structured request pipelines.

Incoming Request โ”€โ”€▶ [ express.json() ] โ”€โ”€▶ [ Custom Middleware ] โ”€โ”€▶ [ Route Handler ] โ”€โ”€▶ Response

Application Setup & Request/Response Cycle

An Express application manages an internal stack of handler functions. Every incoming HTTP request flows sequentially through this stack until a terminal handler writes headers and calls res.send(), res.json(), or res.end().

JavaScript

import express from 'express';

const app = express();

// Body parsers: transforms raw incoming byte streams into JavaScript objects
app.use(express.json({ limit: '10kb' })); // Parse application/json
app.use(express.urlencoded({ extended: true })); // Parse application/x-www-form-urlencoded

// Basic Route Definition
app.get('/health', (req, res) => {
  res.status(200).json({ status: 'ok', timestamp: new Date().toISOString() });
});

app.listen(3000, () => {
  console.log('Server listening on port 3000');
});

Parsing URL Parameters, Query Strings, and Bodies

  • Route Parameters (req.params): Named URL segments used to identify a specific resource.
  • Query Strings (req.query): Key-value pairs appended after ? in the URL, used for filtering, pagination, and sorting.
  • Request Body (req.body): Payload submitted in POST/PUT/PATCH requests (populated by body-parsing middleware).
  • Headers (req.headers): Access headers like authorization or content-type.

JavaScript

// GET /api/v1/products/prod_99?sort=desc&limit=5
app.get('/api/v1/products/:id', (req, res) => {
  const { id } = req.params;            // 'prod_99'
  const { sort, limit = '10' } = req.query; // sort: 'desc', limit: '5'

  res.status(200).json({ productId: id, sort, limit: parseInt(limit, 10) });
});

// POST /api/v1/products
app.post('/api/v1/products', (req, res) => {
  const { name, price } = req.body; // Extracted from parsed JSON body
  res.status(201).json({ created: { name, price } });
});

Middleware Architecture

A middleware function has access to the request object (req), the response object (res), and the next function in the application’s request-response cycle.

JavaScript

function middleware(req, res, next) {
  // 1. Execute custom logic
  // 2. Mutate req or res objects if needed
  // 3. Pass control to the next middleware:
  next();
  // ...or terminate the request immediately:
  // res.status(403).json({ error: 'Forbidden' });
}

The 5 Types of Express Middleware

  1. Application-Level: Bound directly to an instance of app using app.use() or app.METHOD(). Runs on every matching path.
  2. Router-Level: Bound to an instance of express.Router(). Confines middleware execution to a specific domain or resource subset.
  3. Built-in: Bundled with Express: express.json(), express.urlencoded(), express.static('public').
  4. Third-Party: Maintained in the npm ecosystem:
    • cors: Configures Cross-Origin Resource Sharing headers.
    • morgan: HTTP request logger.
    • helmet: Sets secure HTTP response headers.

JavaScript

import cors from 'cors';
import morgan from 'morgan';

// Application-level third-party middleware
app.use(cors({ origin: 'https://trusted-client.com', methods: ['GET', 'POST'] }));
app.use(morgan('combined'));
  1. Error-Handling Middleware: Defined with strictly four arguments: (err, req, res, next). Express identifies this signature via fn.length === 4.

JavaScript

// Centralized Error-Handling Middleware (Must be declared AFTER all routes)
app.use((err, req, res, next) => {
  const statusCode = err.statusCode || 500;
  const message = err.isOperational ? err.message : 'Internal Server Error';

  console.error(`[Error] ${req.method} ${req.url}:`, err);

  res.status(statusCode).json({
    status: 'error',
    message,
    ...(process.env.NODE_ENV === 'development' && { stack: err.stack }),
  });
});

Router-Level Isolation

JavaScript

// routes/users.routes.js
import { Router } from 'express';

const router = Router();

// Router-specific middleware: applies only to routes mounted on this router
router.use((req, res, next) => {
  console.log(`User Router access: ${req.ip}`);
  next();
});

router.get('/', (req, res) => res.json({ users: [] }));
router.get('/:id', (req, res) => res.json({ user: req.params.id }));

export default router;

// app.js
import userRouter from './routes/users.routes.js';
app.use('/api/v1/users', userRouter);

REST API Design Principles

Representational State Transfer (REST) structures APIs around resources (nouns, not verbs) manipulated using standard HTTP protocols.

HTTP Verbs & Standard Status Codes

VerbSemanticsSuccess CodeIdempotentSafe
GETRead resource(s)200 OKYesYes
POSTCreate resource or execute arbitrary action201 CreatedNoNo
PUTReplace a resource completely200 OK / 204 No ContentYesNo
PATCHApply partial modifications to a resource200 OKNo (in spec)No
DELETERemove a resource200 OK / 204 No ContentYesNo
  • Safe: Does not modify server state (read-only).
  • Idempotent: Making multiple identical requests results in the same server state as a single request. DELETE /users/1 is idempotent (the record remains gone whether called once or five times), whereas POST /users creates a new record on each invocation.

Common HTTP Status Codes

  • 2xx Success: 200 OK, 201 Created, 204 No Content
  • 3xx Redirection: 301 Moved Permanently, 304 Not Modified (caching)
  • 4xx Client Error: 400 Bad Request, 401 Unauthorized (no/invalid auth token), 403 Forbidden (authenticated, but insufficient rights), 404 Not Found, 409 Conflict (e.g., duplicate unique field), 422 Unprocessable Entity (validation errors), 429 Too Many Requests
  • 5xx Server Error: 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable

Resource URI Conventions & Versioning

  • Use plural nouns: /api/v1/orders (not /api/v1/getAllOrders).
  • Model relationships hierarchically: /api/v1/users/42/orders.
  • Maintain backwards compatibility through URI versioning: /api/v1/... vs. /api/v2/....

Layered Architecture (Controller-Service-Repository)

Separating concerns isolates HTTP protocol details from core business logic and persistence layers, simplifying automated testing and maintenance.

Request โ”€โ”€▶ Controller (HTTP/Parsing) โ”€โ”€▶ Service (Business Logic) โ”€โ”€▶ Repository (Database)
src/
โ”œโ”€โ”€ controllers/    # Parses req, validates, delegates, formats res
โ”‚   โ””โ”€โ”€ user.controller.js
โ”œโ”€โ”€ services/       # Domain business rules, transactions, orchestration
โ”‚   โ””โ”€โ”€ user.service.js
โ”œโ”€โ”€ repositories/   # Direct DB operations (Prisma, Mongoose, SQL)
โ”‚   โ””โ”€โ”€ user.repository.js
โ”œโ”€โ”€ routes/         # Express Router mapping paths to controllers
โ”‚   โ””โ”€โ”€ user.routes.js
โ””โ”€โ”€ errors/         # Custom operational error classes
    โ””โ”€โ”€ AppError.js

1. Custom Error Definition

JavaScript

// errors/AppError.js
export class AppError extends Error {
  constructor(message, statusCode) {
    super(message);
    this.statusCode = statusCode;
    this.isOperational = true;
    Error.captureStackTrace(this, this.constructor);
  }
}

2. Repository Layer (Data Access Only)

JavaScript

// repositories/user.repository.js
const mockDb = new Map();

export const userRepository = {
  async findByEmail(email) {
    for (const user of mockDb.values()) {
      if (user.email === email) return user;
    }
    return null;
  },

  async create(userData) {
    const id = `usr_${Date.now()}`;
    const record = { id, ...userData, createdAt: new Date() };
    mockDb.set(id, record);
    return record;
  }
};

3. Service Layer (Business Logic Only)

JavaScript

// services/user.service.js
import { userRepository } from '../repositories/user.repository.js';
import { AppError } from '../errors/AppError.js';

export const userService = {
  async registerUser({ email, name, password }) {
    const existing = await userRepository.findByEmail(email);
    if (existing) {
      throw new AppError('A user with this email already exists', 409);
    }

    // Business rule: scrub or transform before persisting
    const newUser = await userRepository.create({ email, name, password });
    
    // Return sanitized domain model
    const { password: _, ...sanitized } = newUser;
    return sanitized;
  }
};

4. Controller Layer (HTTP Protocol Adaptation)

JavaScript

// controllers/user.controller.js
import { userService } from '../services/user.service.js';

export const userController = {
  async register(req, res, next) {
    try {
      const user = await userService.registerUser(req.body);
      res.status(201).json({ status: 'success', data: { user } });
    } catch (err) {
      next(err); // Hands errors off to centralized error middleware
    }
  }
};

Data Validation & Sanitization with Zod

Never trust client payloads. Validate schema contracts before incoming data reaches services or database layers.

Bash

npm install zod

Validation Middleware Factory

JavaScript

// middleware/validate.js
import { ZodError } from 'zod';

export const validate = (schema) => async (req, res, next) => {
  try {
    // Validates and replaces req properties with cleaned, parsed data
    const parsed = await schema.parseAsync({
      body: req.body,
      query: req.query,
      params: req.params,
    });

    req.body = parsed.body ?? req.body;
    req.query = parsed.query ?? req.query;
    req.params = parsed.params ?? req.params;
    next();
  } catch (err) {
    if (err instanceof ZodError) {
      return res.status(422).json({
        status: 'fail',
        errors: err.errors.map((e) => ({
          field: e.path.join('.'),
          message: e.message,
        })),
      });
    }
    next(err);
  }
};

Defining Schemas and Binding Routes

JavaScript

// schemas/user.schema.js
import { z } from 'zod';

export const registerUserSchema = z.object({
  body: z.object({
    name: z.string().trim().min(2, 'Name must be at least 2 characters'),
    email: z.string().trim().email('Invalid email address').toLowerCase(),
    password: z.string().min(8, 'Password must be at least 8 characters'),
    age: z.coerce.number().int().positive().optional(),
  }),
});

// routes/user.routes.js
import { Router } from 'express';
import { userController } from '../controllers/user.controller.js';
import { validate } from '../middleware/validate.js';
import { registerUserSchema } from '../schemas/user.schema.js';

const router = Router();
router.post('/register', validate(registerUserSchema), userController.register);

export default router;

File Handling: Uploads with Multer

Browsers and clients upload files via multipart/form-data. Because express.json() handles only JSON, processing multi-part binary streams requires dedicated middleware like multer.

Bash

npm install multer

Storage Engines: Disk vs. Memory

  • memoryStorage(): Stores files as raw Buffer instances in RAM (req.file.buffer). Recommended when streaming directly to cloud object storage (AWS S3, Google Cloud Storage, Cloudinary). Large uploads can cause memory spikes.
  • diskStorage(): Streams files directly to the local file system. Keeps memory consumption low, but requires managing temporary files and disk cleanups.

Production Multer Configuration

JavaScript

// config/multer.js
import multer from 'multer';
import path from 'node:path';
import crypto from 'node:crypto';
import { AppError } from '../errors/AppError.js';

// Configure storage destination and collision-resistant file names
const storage = multer.diskStorage({
  destination: (req, file, cb) => {
    cb(null, 'uploads/');
  },
  filename: (req, file, cb) => {
    const uniqueSuffix = crypto.randomBytes(16).toString('hex');
    const ext = path.extname(file.originalname).toLowerCase();
    cb(null, `${file.fieldname}-${Date.now()}-${uniqueSuffix}${ext}`);
  },
});

// Strict MIME type validation
const fileFilter = (req, file, cb) => {
  const allowedMimeTypes = ['image/jpeg', 'image/png', 'image/webp'];

  if (allowedMimeTypes.includes(file.mimetype)) {
    cb(null, true);
  } else {
    cb(new AppError('Invalid file format. Only JPEG, PNG, and WebP are allowed.', 400), false);
  }
};

export const upload = multer({
  storage,
  fileFilter,
  limits: {
    fileSize: 5 * 1024 * 1024, // 5 Megabytes maximum
    files: 1,                  // Restrict to a single file per request
  },
});

Route Implementation

JavaScript

// routes/upload.routes.js
import { Router } from 'express';
import { upload } from '../config/multer.js';

const router = Router();

// Single file upload on field 'avatar'
router.post('/avatar', upload.single('avatar'), (req, res) => {
  if (!req.file) {
    return res.status(400).json({ error: 'Please provide a file' });
  }

  // req.file contains metadata
  const { path: diskPath, filename, size, mimetype } = req.file;

  res.status(200).json({
    status: 'success',
    file: { filename, path: diskPath, size, mimetype },
  });
});

// Multiple files upload on field 'documents' (up to 5)
router.post('/documents', upload.array('documents', 5), (req, res) => {
  // req.files is an array of uploaded file objects
  res.status(200).json({
    uploadedCount: req.files.length,
    files: req.files.map((f) => f.filename),
  });
});

export default router;

Module 4: Core Node.js Built-in Modules

File System (node:fs & node:fs/promises)

Node.js provides three APIs to interact with the file system: Synchronous (blocks the event loop), Callback-based (asynchronous, legacy), and Promise-based (asynchronous, modern standard via node:fs/promises).

+------------------+-----------------------+-----------------------------+
| API Style        | Example Method        | Impact on Event Loop        |
+------------------+-----------------------+-----------------------------+
| Synchronous      | fs.readFileSync()     | Freezes Call Stack entirely |
| Callback-based   | fs.readFile()         | Non-blocking (libuv thread) |
| Promise-based    | fsPromises.readFile() | Non-blocking (async/await)  |
+------------------+-----------------------+-----------------------------+

Production Rule: Never use synchronous file operations (*Sync) inside request handlers or loops. Reserve them strictly for initial bootstrapping (e.g., loading a static TLS certificate or local config file before the server starts listening).

JavaScript

import fs from 'node:fs/promises';
import path from 'node:path';

const filePath = path.join(process.cwd(), 'data', 'app.log');

async function handleFileLifecycle() {
  try {
    // 1. Create / Overwrite (Writing)
    await fs.writeFile(filePath, 'Initial system state: OK\n', { encoding: 'utf8' });

    // 2. Append data
    await fs.appendFile(filePath, 'Event: Worker process started\n', 'utf8');

    // 3. Read data
    // Without encoding, it returns a raw Buffer; with encoding, it returns a string
    const contents = await fs.readFile(filePath, { encoding: 'utf8' });
    console.log('File Content:\n', contents);

    // 4. Update / Partial overwrite using a FileHandle
    // Fine-grained control without reading the whole file into RAM
    const fileHandle = await fs.open(filePath, 'r+');
    try {
      const updateBuffer = Buffer.from('CRITICAL');
      // Write buffer at byte offset 0
      await fileHandle.write(updateBuffer, 0, updateBuffer.length, 0);
    } finally {
      await fileHandle.close(); // Prevent file descriptor leaks
    }

    // 5. Check existence & metadata
    const stats = await fs.stat(filePath);
    console.log(`Size: ${stats.size} bytes | Is Directory: ${stats.isDirectory()}`);

    // 6. Delete file
    await fs.unlink(filePath);
    console.log('File successfully removed');
  } catch (err) {
    if (err.code === 'ENOENT') {
      console.error('File or directory does not exist');
    } else {
      console.error('FS error:', err.message);
    }
  }
}

Path Module (node:path)

Operating systems use different segment separators: POSIX (Linux/macOS) uses /, while Windows uses \. The node:path module abstracts these differences and provides defenses against path-based vulnerabilities.

JavaScript

import path from 'node:path';

// 1. path.join vs. path.resolve
// path.join: Concatenates pieces using the platform separator and normalizes relative segments (.. and .)
const joined = path.join('/users', 'local', '..', 'bin', 'node'); 
// Linux/macOS: /users/bin/node

// path.resolve: Processes arguments from right to left until an absolute path is constructed
const resolved = path.resolve('data', 'reports', '2026.pdf');
// Output: /current/working/dir/data/reports/2026.pdf

// 2. Normalization and Extraction
const dirtyPath = '/app/uploads//images/../avatar.png';
const cleanPath = path.normalize(dirtyPath); // '/app/uploads/avatar.png'

const parsed = path.parse(cleanPath);
// {
//   root: '/',
//   dir: '/app/uploads',
//   base: 'avatar.png',
//   ext: '.png',
//   name: 'avatar'
// }

Defending Against Path Traversal Vulnerabilities

Malicious clients can pass inputs like ../../etc/passwd to extract system files. Use path.resolve combined with prefix validation:

JavaScript

import path from 'node:path';

const PUBLIC_DIRECTORY = path.resolve(process.cwd(), 'public');

function safeResolve(userInput) {
  // 1. Resolve target file to an absolute path
  const targetPath = path.resolve(PUBLIC_DIRECTORY, userInput);

  // 2. Validate that targetPath strictly starts within PUBLIC_DIRECTORY
  const relative = path.relative(PUBLIC_DIRECTORY, targetPath);
  const isSafe = !relative.startsWith('..') && !path.isAbsolute(relative);

  if (!isSafe) {
    throw new Error('Access Denied: Path Traversal Attempt Detected');
  }

  return targetPath;
}

// safeResolve('images/cat.png')  -> OK (/project/public/images/cat.png)
// safeResolve('../../../etc/shadow') -> Throws "Access Denied"

Events (node:events)

The EventEmitter class powers Node’s internal architecture (Stream, http.Server, and net.Socket all inherit from it). It implements the Observer pattern for event-driven coordination.

JavaScript

import { EventEmitter } from 'node:events';

class PaymentProcessor extends EventEmitter {
  processPayment(orderId, amount) {
    this.emit('payment:started', { orderId });

    // Simulate payment work
    setTimeout(() => {
      if (amount > 0) {
        this.emit('payment:success', { orderId, amount, transactionId: 'txn_987' });
      } else {
        // Crucial: Unhandled 'error' events crash the entire Node.js process!
        this.emit('error', new Error(`Invalid amount: ${amount} for order: ${orderId}`));
      }
    }, 100);
  }
}

const processor = new PaymentProcessor();

// Standard listener: triggers on every emission
processor.on('payment:started', ({ orderId }) => {
  console.log(`[LOG] Payment init for order ${orderId}`);
});

// Single-use listener: auto-deregisters after the first call
processor.once('payment:success', (data) => {
  console.log(`[ALERT] First successful payment processed:`, data);
});

// Mandatory error listener
processor.on('error', (err) => {
  console.error(`[CRITICAL] Processor failed: ${err.message}`);
});

Mitigating Memory Leaks with Event Listeners

Attaching listeners inside repeated function calls without removing them will retain referenced variables in memory permanently.

JavaScript

// By default, EventEmitter warns if > 10 listeners are added to a single event
processor.setMaxListeners(20); 

function onOrderComplete(data) {
  console.log('Sending metrics...');
}

// Register
processor.on('order:complete', onOrderComplete);

// Cleanup: removeListener / off requires a reference to the exact named function
processor.off('order:complete', onOrderComplete);

// Alternatively: Node.js 16+ supports auto-cleaning via AbortController
const ac = new AbortController();
processor.on('data', (chunk) => console.log(chunk), { signal: ac.signal });

// Instantly unsubscribes without holding function references:
ac.abort();

Buffers & Streams

JavaScript was originally designed for strings, not raw binary bytes. Node.js introduced Buffers to allocate fixed-size memory outside the V8 heap and Streams to handle datasets larger than system memory.

Binary Data Handling with Buffer

Buffers represent chunks of raw binary data as sequences of octets (integers from 0 to 255).

JavaScript

// 1. Allocation (allocates zero-filled safe memory)
const bufZero = Buffer.alloc(10); // 10 zeroed bytes

// Unsafe allocation: faster, but retains uninitialized/dirty memory (could leak old RAM data)
// const bufUnsafe = Buffer.allocUnsafe(10); 

// 2. Creation from existing data
const bufFromString = Buffer.from('Hello World', 'utf8');
console.log(bufFromString); 
// <Buffer 48 65 6c 6c 6f 20 57 6f 72 6c 64> (hex representation)

// 3. Conversions
console.log(bufFromString.toString('base64')); // SGVsbG8gV29ybGQ=
console.log(bufFromString.toString('hex'));    // 48656c6c6f20576f726c64

// 4. Mutation & Slicing
bufFromString[0] = 74; // ASCII for 'J'
console.log(bufFromString.toString('utf8')); // Jello World

The 4 Core Stream Types

1. Readable   (fs.createReadStream, req)      โ”€โ”€> Data Source (Producer)
2. Writable   (fs.createWriteStream, res)     โ”€โ”€> Data Sink (Consumer)
3. Duplex     (net.Socket)                    โ”€โ”€> Both Readable & Writable simultaneously
4. Transform  (zlib.createGzip, crypto)       โ”€โ”€> Duplex that modifies data during pass-through

Backpressure, Piping, and stream.pipeline

Backpressure occurs when a Readable stream pushes data into a Writable stream faster than the consumer can write it (e.g., fast disk reading vs. slow network client). Without backpressure handling, unwritten chunks accumulate in RAM, triggering high memory usage and crashes.

The Anti-Pattern: readable.pipe(writable)

While pipe() handles backpressure, it does not cleanly clean up file descriptors or forward errors if one stream in the chain fails, often leading to silent stream leaks.

The Modern Pattern: stream.promises.pipeline

Use pipeline from node:stream/promises. It manages backpressure, destroys all streams if any stream encounters an error, and returns a Promise.

JavaScript

import fs from 'node:fs';
import { pipeline } from 'node:stream/promises';
import { createGzip } from 'node:zlib';
import path from 'node:path';

async function compressLargeFile(source, destination) {
  try {
    // Transfers 10GB+ files using only ~64KB of RAM at any given time
    await pipeline(
      fs.createReadStream(source, { highWaterMark: 64 * 1024 }), // 64KB chunk buffer
      createGzip(),                                              // Transform stream
      fs.createWriteStream(destination)                          // Destination sink
    );
    console.log('Compression complete without RAM spikes.');
  } catch (err) {
    console.error('Pipeline failed:', err.message);
    // All underlying file descriptors and internal stream buffers are safely closed
  }
}
Chunk-by-Chunk Processing with Async Iterators

Readable streams implement the Symbol.asyncIterator contract, allowing processing of line items, JSON rows, or chunks using standard for await...of:

JavaScript

import fs from 'node:fs';

async function processLogChunks(filePath) {
  const readStream = fs.createReadStream(filePath, { encoding: 'utf8' });
  let totalBytes = 0;

  for await (const chunk of readStream) {
    totalBytes += chunk.length;
    // Inspect each individual memory chunk as it arrives
    console.log(`Received chunk of size ${chunk.length}`);
  }

  console.log(`Finished processing ${totalBytes} characters.`);
}

OS & HTTP (node:os & node:http)

The node:os Module

Used to inspect host hardware parameters, allocate thread pools, or run cluster workers.

JavaScript

import os from 'node:os';

console.log('Platform:', os.platform());         // 'linux', 'darwin', 'win32'
console.log('CPU Architecture:', os.arch());     // 'x64', 'arm64'
console.log('Available CPU Cores:', os.cpus().length);
console.log('Free RAM:', (os.freemem() / 1024 / 1024).toFixed(2), 'MB');
console.log('Total RAM:', (os.totalmem() / 1024 / 1024).toFixed(2), 'MB');
console.log('Network Interfaces:', Object.keys(os.networkInterfaces()));

Bare-Bones HTTP Server (node:http)

Under the hood, all major frameworks (Express, Fastify, NestJS) wrap node:http. In this API:

  • req is a Readable Stream (http.IncomingMessage).
  • res is a Writable Stream (http.ServerResponse).

JavaScript

import http from 'node:http';

const server = http.createServer(async (req, res) => {
  const { method, url } = req;

  // 1. Simple route matching
  if (url === '/api/health' && method === 'GET') {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ status: 'healthy', uptime: process.uptime() }));
    return;
  }

  // 2. Reading payload body from the incoming Readable Stream
  if (url === '/api/echo' && method === 'POST') {
    const buffers = [];

    // Collect binary chunks from the network socket
    for await (const chunk of req) {
      buffers.push(chunk);
    }

    const rawBody = Buffer.concat(buffers).toString('utf8');

    try {
      const parsedBody = JSON.parse(rawBody || '{}');

      // Writing response headers and payload
      res.writeHead(200, {
        'Content-Type': 'application/json',
        'X-Powered-By': 'NodeJS-Native-Core'
      });
      res.end(JSON.stringify({ received: parsedBody, timestamp: Date.now() }));
    } catch {
      res.writeHead(400, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ error: 'Malformed JSON payload' }));
    }
    return;
  }

  // 3. Fallback 404 Handler
  res.writeHead(404, { 'Content-Type': 'text/plain' });
  res.end('Resource Not Found');
});

const PORT = 3000;
server.listen(PORT, '127.0.0.1', () => {
  console.log(`HTTP Server running at http://127.0.0.1:${PORT}/`);
});

Module 3: Node Package Manager (NPM / PNPM / Yarn)

Package Management

Node.js projects rely on package managers to resolve, download, verify, and organize third-party libraries within the node_modules directory.

package.json vs. package-lock.json

FilePurposeKey ContentsVersion Control
package.jsonProject manifest describing metadata, commands, and dependency constraints.Names, scripts, loose semver ranges (dependencies, devDependencies, peerDependencies).Commit to Git.
package-lock.json (yarn.lock / pnpm-lock.yaml)State snapshot locking the exact dependency graph.Exact versions installed, integrity hashes (SHA-512), resolved URLs, transitive dependencies.Always commit to Git.
  • dependencies vs. devDependencies:
    • dependencies: Packages required for the application to function in production (e.g., express, pg, zod). Installed when running npm install --omit=dev.
    • devDependencies: Tools used solely for local development, compilation, linting, or testing (e.g., typescript, jest, eslint, nodemon).
  • Deterministic Builds (npm ci): In CI/CD pipelines and production deployments, run npm ci instead of npm install. It deletes node_modules, strictly validates against package-lock.json, bypasses version resolution, and throws an error if the lockfile and package.json are out of sync.

Semantic Versioning (SemVer)

SemVer follows the format MAJOR.MINOR.PATCH:

  • MAJOR (X.0.0): Incompatible API changes (breaking changes).
  • MINOR (0.X.0): Backwards-compatible new features.
  • PATCH (0.0.X): Backwards-compatible bug fixes.

Package managers use prefix operators to manage automatic update ranges during npm install or npm update:

JSON

{
  "dependencies": {
    "exact-pkg": "1.4.2",      // Strict match: installs exactly 1.4.2
    "patch-pkg": "~1.4.2",      // Tilde (~): Accepts PATCH updates (>= 1.4.2 < 1.5.0)
    "minor-pkg": "^1.4.2",      // Caret (^): Accepts MINOR & PATCH updates (>= 1.4.2 < 2.0.0)
    "pre-zero-pkg": "^0.2.3",   // Exception: If MAJOR is 0, ^ locks MINOR (>= 0.2.3 < 0.3.0)
    "any-pkg": "*"              // Wildcard: Installs latest version (risks breaking changes)
  }
}

Local vs. Global Dependencies

  • Local Dependencies (npm install <pkg>):
    • Stored in ./node_modules at the project root.
    • Scoped directly to the project, preventing version clashes between different projects on the same machine.
  • Global Dependencies (npm install -g <pkg>):
    • Stored in the system’s global directory (e.g., /usr/local/lib/node_modules on Linux/macOS or %AppData%\npm on Windows).
    • Exposes binaries globally across your shell (e.g., npm, pnpm, pm2).
    • Avoid installing project-specific CLI tools globally (e.g., use local typescript or eslint instead of global installations to prevent environment mismatches across team members).

Modern Alternatives: PNPM & Yarn

  • NPM: Standard default, uses a flattened node_modules layout. Can lead to “phantom dependencies” (importing packages your dependencies use that you didn’t explicitly define).
  • Yarn: Introduced lockfiles, parallel downloads, and offline caching. Modern Yarn (Berry) supports Plug’n’Play (PnP), removing node_modules entirely in favor of an optimized lookup table.
  • PNPM: Uses a content-addressable store on disk. All files are saved once in a global store and hard-linked into projects.
    • Saves gigabytes of disk space across multiple projects.
    • Uses isolated (non-flat) symlinked node_modules to eliminate phantom dependencies.

Scripts & Workflows

The scripts object in package.json allows defining custom commands, pre/post hooks, and environment configurations.

JSON

{
  "scripts": {
    "prebuild": "npm run clean",
    "build": "tsc",
    "postbuild": "node scripts/copy-assets.js",
    "start": "node dist/index.js",
    "dev": "node --watch --env-file=.env src/index.js",
    "test": "jest",
    "lint": "eslint .",
    "clean": "rm -rf dist"
  }
}

Lifecycle & Pre/Post Hooks

  • Any script prefixed with pre or post executes automatically before or after the base command. Running npm run build sequentially executes:
    1. npm run prebuild
    2. npm run build
    3. npm run postbuild
  • Default commands like npm start and npm test do not require the run keyword; custom scripts require npm run <name>.

Passing Arguments

To pass flags or options through an npm script to the underlying binary, separate them with a double hyphen (--):

Bash

# Passes --coverage down directly to Jest
npm test -- --coverage

# Passes a port flag to a local dev server
npm run dev -- --port=8080

NPX (Node Package Execute)

npx comes bundled with NPM and serves two primary purposes:

  1. Running locally installed binaries without global installation: Instead of referencing ./node_modules/.bin/eslint ., run:Bashnpx eslint .
  2. One-off execution of uninstalled packages:npx downloads the package to a temporary cache, executes the binary, and removes it immediately:Bashnpx create-next-app@latest my-app

Managing Secrets

Backend applications require sensitive credentials (database passwords, API tokens, encryption keys) that must never be hardcoded into the source code.

The .env File Conventions

  1. .env: Contains real local development credentials. Must be added to .gitignore.
  2. .env.example: A committed template displaying all required environment keys with blank or dummy values:Ini, TOML# .env.example PORT= DATABASE_URL= JWT_SECRET= NODE_ENV=

Method 1: Using dotenv (Standard Cross-Version Approach)

Install the package:

Bash

npm install dotenv

Load it as early as possible in your application lifecycle:

JavaScript

// ESM syntax
import 'dotenv/config';

// Or programmatic CJS syntax
// require('dotenv').config({ path: './custom.env' });

const port = process.env.PORT || 3000;
const dbUrl = process.env.DATABASE_URL;

if (!dbUrl) {
  throw new Error('FATAL: DATABASE_URL is not set in environment.');
}

Preloading without modifying source code (recommended for production builds):

Bash

node -r dotenv/config dist/server.js

Method 2: Native Node.js .env Loading

Node.js provides built-in .env file parsing via the --env-file CLI flag, eliminating the need for third-party packages like dotenv:

Bash

# Load default .env
node --env-file=.env server.js

# Load environment-specific file
node --env-file=.env.production server.js

Validating Environment Variables (Fail-Fast Pattern)

Relying on raw process.env leaves room for runtime bugs if a required string is missing or misspelled. Validating the environment at startup ensures the process fails fast before accepting traffic.

Example with Zod:

JavaScript

import { z } from 'zod';

const envSchema = z.object({
  NODE_ENV: z.enum(['development', 'test', 'production']).default('development'),
  PORT: z.coerce.number().default(3000),
  DATABASE_URL: z.string().url(),
  JWT_SECRET: z.string().min(32, 'JWT_SECRET must be at least 32 characters')
});

// Parse process.env; throws a detailed ZodError on failure
export const env = envSchema.parse(process.env);

Module 2: Node.js Runtime & Architecture

Internal Architecture

Node.js is not an independent language or framework; it is an open-source, cross-platform JavaScript runtime environment built on Chrome’s V8 engine and libuv.

+-------------------------------------------------------+
|                   Node.js Bindings                    |
|          (node:fs, node:http, node:crypto...)         |
+-------------------------------------------------------+
|               Node.js C++ Core & Addons               |
+---------------------------+---------------------------+
|         V8 Engine         |           libuv           |
|  - Compiles JS to Machine |  - Event Loop             |
|    Code (JIT)             |  - libuv Thread Pool      |
|  - Call Stack             |  - Epoll / Kqueue / IOCP  |
|  - Memory Heap (GC)       |  - Cross-platform Async   |
+---------------------------+---------------------------+
|          c-ares           |  OpenSSL  |  zlib  |  llhttp
+---------------------------+---------------------------+
|                 Operating System Kernel               |
+-------------------------------------------------------+

The V8 Engine

Written in C++, Googleโ€™s V8 engine compiles JavaScript directly into native machine code before executing it, using an optimizing JIT (Just-In-Time) compiler pipeline.

  • Call Stack: Executes synchronous JavaScript instructions via LIFO (Last In, First Out). Since JavaScript is single-threaded, there is only one Call Stack on the main thread.
  • Memory Heap: Allocates memory for objects, closures, and variables. Garbage Collection (GC) sweeps this area automatically (using generational garbage collection algorithms).

libuv

libuv is a C library that handles the platform abstraction layer for asynchronous I/O. It provides:

  1. The Event Loop: Orchestrates non-blocking execution across different task queues.
  2. The Worker Thread Pool: Handles operations that cannot be completed non-blockingly at the OS level.
  3. Cross-Platform System Polling: Interfaces directly with native kernel event mechanisms:
    • Linux: epoll
    • macOS / BSD: kqueue
    • Windows: IOCP (Input/Output Completion Ports)

The libuv Thread Pool

Node.js executes JavaScript on a single thread. However, low-level tasks that have no asynchronous OS system calls are delegated to the libuv thread pool.

Tasks handled by the thread pool include:

  • All File System Operations (fs): Operating systems generally do not support true non-blocking disk I/O.
  • DNS Resolution: Specifically dns.lookup() (which uses synchronous getaddrinfo(3)).
  • CPU-Heavy Cryptography: crypto.pbkdf2(), crypto.scrypt(), crypto.randomBytes().
  • Compression: zlib operations.

Thread Pool Configuration: The default thread pool size is 4. You can increase it up to 1024 by setting the environment variable before the runtime initializes:

Bash

UV_THREADPOOL_SIZE=16 node server.js

Note: Network I/O (HTTP requests, incoming TCP sockets, WebSockets) does not use the thread pool. The kernel directly notifies libuv when network packets arrive, allowing a single thread to handle tens of thousands of concurrent connections.

Non-Blocking I/O Workflow

  1. A JavaScript function calls an asynchronous API (e.g., fs.readFile() or https.get()).
  2. The Node.js C++ binding hands the request over to libuv.
  3. If it’s a network socket, libuv registers a file descriptor with the OS kernel (epoll/kqueue). If it’s disk I/O, it pushes the work to the thread pool.
  4. The JavaScript Call Stack is immediately freed; the runtime does not block.
  5. When the OS or thread pool finishes the task, libuv enqueues the associated callback into the corresponding Event Loop phase.

Event Loop Phases

The Event Loop is an infinite loop managed by libuv that processes queued callbacks. Each iteration of this loop is called a tick.

   โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”Œโ”€>โ”‚                 timers                   โ”‚
โ”‚  โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
โ”‚  โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚  โ”‚             pending callbacks            โ”‚
โ”‚  โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
โ”‚  โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚  โ”‚               idle, prepare              โ”‚
โ”‚  โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
โ”‚  โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚  โ”‚                   poll                   โ”‚ <โ”€โ”€ Incoming connections,
โ”‚  โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜     data, I/O callbacks
โ”‚  โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚  โ”‚                   check                  โ”‚ <โ”€โ”€ setImmediate()
โ”‚  โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
โ”‚  โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚  โ”‚              close callbacks             โ”‚
โ””โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
PhaseDescription & Responsibilities
1. TimersExecutes callbacks scheduled by setTimeout() and setInterval() whose threshold has expired.
2. Pending CallbacksExecutes I/O callbacks deferred from the previous loop iteration (e.g., system-level errors like ECONNREFUSED reported by TCP sockets).
3. Idle, PrepareInternal phase used strictly by libuv for internal subsystem housekeeping. JavaScript callbacks do not run here.
4. PollRetrieves new I/O events. If callbacks are queued in this phase, it runs them until the queue is empty or a system-dependent limit is reached. If the queue is empty, the loop either blocks here waiting for incoming I/O, or immediately advances to the check phase if setImmediate() callbacks are scheduled.
5. CheckDedicated phase for executing callbacks registered with setImmediate().
6. Close CallbacksHandles sudden connection drops and cleanup events (e.g., socket.on('close', ...), process.exit()).

setTimeout(fn, 0) vs setImmediate(fn)

  • When invoked inside an I/O cycle (e.g., inside an fs.readFile callback), setImmediate always executes first because the poll phase moves directly to the check phase.
  • When invoked from the global scope (outside an I/O cycle), the execution order is non-deterministic and depends on process performance and OS timer resolution:

JavaScript

const fs = require('fs');

fs.readFile(__filename, () => {
  setTimeout(() => console.log('Timer 1'), 0);
  setImmediate(() => console.log('Immediate 1'));
  // Output is always:
  // Immediate 1
  // Timer 1
});

Microtask Queue vs. Macrotask Queue

The Event Loop phases manage Macrotasks (Timers, I/O, setImmediate). However, Node.js also maintains two distinct Microtask Queues that sit outside the libuv loop.

       [ Call Stack Empties ]
                 โ”‚
                 โ–ผ
    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
    โ”‚  process.nextTick Queue   โ”‚ <โ”€โ”€ Highest Priority
    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                 โ”‚ (Drained completely)
                 โ–ผ
    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
    โ”‚    Promise Microtasks     โ”‚ <โ”€โ”€ Resolved Promises, queueMicrotask()
    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                 โ”‚ (Drained completely)
                 โ–ผ
    [ Advance to Next Event Loop Phase ]

The Priority Hierarchy

  1. process.nextTick() Queue: Evaluated immediately after the currently running operation completes on the Call Stack, regardless of which phase the Event Loop is currently in.
  2. Promise Microtask Queue: Evaluates resolved Promise chains (.then(), .catch(), .finally()) and queueMicrotask(). Runs immediately after the nextTick queue is fully drained.
  3. Macrotask Queue: Evaluates timer callbacks, poll events, or check events depending on the active Event Loop phase.

Priority Execution Verification

JavaScript

const fs = require('fs');

console.log('1. Synchronous Start');

setTimeout(() => console.log('2. setTimeout (Macrotask - Timers)'), 0);
setImmediate(() => console.log('3. setImmediate (Macrotask - Check)'));

Promise.resolve().then(() => {
  console.log('4. Promise Microtask');
});

process.nextTick(() => {
  console.log('5. process.nextTick Microtask');
});

console.log('6. Synchronous End');

// Execution Order:
// 1. Synchronous Start
// 6. Synchronous End
// 5. process.nextTick Microtask   (nextTick queue drains first)
// 4. Promise Microtask            (Promise microtask queue drains second)
// 2. setTimeout                   (Event loop enters Timers phase)
// 3. setImmediate                (Event loop reaches Check phase)

Warning: Recursively calling process.nextTick() starves the Event Loop by preventing it from ever advancing to the next phase, causing complete I/O starvation.

Global Objects & CLI Execution

Node.js provides a set of globally available objects, references, and execution primitives that do not require explicit module imports.

Key Globals

  • global: The root global namespace object (analogous to window in web browsers). Attaching properties to global makes them accessible everywhere, but this is an anti-pattern that leads to hidden dependencies and memory retention.
  • process: Provides control and diagnostic information over the running Node.js process:
    • process.env: Exposes system environment variables.
    • process.argv: Array containing command-line arguments.
    • process.cwd(): Returns the Current Working Directory from which the process was launched (different from the file location).
    • process.exit(code): Terminates the process (0 for success, non-zero for failure).
    • process.memoryUsage(): Returns an object describing heap allocation, resident set size (RSS), and external memory.

File Resolution Identifiers (CommonJS Only)

  • __dirname: The absolute directory path of the file currently being executed.
  • __filename: The absolute path including the filename of the file currently being executed.

JavaScript

// Given a file at /apps/backend/src/server.js:
console.log(__dirname);  // Output: /apps/backend/src
console.log(__filename); // Output: /apps/backend/src/server.js
console.log(process.cwd()); // If run from /apps/backend -> Output: /apps/backend

CLI Execution Primitives

Node.js includes powerful CLI flags for debugging, inspection, and memory configuration:

Bash

# Start an interactive Read-Eval-Print Loop
node

# Execute inline JavaScript without creating a file
node -e "console.log(crypto.randomUUID())"

# Start process with the V8 Inspector attached (debug via Chrome DevTools or VS Code)
node --inspect index.js

# Break immediately at the first line of code on startup
node --inspect-brk index.js

# Override V8 maximum old generation heap memory (e.g., set to 4GB)
node --max-old-space-size=4096 index.js

# Watch mode (automatically restarts on file changes - built-in since Node v18+)
node --watch server.js

Module Systems: CommonJS vs. ES Modules

Node.js supports two distinct module architectures: CommonJS (CJS) and ECMAScript Modules (ESM).

+---------------------------+-----------------------------------------------+
| CommonJS (CJS)            | ES Modules (ESM)                              |
+---------------------------+-----------------------------------------------+
| Default legacy system     | Modern standard (ECMAScript specification)    |
| Synchronous loading       | Asynchronous parsing and loading              |
| Dynamic requires allowed  | Static imports by default                     |
| Evaluated at runtime      | Parsed before evaluation (Static Analysis)    |
| Node.js specific          | Cross-platform (Browser + Node.js)            |
+---------------------------+-----------------------------------------------+

Syntax Comparison

CommonJS (.cjs or default in standard package.json)

JavaScript

// Exporting: math.js
const add = (a, b) => a + b;
const subtract = (a, b) => a - b;

module.exports = { add, subtract };
// Alternative: exports.add = add;

// Importing: app.js
const { add, subtract } = require('./math');
console.log(add(5, 2));
ES Modules (.mjs or with "type": "module" in package.json)

JavaScript

// Exporting: math.js
export const add = (a, b) => a + b;
export const subtract = (a, b) => a - b;
export default function multiply(a, b) { return a * b; }

// Importing: app.js
import multiply, { add, subtract } from './math.js'; // Extension required
console.log(multiply(5, 2));

Deep Architectural Differences

1. Module Wrapper Function (CommonJS)

Under the hood, before executing any CJS file, Node.js wraps its content inside an IIFE (Immediately Invoked Function Expression):

JavaScript

(function(exports, require, module, __filename, __dirname) {
  // Your file code actually lives here
});

This is why variables like exports, require, module, __filename, and __dirname appear global in CJS when they are actually locally scoped parameters.

2. Absence of __dirname and __filename in ESM

In ES Modules, the wrapper function is not used, so __dirname and __filename do not exist. You reconstruct them using import.meta.url:

JavaScript

import { fileURLToPath } from 'node:url';
import { dirname } from 'node:path';

const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
3. Loading Mechanism: Synchronous vs. Asynchronous
  • CommonJS: require() is synchronous. It stops execution, reads the file from the disk, compiles it, caches it, and returns the exported object. You can dynamically call require() anywhere (inside if statements or loops).
  • ES Modules: Import references are hoisted and resolved statically before the code is evaluated. This enables tree-shaking (removing unused code). Dynamic loading in ESM is achieved using the asynchronous import() function:

JavaScript

// Dynamic import inside an ESM file
if (process.env.NODE_ENV === 'development') {
  const devTools = await import('./devTools.js');
  devTools.init();
}
4. Caching & References

Both systems cache loaded modules on the first call. However, they pass values differently:

  • CommonJS: Exports a copied value (or reference copy for objects). Modifying an exported primitive variable inside the source module after export does not update the imported value in the consumer.
  • ES Modules: Exports a live binding. If the source module changes the value of an exported variable, the importer sees the updated value immediately.