Parallelize Webpack Builds

The webpack build in one of my larger projects takes over 5 minutes to complete. I found an easy way to cut the build-time: running multiple builds in parallel!

Disclaimer: This works because my webpack config has several independent entry points. My webpack config looks like this:

import webpack from "webpack";
import glob from "glob";

const INIT_DIR = path.join(__dirname, "src", "init");
const OUT_DIR = path.join(__dirname, "dist");

const entryPoints = glob.sync(path.join(INIT_DIR, "*.js"))
  .reduce((obj, filePath) => {
    const pieces = path.parse(filePath);
    return {
      ...obj,
      [pieces.name]: path.join(INIT_DIR, pieces.name)
    };
  }, {});

export default {
  entry: entryPoints,
  output: {
    path : OUT_DIR,
    filename: "[name].min.js"
  }
};

The top-level JS entry points are contained in the folder src/init:

src/
└── init/
  ├── entry1.js
  ├── entry2.js
  └── entry3.js

The entryPoints object would be:

{
  entry1: "src/init/entry1.js",
  entry2: "src/init/entry2.js",
  entry3: "src/init/entry3.js"
}

Webpack generates a bundle out of each of these files and dumps them into the dist folder, appending min.js to each bundle name.

src/
├── init/
│   ├── entry1.js
│   ├── entry2.js
│   └── entry3.js
dist/
    ├── entry1.min.js
    ├── entry2.min.js
    └── entry3.min.js

The secret sauce

Instead of each bundle making the others wait for it to finish compiling, I split up the workload and fire off multiple webpack processes in parallel. I set the number of splits to 4. Increasing it further didn't appear to improve build times much.

import chalk from "chalk";
import path from "path";
import getopt from "node-getopt";
import { exec } from "child_process";
import webpack from "webpack";
import config from "./webpack.config.prod";

const args = getopt.create([
    [ "", "procid=ARG", "Proc ID for parallel builds" ],
    [ "", "help",  "display this help" ]
  ])
  .bindHelp()
  .parseSystem();

let procid = args.options.procid;
const MAX_PROC = 4;

const entryPoints = config.entry;
const bundleNames = Object.keys(entryPoints);
const QUEUE_SIZE = Math.ceil(bundleNames.length / MAX_PROC);

if(!procid || isNaN(parseInt(procid))) {
  console.log(chalk.cyan("Starting parallel webpack build ..."));
  let children = 0;
  for(let i=0; i<bundleNames.length; i+=QUEUE_SIZE) {
    const cmd = `node_modules/.bin/babel-node build.js --procid=${i}`;
    const child = exec(cmd, (error, stdout, stderr) => {
      if (error) {
        console.error(`${error}`);
        return;
      }
      console.log(`${stdout}`);
      console.log(`${stderr}`);
    });
    ++children;
    child.on("close", () => {
      --children;
      if(children==0) {
        console.log(chalk.cyan("All builds completed"));
        console.log(chalk.green("Static file bundles generated"));
      }
    });
  }
} else {
  procid = parseInt(procid);
  config.entry = bundleNames.slice(procid, procid + QUEUE_SIZE).reduce((acc, cur) => {
    return {
      ...acc,
      [cur]: config.entry[cur]
    };
  }, {})
  webpack(config).run((error, stats) => {
    if(error) {
      console.error(chalk.red(error));
      return 1;
    }

    const jsonStats = stats.toJson();
    const stringStats = stats.toString(config.stats);

    if (jsonStats.hasErrors) {
      return jsonStats.errors.map(error => console.log(chalk.red(error)));
    }
    if (jsonStats.hasWarnings) {
      console.log(chalk.yellow("Webpack generated the following warnings: "));
      jsonStats.warnings.map(warning => console.log(chalk.yellow(warning)));
    }
    console.log(stringStats);
  });
}

Please note that my npm script and the webpack config use ES6. They will need babel_node to run:

{
  "presets": [ "env", "stage-1" ]
}

Results

My build time came down from 5m 2.364s to 2m 13.105s.

Nearly 50% improvement is pretty impressive for a 10 minute hack, eh?