Many years ago, when I first tried my hand at web development, the rapid development cycles were a welcome change; simply save your changes and hit refresh. This was in contrast to the C++ work I was also doing which was quite tedious, waiting for the code to compile, link and eventually execute after each and every change.
How things have changed.
I’ve recently been working on an open source project, d3fc, which has a fairly typical grunt build. Over time we’ve added more steps to the build and more code to the project and things have started to get pretty slow. I’d really like to get back to the almost instant feedback that we know is possible with these technologies.
This blog post shares a few steps I took to improve the performance of our grunt build, hopefully some of the tools I used will be of use to others.
We all know that the first step in improving performance is to instrument. You need to know which steps in your build are time-consuming before trying to optimise them.
Thankfully with grunt this is really quite simple, just add the
time-grunt plugin to your build:
When the build has finished you get a neat little summary of the execution time for each task:
As you can see the build takes 14s, which is fine for a CI or release build, but for development I want much more rapid feedback, ultimately as a way to support work in small iterations.
You don’t need to know what d3fc is to follow this post, but it is worth knowing what these build steps are:
jscs:components- these run JSHint and JSCS on our library, providing consistent code style.
rollup:components- this is our dependency mechanism, where the various files are rolled-up into a single library file.
jscs:test- again code style rules, this time applied to tests
jasmineNodejs:test- unit tests
jscs:visualTests- more code style!
assemble:visualTests- this probably needs a bit of explanation, our library is visual in nature, so we have a suite of ‘visual tests’, i.e. HTML tests for visual inspection. This task uses assemble, a static site generator, to generate the test pages.
One repeated pattern in the above build is the need to run JSHint and JSCS against each logic component of the codebase.
Parallel builds, take one
One of the reasons people favour gulp over grunt is its built-in support for parallel task running. However, you can very easily make grunt run tasks in parallel via the
grunt-concurrent task. My initial thought was to run JSCS and JSHint in parallel for each component.
grunt-concurrent is very easy, just configure the task with an array of sub-tasks to run in parallel:
The use the
concurrent:componentCheck task in place of the ones it replaces.
However, with the above tasks running in parallel, the build was actually slower by ~1.5 seconds!
time-grunt also produces a report for concurrent tasks, immediately revealing the issue:
Running “concurrent:componentCheck” (concurrent) task
Each concurrent task is loading all of the grunt tasks required by the Gruntfile. Looking at the implementation of
time-grunt you can see why this is the case, it uses
grunt.util.spawn to spawn a new process for each tasks, each executing grunt. Clearly this approach doesn’t make sense for parallelising a small number of relatively rapid tasks.
Grunt loads all the referenced tasks regardless of whether they are used or not, this is a known issue. It’s also exacerbated by
matchdep, the plugin that loads tasks which are referenced in your
It’s very easy to forget to clean up your
package.json resulting in your grunt build loading tasks unnecessarily.
JIT task loading
As ever, when grunt doesn’t support something, you can almost guarantee that there will be a plugin that does! In this case it’s
jit-grunt, a just-in-time plugin loader. Simply replace the manual
matchdep step and replace with the following:
For our project build this gave an immediate improvement of ~2.5 seconds:
It also has the added benefit that if you want to execute a single task, it is also very fast:
jit-grunt doesn’t support
grunt.renameTask, I’ve raised an issue and might look into a a fix for this.
Parallel builds, take two
With the task loading optimised it was time to return to running tasks in parallel. With this project a much better split is to run the tasks relating to the two logic components (library code, visual test harness) in parallel:
This gives a significant improvement, bringing the build time down to around 8 seconds:
From JSHint+JSCS to ESLINT
Both do their job very well, however, running two separate analysis tools over the code does feel a little inefficient.
An alternative I’ve used a few times recently is ESLint which has rules that cover both languages constructs and style. Hopefully just using ESLint should be faster than JSHint and JSCS combined!
However, the first problem is migrating to ESLint, with each of these tools supporting 100s of rules I didn’t want to manually map between them. This got me thinking, what if you could auto-configure the ESLint ruleset to the most ‘aggressive’ set of rules based on an existing codebase?
Of course I’m not the first person to think this, there’s a tools called
dryer available on GitHub that does just that, running the ESLint CLI, checking which rules fail, then building a configuration based on the results.
Of course there are a number of rules it cannot readily derive from the code, such as the level of indentation, but it does give a big head start.
So how does it compare?
Here’s JSCS + JSHint:
And Here’s ESLint:
Around 3.6 seconds, versus 2.6 seconds. A small improvement, but worthwhile. Also, it does make your build simpler, and your associated tooling (i.e. your editor only needs one plugin for code style checking).
Interestingly I found out that the ESLint team are currently working on an auto-configuration tool
The end result
With all these changes in place the overall build time was reduced from 14 to 6.5 seconds. This might not sound like much, but for developer productivity it is a big improvement.
It’s probably about time I stopped fiddling around with the build and got some real work done …
Regards, Colin E.