Less.js can be run in two ways, firstly through node and secondly in the browser. A great deal of the code is shared, but not all of it - particularly the way it deals with imports, paths and url's differs. Whilst typing "make test" in the node less.js repository will run unit tests for the node part, there is nothing testing browser execution. Recently I merged a commit which I later found broke the error handling in the browser and that was when I decided we needed some way to test the browser code to stop obvious regressions.
What testing framework?
- + Runs in node (so, cross OS)
- - Requires building, so on windows you need python, visual studio & visual studio paths setup
- + available for Windows, Mac, Linux
- + very popular
I went for PhantomJS - it is difficult enough to get people to run unit tests before submitting patches, so I wanted the path to entry to be as simple as possible.
How would it work?
The node version of less.js has a folder full of less files and a folder full of css files, sub-folders are ignored and contain imported files. A custom node tester iterates through every file in the less directory, compiles it and then compares it to a file of the same name in the css folder. If it's different then it uses js diff (through node) to output a diff of the files. A similar process is employed for errors, except the error produced is compared to a file of the same name ending in .txt.
My first job was to get the existing tests running in the headless browser - I didn't think it would detect any errors, but it was a good proof that the browser tests were going to work.
I didn't want to mantain two copies for the tests, so a testing framework should load all of the css files, get all of the converted css and then compare them. I went for jasmine as I thought in the future we may have more 'normal' unit tests and it would mean I could use the PhantomJS jasmine examples in order to get PhantomJS to run the tests. We can always switch Jasmine out if we find ourselves not using it.
My initial idea runs along these lines...
All communication would be over the file system and it would just work.
XHR requests on the file system
I started by making up a test page and running it locally. Immediately I hit the problem that for security reasons you cannot do an XHR request on the filesystem. In IE you have to use a different mechanism which less.js doesn't yet support and in chrome you have to pass a command line parameter (--allow-file-access-from-files) to enable it. So, probably, not too many people are using less.js this way and I found some problems with enabling it in PhantomJS.
The conlusion was that I would need a webserver. Luckily, PhantomJS includes a webserver! This would allow me to automate the tests without requiring that someone starts a webserver. There are included examples for a contrived "fake web server", but surprisingly none for a web server that can server any file in a subset of the filesystem. Turns out however that adapting the example is easy..
and testing it in the browser showed that it worked fine. Because this is asyncronous I can start this off inside phantomjs and in the same script instance I can do the testing of the page - this stops having to mess around with starting a process up and then finding the process id so I could kill it at the end of the tests. Next, I added in the jasmine example to call the page
You can see the basic concept.. you
- setup a web server
- setup a PhantomJS page
- which requests from the webserver
- then it tests the dom in that page
- if it isn't there, wait and try again
- once it is there, retrieve the result
However, this fails.
So, I added the following to make less.js asyncronous (it is the current way of setting parameters, you create a less object with the right parameters and then put the less.js file link after it) -and the initial test page worked.
Automating all the files
Next up we just required a test runner that will look through all the links in the page and then load the css file using an xhr request and compare the sheets created by less.js with those loaded as expected output..
It wasn't much work and now we have the huge benefit of basic, automated browser testing without having to loading one. This work was added in two commits and you can run "make browser-test" which runs the PhantomJS tests (it builds the browser distributable, builds the test pages and then runs PhantomJS). If you want to debug the tests "make browser-test-server" starts the PhantomJS web server without running any tests, so you can navigate to the test causing the problem and see it in your browser.