I tested the Node.js test runner, so you don’t have to.
The release of Node.js 18 brought a surprise: an experimental test runner, like other languages like Python, Go or Deno. Node.js has been around since 2009, but given that until now it has not had an official test runner, several projects have attempted to take that spot. Although the runner is still experimental, given that this version will turn into a LTS version, I decided to compare to other established test runners.
In this article, I am not:
- Running any benchmarks.
- Going deep into details.
- Comparing with every single test runner around.
In this article, I am:
- Giving a gentle introduction to the new test runner.
- Comparing it feature wise with other frameworks I have used.
- Comparing the developer experience with other frameworks I have used.
As the test runner is still marked as experimental, this means that it could change at any time, or even completely dropped. Therefore, there is a chance that the content of this article is irrelevant if read it in the future.
Node.js test runner in a nutshell.
The specification of the test runner is short and concise. It can be found in the official documentation, but I will try to make even a shorter summary of it.
The test runner is executed by passing the —-test option to node. If no file paths are provided as input, it will recursively search for test files. If it can find a test folder, it will execute its contents as a test, else, it will search for files starting or ending in test.
Every file will be executed in a different child process. If all the test cases are successful, it will exit with a 0, otherwise it will exit with a 1. The results are returned using the TAP format, an old friend of Perl developers.
The test cases can be described using the test format or the describe format, both common in other testing libraries. You are free to use the one you prefer, and both are available as part of the package node:test. Independently of which one you use; the evaluation of the test cases will be the same. A synchronous test will fail if it throws an exception. An asynchronous test would fail if the promise returned rejects. A call-back function (yes, those still exist) will fail if the first argument of the call is falsy. The way you return these values is up to you. You can use the assert library of Node.js or Chai for instance. The tests also have some basic testing features like subtest, skipping tests, to-do’s and executing selected subsets.
The comparison
The comparison consists of setting a small test suite using different libraries. Each test suite is composed of two files, one which shows the basic test setup, and one which displays some extra features. The source code can be found in on Github, although it has no more value than a code example of how to implement the tests. The comparison is between node:test, Vitest, Jest, and Mocha.
The good
- The test format is very similar to the other libraries: In case you want to migrate from a different test runner, it will not be very dramatic. Testing libraries that use exceptions like Chai work out of the box.
- TAP format: Although not that common any more (especially in the JavaScript world), this format for reporting test results is still very common and has a lot of tooling around. I appreciate that they chose a standard for reporting rather than coming with a new standard.
- Multiprocessing per test: This is a nice to have. It gives you test isolation and possibly a speed-up.
- It seems to be the “fastest”: Take this statement with caution, but of all the test I implemented, Node.js was the fastest in my machine.
The bad
- BUGS: Skipped messages are not printed, the only flag is not able to find test, skipped tests are not correctly reported… I understand it is an experimental feature, but it has a lot of rough edges at the moment.
- Way too basic: Snapshots? Code coverage? Retries? Transpilation? Browser support? Mocking? Test watchers? Setup and tear down hooks? Before and after each hook? Global injection? It lacks features we are used to having nowadays. Maybe some years ago this would have been acceptable, but currently it is missing a lot of features that we are taking for granted.
The ugly
- The logging of the test could be friendlier. It makes things like a skipped test look worrisome.
- Configuring Jest is as annoying as I remembered it. I am glad that there are more alternatives.
Conclusion
It is very basic. It has rough edges. BUT it does the job.
It was about time for Node.js to get a test runner in the standard library when other languages like Python, Golang, or Rust have had it for years. If you are looking for a test runner to run exclusively in the backend (at the end, it is for Node.js), once it is stable, I believe this will be a legit option to consider.
If you already have a testing tool implemented in your project, I don’t think you will migrate to this one. Most of them have more features and more support from the ecosystem. However, if you are going to work in a green field project in Node.js, give it a chance. At least it will be less painful than setting up Jest.
