Which test framework should I use?
At Ponicode, we have asked our users why they use the test framework they use, and very interestingly in the majority of cases the answer was “it’s the one I know the best”. Once introduced to a specific framework - be it by choice or because of a work project, people tend to stay with what they know. Once they become familiar with a framework, they apply that same knowledge to the next projects they work on, regardless of whether it’s the most adapted or not, until they encounter large enough hurdles that force them to reconsider their choices.
For this reason, we at Ponicode thought it would be a good idea to make a bit of clarity on the topic of the most popular Unit Testing software.
In this article, we are going to focus on unit tests specifically - and therefore will not talk about other useful tools such as Cypress, testing-library, Storybook and others which are more suited for integration, UI and end-to-end testing.
Unit testing tools are not all the same: they can serve different functionalities or different combinations of functionalities. The main categories of unit testing software are the following:
- Test runners:
These are libraries or tools that launch and execute your tests and aggregate their results. Tests in JS can be launched in different environments: in the browser, in a headless browser, or in Node.js. All three require different configurations and have different advantages and constraints, which is why we have different tools for running tests. Almost all test runners are also able to generate coverage reports, which detail how much of your code has been executed by your unit tests.
- Testing frameworks:
These are used to organise tests: they provide the wireframing that makes them structured and readable.
- Assertion libraries:
These provide functions that can be used to verify the validity of a statement. They are effectively syntaxic sugar that avoids us having to write a lot of if statements and try/catches, effectively making unit tests a lot more readable
- Test double libraries
These are also called mocking libraries, and are plugins that hook into assertions and testing frameworks and provide access to test doubles. They are used to isolate the unit we want to test and simulate specific test scenarios.
Now that we know how to classify unit testing software, let’s go through some of the most popular tools on the market.
Jasmine is an open source testing framework that comes ready out-of-the box, without a need for additional assertion libraries nor mocking libraries (although more libraries can be added if specific features are required). The fact it has no external dependencies means it’s relatively fast.
Jasmine creates test globals (keywords that can be used to test, such as “describe” or “test”) by default, so there is no need to import them in every single file, and it allows grouping and nesting tests in suites.
Its syntax is relatively straight forward. For example, the keyword “it” is used to introduce a test case, and it is supposed to be followed by a sentence that describes the indented test behaviour, in a format that makes it human readable. In this, it is inspired by the principle that non-technical people should be able to understand what is being tested, and for this reason it is a popular choice for all BDD lovers.
Despite coming into its 11 years of age and despite other, newer tools having gained in popularity, today Jasmine is still recommended as a first choice by Angular.
Jest takes the “ready-to-go” approach very seriously: it has its own runner, built-in assertions and mocking libraries. It also comes with a snapshot library, a functionality that sets it apart from Jasmine and other similar tools. Almost no configuration is needed to start testing a project with Jest, although advanced customisation is always possible. This means it is very accessible and adapted to beginners and pros alike.
In terms of syntax, it is very similar to Jasmine, with default globals and a BDD compliant test structure organised in suites.
One of Jest’s main strengths is its popularity: the fact it’s used on a large number of complex projects and that it has a large community, as well as the fact that it’s backed by the Facebook team, serve as guarantees of reliability.
Another reason why people would choose Jest over Jasmine is the performance of its test runner: thanks to parallel testing, Jest is able to significantly reduce the time it takes to run large suites of tests. The Jest runner runs tests in Node.js, not on a browser.
Not surprisingly, given it comes from the same team, Jest comes recommended by React. Given that it integrates well with other front-end testing tools such as Enzyme, it is generally particularly popular for front-end frameworks.
Differently from Jest and Jasmine, Mocha is only a test framework: it has no test runner, no built-in assertion libraries and no default mocking tools, therefore it requires its users to choose third-party plugins depending on their project and their testing needs. The most popular choice of assertion library to pair with Mocha is Chai, but other options are Should.js, Express.js, Unexpected and Better-assert.
Being a pretty bare-bone and lightweight framework gives Mocha some disadvantages but also some advantages over the competition: while it is definitely a little harder to get started with - due to the need for a bit more setup and configuration, it offers pretty much endless possibilities for advance configuration and integration, which allow the advanced user to achieve the perfect test environment for a specific project.
Mocha has a command line utility which runs tests serially, without an option for parallel testing, making it comparatively slow.
Mocha was designed originally to work with Node.js, but today works on a large range of frameworks - including front-end frameworks such as React, Vue and Angular, as long as you are willing to get your hands dirty in a bit of config.
It follows a BDD / TDD approach, which means that its assertions tend to follow a syntax that is very similar to human language, but it gives the user a choice as to the specific keywords they want to use: should, expect or assert.
While Chai does little more than provide an API for assertions, it can be extended to include more functionality through a wide range of plugins created by the community, or even custom plugins.
Sinon is a popular open source library for test doubles: stubs, mocks and spies. In this domain, it pretty much allows doing all that it’s feasible by the JS language.
While mocking is not a trivial topic, Sinon attempts to make the syntax relatively simple by using assertions that read out similar to human language.
It It works with any unit test framework, but it’s most often associated with Mocha.
In practice, Karma provides a command line tool that spins up one or several browsers and runs the tests inside of them all, displaying the results directly on the standard output rather than in a separate browser window. It was created with a focus on minimal configuration, and focused on making the experience of unit testing in the browser more similar to that of unit testing in Node.js.
Karma is testing framework agnostic, but in practice its most popular application is in pair with Jasmine for AngularJS tests.
Ava is another open source test runner and test framework for Node.Js.
It is more ready-to-go than Mocha, but less so than Jest or Jasmine, because it comes with its own built-in assertion library but not with a dubbing library. Sinon or other similar plugins can of course be used to achieve this functionality. It also has an out-of-the box snapshot library.
Overall, Ava is pretty opinionated in its choices of syntax and test structure. While it is not limited in terms of functionality, it forces writing tests that follow very simple structure and assertions. For instance, Ava doesn’t have explicit test suites, so you can’t group your tests at a more nested level than the file level.
Similarly - differently from Jest, Mocha and Jasmine, Ava does not create default globals, which means that in every Ava test you will see an import of the testing framework itself.
Another advantage of Ava is that it’s able to run tests in parallel as separate Node.js processes and therefore in isolated environments for each file, making it a serious contender in terms of speed. The community also loves Ava’s extremely clean reporting.
Until recently, Ava was not easily usable on TypeScript files because it does not include source file compilation (a reason for its speed!). These days, ava-ts solves this problem.
Ponicode is not a unit testing framework like Jest or Mocha, nor a test runner like Karma, nor a library for asserting or mocking like Chai or Sinon...
Ponicode is a piece of software that avoids you having to do all the heavy lifting in terms of writing the test, choosing the correct syntax, defining what to test and writing expectations.
We are not just talking about snippets generation: Ponicode is able to create not only the structure of your tests, but also their content, thanks to an AI-based test case suggestions engine and an integrated runner which evaluates the output of your function for the given inputs.
Effectively, Ponicode allows you to keep control of your unit tests through a Graphical Interface, and push them to the extent you want (with specific assertions, mocking, etc), but without having to worry about their syntax or about inventing values to pass as input.
We plan to expand our support to several different JS test frameworks in the future, starting with Mocha, but we’d love to hear your opinion too! What’s a framework you think Ponicode would marry well with?
If you are interested in our Python support, our choice of framework was Pytest, for very much the same reasons as Jest: loved by the community and large array of functionality with relatively simple syntax.