Improving our Cypress test automation time by 3x

At Hasura, we use Cypress, an e2e testing framework to test our console ui, which is the interface powering our graphql-engine. Cypress tests are written using Mocha and Chai, and hence the syntax is familiar territory for modern javascript developers, amongst other reasons to move from selenium.

We had written a bunch of spec files to test our UI, a few to simulate user behaviour and also verifying through our APIs to check the correctness of the UI. Now all of these tests ran synchronously which meant that it took more than 30 mins to test all our core UI features.

To increase coverage, we had to add more tests and that meant the time for tests will only go up. This was blocking our quick test <-> iterate -> deployment process of graphql-engine.

We resorted to running parallel tests by spawning multiple processes in the same container, knowing that it depends on the hardware resource to run as many as it can.

But we ran into the issue of Could not start Xvfb , Could not stop Xvfb due to a smaller timeout (initially 2 seconds, which was increased to 5 seconds) in detecting if Xvfb was installed and running. There are a few open issues on github, which deals with the above. The only frustrating bit was that these issues kept happening randomly (few builds would succeed and few would fail) and hence this optimisation could never be merged to our deployment process.

While we were figuring out how to optimise tests duration, Cypress released v3.1.0 last month, which enables parallelism on CI systems with a simple --parallel flag. By enabling a cluster of docker containers in your CI, cypress automatically detects your CI provider’s environment variables and parallelises your tests.

To setup parallel tests, we also had to enable --record flag and that required a project to be setup on Cypress Dashboard service to record our tests.

Note: The dashboard service is free for open source projects.

The docs are straight-forward to setup a project using the Test Runner. Each project will have a projectId which needs to be configured in cypress.json and a record key which needs to be passed to --key flag.

We use CircleCI for CI/CD and we just had to enable

parallelism: 4

in our configuration file. (4 is the number of containers). Parallelism Docs for the same are straightforward as well.

We had categorised our tests into different folders and spec files depending on the service.

Finally the cypress command that we had to run to get all of this working:

cypress run --spec 'cypress/integration/**/**/test.js' --key $CYPRESS_KEY --parallel --record

And the end result… our overall build times came down from ~45 mins to ~15 mins. Here’s one of our recent PR build on CircleCI that builds console with parallelism enabled.

Note the time taken for test console in before and after parallelism tests.