Improving our Cypress test automation time by 3x
We had written a bunch of spec files to test our UI, a few to simulate user behaviour and also verifying through our APIs to check the correctness of the UI. Now all of these tests ran synchronously which meant that it took more than 30 mins to test all our core UI features.
To increase coverage, we had to add more tests and that meant the time for tests will only go up. This was blocking our quick test <-> iterate -> deployment process of graphql-engine.
Parallel tests, multiple processes
We resorted to running parallel tests by spawning multiple processes in the same container, knowing that it depends on the hardware resource to run as many as it can.
But we ran into the issue of
Could not start Xvfb ,
Could not stop Xvfb due to a smaller timeout (initially 2 seconds, which was increased to 5 seconds) in detecting if
Xvfb was installed and running. There are a few open issues on github, which deals with the above. The only frustrating bit was that these issues kept happening randomly (few builds would succeed and few would fail) and hence this optimisation could never be merged to our deployment process.
Parallel tests, multiple containers
While we were figuring out how to optimise tests duration, Cypress released v3.1.0 last month, which enables parallelism on CI systems with a simple
--parallel flag. By enabling a cluster of docker containers in your CI, cypress automatically detects your CI provider’s environment variables and parallelises your tests.
To setup parallel tests, we also had to enable
--record flag and that required a project to be setup on Cypress Dashboard service to record our tests.
Note: The dashboard service is free for open source projects.
The docs are straight-forward to setup a project using the Test Runner. Each project will have a
projectId which needs to be configured in
cypress.json and a record key which needs to be passed to
We use CircleCI for CI/CD and we just had to enable
in our configuration file. (4 is the number of containers). Parallelism Docs for the same are straightforward as well.
We had categorised our tests into different folders and spec files depending on the service.
Finally the cypress command that we had to run to get all of this working:
cypress run --spec 'cypress/integration/**/**/test.js' --key $CYPRESS_KEY --parallel --record
And the end result… our overall build times came down from ~45 mins to ~15 mins. Here’s one of our recent PR build on CircleCI that builds console with parallelism enabled.
Note the time taken for
test console in before and after parallelism tests.