Hasura Pro: Regression testing for your GraphQL API

Hasura Pro has a comprehensive testing framework that lets you validate your GraphQL schema or test its integrity manually and/or programmatically. This helps you get early feedback on potential regressions in your GraphQL API.

Testing a Hasura-based GraphQL API needs to factor in the following aspects of Hasura’s architecture:

• Hasura generates a different GraphQL schema for each role defined in its built-in authorization system.
• A large part of GraphQL engine is essentially a compiler that converts GraphQL requests into performant SQL.

Based on this, here’s how you could think about some of the traditional API test types:

1. Unit tests: Simple API unit tests are completely unnecessary with Hasura because you are essentially testing Hasura’s compiler :)
   
2. Load tests: While GraphQL engine is performant and scalable, you should load test your GraphQL API to identify any bottlenecks in your database.
   
3. Regression tests: Arguably the only kind of tests you need with GraphQL engine - regression tests ensure continued support for operations required by your frontend apps or users i.e. validating changes to the GraphQL schema (schema integrity) against these operations to ensure that there are no breaking changes or regressions in your GraphQL API. Changes to the underlying Postgres schema and/or the Hasura configuration could potentially lead to unwanted regressions in your schema. Therefore, schema validation is crucial to get early feedback when iterating on new features.
   

In the next section, we’ll take a look at how to configure a Regression test suite and test changes in your local or dev instance.

Hasura Pro comes with a comprehensive test bench that lets you seamlessly compile a test suite on each instance and execute them on any GraphQL engine instance. Here’s a reference development workflow that is enabled by Hasura Pro:

1) Build your database schema and configure Hasura as required by your frontend apps or public GraphQL API.

2) Deploy changes to production after testing them.

3) Create a regression suite on production.

4) Iterate on your GraphQL schema to support new features or edits

    a) Test changes in your dev instance against the production instance’s regression test suite. Fix any issues highlighted by the tests or plan to communicate regressions to affected stakeholders.

5) [Coming Soon] Run all changes through a CI/CD pipeline

   b) Run Regression tests programmatically for all changes in the team

6) Promote changes to prod

Creating a test suite

Each Hasura Pro instance can be configured with a separate test suite. Hasura Pro captures an immutable list of operations received by an instance for you to select the operations you want to add to its test suite. Ideally, you want to create a regression test suite on an instance which has received requests with operations you’d like to continue supporting or ensure are not “broken” - your production instance or a shared Q.A instance which receives operations in your app or, if you have a public GraphQL API,  those from your consumers.  

Here’s an example of operations from a production instance being added to its tests suite:

Running a test suite

A good development workflow would require that tests be run as follows:

• As early as possible in the workflow for catching regressions.
• Independently for all changes without exclusively relying on an individual developer to do so manually.

A test suite configured on a Hasura Pro instance can be run on the same instance or any other Hasura Pro instance registered to your team or organisation, including local ones. This is how we recommend that you incorporate regression tests into your GraphQL engine workflows:

Test manually with schema and metadata migrations

Let’s say you’re a developer iterating on a feature and ,as part of your work, need to modify your Postgres schema or the Hasura configuration. It is likely that you are doing so by running the console via the Hasura CLI to generate migrations that you can version control. Before committing your changes in git, you should run tests to get an early warning for potential regressions. Your team may want to designate the test suite from your production instance (or a suitable alternative) as the default suite to be used for this, and you can choose to run this test suite on your local or development instance.

For example, let's say the column 'title' (in a typical authors and articles schema) has been modified as part of a feature iteration. Assuming the operation from the previous example is part of the test suite on production, here's how the feedback on this change looks like:

As you can see, one of the tests fails because it expects a field, title, to be part of the type articles - which is something our proposed change just modified and removed support for.

[Coming Soon] Run tests programmatically in your CI/CD pipeline

We are working on a feature where the Hasura CLI exposes commands for you to programmatically trigger execution of a test suite in your automated testing setup, typically in CI scripts. Running these tests programmatically ensures that your team catches potential breaking changes even if individual developers don't manually test for this. Keep watching this space!

Regression testing is part of the Hasura Pro feature set - if you are interested in checking it out, set up a chat with us!