Using JSonnet for programmable Hasura metadata

The graphql-engine configuration (or "metadata") describes everything related to your tables, relationships and permissions that is needed to get Hasura running against your Postgres database, except for the Postgres instance itself. For that reason, it can be useful to export the configuration for backup, which is supported in the Hasura CLI, and also via the console's settings page.

However, while the exported configuration is useful for this backup and restore workflow, it is not stored in a very human-readable format, which makes it less useful for direct or programmatic manipulation.

Why would we want to programmatically generate or manipulate the Hasura configuration? For some more advanced use cases, it might be beneficial to generate the configuration from another data source. It might also be useful to reduce duplication in the configuration file in the interests of maintainability, especially if the configuration might change a lot over time. I'll also discuss some more possible use cases at the end of the article.

In this post, I'm going to work with the "multiple roles" example from the Hasura documentation, and show how we can refactor the exported configuration into a more readable form using the Jsonnet templating language.

Jsonnet is a "data templating language" which can be used to generate configuration files. This is preferable to using JSON directly, since JSON does not provide any programming capabilities of its own. Notably though, Jsonnet is not a full-fledged programming language, and its expressions cannot have side-effects, so it is a good fit for this sort of problem.

I'd like to point out that this is hardly the only valid approach to this problem. If you find this approach useful, you might like to consider other similar data templating languages such as Cue or Dhall, but it should also be possible to use your favorite programming language to generate a config programmatically, with different trade offs.

If you would like to follow along, you can recreate the database schema using the following script:

CREATE TABLE users (
    id INT PRIMARY KEY,
    name TEXT,
    profile JSONB,
    registered_at TIMESTAMP
);

CREATE TABLE articles (
    id integer PRIMARY KEY NOT NULL,
    title text,
    author_id integer REFERENCES users (id),
    is_reviewed boolean DEFAULT false,
    review_comment text,
    is_published boolean DEFAULT false,
    editor_rating integer
);

CREATE TABLE public.editors (
    editor_id integer PRIMARY KEY NOT NULL REFERENCES users (id)
);

CREATE TABLE public.reviewers (
    id integer PRIMARY KEY NOT NULL,
    article_id integer REFERENCES articles (id),
    reviewer_id integer REFERENCES users (id)
);

You will also need to make sure the jsonnet executable is available on your path.

If we export the Hasura configuration directly from the console (click the settings icon in the top right, or navigate to /console/settings), we receive a large amount of JSON:

Raw metadata

This file is slightly complicated, even for such a small example database, but it is easy to discover its structure: at the top level, we see a list of tables, each of which describes its own set of object and array relationships to other tables, and permissions for each role and operation. There is a lot of repetition which we can try to reduce by refactoring in small steps using Jsonnet.

The simplest refactoring we can make is to hoist out repeated constants to the top level. For example, one string which appears many times is the name of the Hasura user ID custom header, X-Hasura-User-Id, so let's declare that once as a Jsonnet local:

-- The HTTP header used to specify the user ID
local XHasuraUserId = "X-Hasura-User-Id";

Now we can refer to the constant using its given name in these places.

Notice another small but important benefit of Jsonnet: we can add comments to our configuration file, something we could not do with JSON. It's hard to overstate how useful this simple feature can be for writing a clear, readable configuration.

If we save this modified JSON file with a .jsonnet extension, we can pass it to the jsonnet executable and compare the output to the original JSON file to make sure we haven't changed the configuration:

jsonnet input.jsonnet | diff - original.json 

We can also hoist out objects and arrays which are duplicated across our configuration file. For example, the following two filter expressions are used more than once in our permissions, so we can move them out and define them once:

local authorFilter =
  {
    "author_id": {
      "_eq": XHasuraUserId
    }
  };
  
local reviewerFilter = 
  {
    "reviewers": {
      "reviewer_id": {
        "_eq": XHasuraUserId
      }
    }
  };

Note that we can make use of constants like XHasuraUserId that we have already defined. In this way, we're building up a library of constants, so that our configuration file can be understood in small steps.

Again, after this change, we can run our modified file through the jsonnet executable to make sure we haven't modified the original configuration.

For the next change, we can see that we mention the list of columns in the articles table multiple times, so again, let's move that out:

local allColumns = 
  [
    "author_id",
    "editor_rating",
    "id",
    "is_published",
    "is_reviewed",
    "review_comment",
    "title"
  ];

However, two roles have access to all columns except for one. It would be repetitive to list out the columns again, just for the sake of one column being removed from the set. Instead, let's express the set of allowed columns by subtracting the smaller set of disallowed columns from the complete set, using a standard library function:

local allColumnsExceptEditorRating = 
  std.setDiff(allColumns,
  [
    "editor_rating"
  ]);

This more clearly expresses our intent: the set of allowed columns is not arbitrary, but the full set minus just one disallowed column.

So far, we have only used constants, but Jsonnet also allows us to reduce duplication using functions, by adding arguments to our locals. As a simple example, we notice that the object and array relationships are very verbose compared to their actual information content, so we can deduplicate them by extracting the common structure into a pair of functions:

local objectRel(name, foreignKey) = 
  {
    "name": name,
    "using": {
      "foreign_key_constraint_on": foreignKey
    }
  };
  
local arrayRel(schema, name, column) =
  {
    "name": name,
    "using": {
      "foreign_key_constraint_on": {
        "column": column,
        "table": {
          "schema": schema,
          "name": name
        }
      }
    }
  };

Now, we can call these functions using syntax familiar to Javascript developers:

{
  ...,
  "object_relationships": [
    objectRel("user", "author_id")
  ],
  "array_relationships": [
    arrayRel("public", "reviewers", "article_id")
  ],
  ...
}

We are getting closer to a simple DSL (domain-specific language) for describing our configuration in Jsonnet. With a few functions like these, we can drastically reduce the amount of duplication, and succinctly express the configuration of the server.

The resulting Jsonnet file is considerably more readable than the original JSON.

In this post, we've used Jsonnet to tidy up an existing server configuration, which can be useful for maintenance purposes if the configuration is likely to change a lot over time. However, there are many ways in which a programmable configuration file can reduce complexity, and even support features which are not directly supported by graphql-engine:

• graphql-engine supports flat roles, but not hierarchical roles. We can flatten the hierarchy to emulate hierarchical roles, but the resulting configuration could easily become very complicated. With a configuration-driven workflow based on something like Jsonnet, we can perform that flattening in the configuration language. This allows us to compactly specify the hierarchical roles using a small set of functions, but hand off a set of flattened roles to graphql-engine in the form of the expanded configuration file.
• Similarly, graphql-engine only supports one role per user. There has been a proposal to add support for multiple roles, but any such proposal has to decide how to merge overlapping permissions between roles. Instead, we could delegate such a decision to a function in the configuration language. Additive permissions could become just one option, expressed as a reusable function in a permissions DSL.
• Permissions are currently attached directly to tables and views, but it can be useful to add a layer of indirection, attaching permissions to other sorts of resources, and then mapping those resources onto tables. Using a configuration language, we wouldn't need to build support for this into graphql-engine directly, and instead we could let the configuration language perform the mapping from resource-based permissions to table-based permissions.
• Permissions rules such as "if a user can write X, then they should be able to read X as well" can be easily expressed as reusable functions in the configuration language.
• Any sort of duplication of tables in the configuration (for example, for isolation purposes, or for sharding) could be easily deduplicated using functions.

We'll have more to say on the subject of programmatic configuration soon, but in the meantime, if you are using Jsonnet or another configuration language with Hasura, please let us know!