firebase2graphql: Moving from firebase to realtime GraphQL on Postgres

What if you could move from Firebase ? to a “real” open-source database ?preserving the same application development experience including realtime features?

Over the last few weeks we’ve been working on a small open-source utility to help you migrate data from firebase realtime db into Postgres and then expose realtime GraphQL on it, in a single command!

firebase2graphql -d firebase-data-export.json https://my-hasura-app.com

We use the Hasura GraphQL engine to expose GraphQL on postgres by dynamically creating the postgres schema and the GraphQL relationships from our CLI.

The two main problems we’ve solved are:

1. Retaining the same query and data model structure as data is migrated from firebase into Postgres
2. Automatically normalizing the denormalized firebase data model

In this post, I’ll talk about

• Our motivation behind this tool
• The engineering behind how this was done

Firebase’s realtime database is a revolutionary product for web and mobile app developers. We love the Firebase experience of being able to query whatever data is required on the frontend without needing to write backend APIs and have anything in our firebase “database” be realtime to boot.

GraphQL promises application developers the same thing!

1. Frontend first development: The ability to iterate on your application querying for precise slices of data as required.
2. The client-side SDK that made the developer experience of using Firebase awesome is analogous to the tremendously amazing experience of using a GraphQL client with GraphiQL to browse and explore the GraphQL API.
3. Firebase’s realtime features can now be captured as GraphQL subscriptions which offer a similar level of abstraction and convenience to us without having to much about with the underlying sockets, connections and things.

This is an example of a live-query that fetches user with id 1, their articles and any changes to that slice of data (spanning multiple tables) will automatically be sent to the client:  

subscription {
  user(where: {id: {_eq: 1}}) {
    id
    name
    email
    articles {
      id
      title
      content
    }
  }
}

While firebase is arguably still more convenient GraphQL is more generic and allows app developers to bring a similar joy in consuming a backend API to any backend.

One of the major problems with Firebase is the inability to scale to more complex requirements that invariably arise. Simple features like being able to query the database or create aggregations/transformations over large volumes of data become inconvenient and in some cases impossible.

Considering that we have an opportunity to preserve the same developer experience with GraphQL, including realtime features and move to the world’s most advanced open-source database, why not!

We built the Hasura GraphQL engine (github) to give app developers an instant realtime GraphQL API on postgres. This combines the awesomeness of Postgres as a database with the developer experience of GraphQL to be able to build applications without writing any code on the backend.

Our CLI tool firebase2graphql leverages Hasura to export data from firebase to Postgres, and optionally also normalise your data model to remove data duplication so that you still get easy queries with GraphQL to fetch related information without having to make multiple writes (mutations).

When you run this command:

firebase2graphql firebase-data-export.json https://my-hasura-app.com

…we need to get data from firebase into Postgres tables. The key ideas are:

1. Each node of data maps to a “row”
2. For every nested node, we create another table with a relationship to the parent table
3. If the nested node, is an “array” type we create what we call an array relationship, basically a one-to-many relationship
4. Otherwise, we create a “object” relationship, basically many-to-one relationship

With these high-level rules, we’re able to map the following firebase data:  

{
  "posts": {
    "-LMbLFOAW2q6GO1bD-5g": {
      "author": "Eena",
      "authorPic": "...photo.jpg",
      "body": "Content body of this article",
      "starCount": 0,
      "title": "My first post",
      "uid": "4UPmbcaqZKT2NdAAqBahXj4tHYN2"
    },
    "-LMbLIv6VKHYul7p_PZ-": {
      "author": "Eena",
      "authorPic": "...photo.jpg",
      "body": "Content body of this article",
      "starCount": 0,
      "title": "Whatta proaaa",
      "uid": "4UPmbcaqZKT2NdAAqBahXj4tHYN2"
    }
  },
  "user-posts": {
    "4UPmbcaqZKT2NdAAqBahXj4tHYN2": {
      "-LMbLFOAW2q6GO1bD-5g": {
        "author": "Eena",
        "authorPic": "...photo.jpg",
        "body": "Content body of this article",
        "starCount": 0,
        "title": "My first post",
        "uid": "4UPmbcaqZKT2NdAAqBahXj4tHYN2"
      },
      "-LMbLIv6VKHYul7p_PZ-": {
        "author": "Eena",
        "authorPic": "...photo.jpg",
        "body": "Content body of this article",
        "starCount": 0,
        "title": "Whatta proaaa",
        "uid": "4UPmbcaqZKT2NdAAqBahXj4tHYN2"
      }
    }
  },
  "users": {
    "4UPmbcaqZKT2NdAAqBahXj4tHYN2": {
      "email": "[email protected]",
      "profile_picture": "...photo.jpg",
      "username": "Eena"
    }
  }
}

…to this postgres schema:  

users (
	_id text not null primary key,
	email text,
    profile_picture text,
	username text
)

posts (  
	_id text not null primary key,
	title text,
	body text,
	starCount int,
	author text,
	authorPic text,
	uid text
)

user_posts (  
	_id text not null,
	_id_2 text not null,
	title text,
	body text,
	starCount int,
	author text,
	authorPic text,
	uid text
)

With this schema created and the data imported, Hasura now allows you to start making GraphQL queries (reads), mutations (writes) and subscriptions (realtime) on your data.

If you’re familiar with Firebase, you know that Firebase prefers your models to be as denormalised as possible. Write-many, read-once is the Firebase philosophy. This means that the data you import into Postgres with Phase I will have a lot of data duplication.

At this point, you can continue using GraphQL on postgres with the write-many approach (make a series of mutations, which will automatically be in a transaction) or we can ask firebase2graphql to attempt a normalisation.

This is the rough normalisation algorithm we follow:

1. Look at column names of each table, if another table has similar column names we try to see if they have overlapping data. This is done through a series of SQL queries on the imported data.
2. If an exact subset or duplicate table is found, the subset containing table is deleted and any tables related to it, are now changed to be related to the superset table.

This is more to help you get started with normalisation rather than be a final end. You can continue the normalisation process manually via the Hasura console by removing and creating relationships, dropping duplicate data and s o on.

Do go through our README for a more exhaustive guide: https://github.com/hasura/graphql-engine/tree/master/community/tools/firebase2graphql

Do try firebase2graphql and let us know what you think :) Refer to the Hasura docs if you’re looking for things like setting up security/access control on your GraphQL API.

We hang on discord, on github and our website intercom. Feel free to ask us questions :)