Introduction
On November 21st, Hasura became aware of a critical security vulnerability within our GraphQL Engine “Update Many” API, which was introduced in version 2.10.0. The vulnerability was identified by Morten Hillbom and Issaaf Kattan from Nhost's customer company Celsia.io. There was a Missing Authorization vulnerability which allowed some expansion of update capabilities of a user. This vulnerability impacted row level authorization on Update Many calls to Postgres datastores.
Details
In the implementation of the new Update Many API for Postgres backends, there was a bug in the code which would enforce the row level authorization mechanisms. Since this was within the Update Many API, a user could perform an update on any column they had update permissions to on any row within the target table, and subsequently retrieve any columns they had select permissions on from any impacted rows. In order for this to be exploited, the Update Many API would need to be called against a Postgres datastore, the API would need to not have been disabled using the configuration added in 2.14.0, the caller of the API would need to have update permissions on the target table, and the API would need to not be restricted by Allow Lists.
Timeline
As noted in the introduction, we became aware of the vulnerability on November 21st, however, this is not a complete representation of the issues encountered. Morten Hillbom and Issaaf Kattan identified an issue around row level matching and permissions around November 16th which they tracked down to the GraphQL Engine Update Many API. They reached out following our documented process of emailing [email protected] and provided details about the estimated scope and impact of the vulnerability, however, the email alias listed in the documentation had become misconfigured and was no longer sending its emails to the support ticketing tool. This resulted in our team not seeing the email. Fortunately, they persisted and reached out to Johan Eliasson at Nhost, who was able to use direct contacts to bring the vulnerability to our attention on November 21st.
Recognizing the dangers inherent in missing authorization and authorization bypass, we quickly investigated and began patching the vulnerability. By the end of the day on November 21st, we had produced a patch and deployed it to our cloud hosted environments. During the night and into the next day, November 22nd, we completed backports of the patch and released builds of all affected versions back to 2.10. Once patches were available, a security advisory was published on GitHub and notifications were sent out on our security mailing list and to our Cloud and Enterprise customers with limited details and ~10 business days lead to this notification, in order to give users who self-manage Hasura deployments the time needed to apply the patches.
Starting on November 22nd, we created queries to try to track adoption so we would know when it was safest to release these details.
Remediating Activities
In response to this vulnerability, Hasura has released patches for all impacted versions starting from 2.10 when the vulnerability was introduced and removed all vulnerable versions from docker hub. Hasura continues to monitor usage of vulnerable versions and strongly encourages users to update as soon as possible. For users on 2.14.0 or later, they may turn off the impacted API using --experimental-features=hide_update_many_fields.
In addition to supplying patched versions, Hasura is doing the following:
- Updating all documentation to use email distribution lists/aliases in order to ensure a human is receiving all messages, even if they are also going into an automated message box
- Completing an audit of existing mailboxes to ensure all reports have been properly tracked and addressed
- Updating testing procedures and automated tests in order to prevent regression
- Redesigning the authorization implementation in order to better ensure all APIs enforce every type of authorization
- Reviewing Dynamic Application Security Testing (DAST) tool coverage and processes
- Improving Vulnerability Management processes to ensure layered defenses
- Building and managing customer notification lists, so we can more easily and readily communicate vulnerabilities in particular types of deploys (Cloud, CE, EE, Postgres, Snowflake, etc.)
- Updating Security Best Practices for using Hasura to include layered defenses such as Allow Lists and access reviews.
Identifying Impacts
You can look into calls to the Update Many API within your services and review any tables which have been the target of these calls. If you have disabled API and telemetry logging or are not certain you keep all of these records long enough, then you should do a consistency check across all of your Postgres datastores behind Hasura 2.10.0 to 2.15.0.
We take these issues very seriously and will be making numerous changes to prevent and respond better in the future. If you have questions, or need assistance, please reach out to [email protected].
