Git Push to Deploy Lambdas using CircleCI

TL;DR:

We want to setup a git push workflow to deploy our functions on AWS Lambda:

$ git push origin master

 * branch              master -> FETCH_HEAD
   9e364190..0e14a1fd  master -> origin/master
Updating 9e364190..0e14a1fd

$ #triggers a circleci job that deploys your serverless function

$ curl aws-myapi-gateway.com/function
"Hello world"

We can do this in 4 easy steps:

1. Setup a repo with functions
2. One-time setup of AWS and CircleCI
3. Configure CircleCI
4. Git push

Functions-as-a-service is a service offered by cloud platforms in which you can deploy functions and get API endpoints to invoke the same. Building on FaaS can be really simple and powerful for use-cases which do not require the need of a full-fledged web service. AWS Lambda is the most popular FaaS platform out there today. A function on AWS Lambda is also simply called: a lambda (we will use the terms interchangeably).

In this post, we will solve one of the major pain points of using Lambda in production i.e. CI/CD. There are many tools available in the market which help you build and deploy Lambdas safely but, in this post, we will showcase the way to do it using the traditional approach of using git (aka gitops style).

Our aim is to keep 3 environments in the cloud:

1. Dev
2. Stg
3. Prod

The CI/CD system will deploy our application, comprised of multiple functions, to each environment based on the branch on which the code is pushed:

The end output is that we will have an environment specific HTTP endpoint for each of our functions.

Before we jump straight to deploying our lambdas, lets talk a bit about code organization. Our deployment configuration will depend on this.

Here is the repo structure which we will be using in this blogpost:

.
├── functions
|   |── echo
|   |   |── index.js   
|   |── helloWorld
|   |── myFunc
|   |   ...
├── .circleci
|   |── config.yml            
|   |── deploy.sh            
|── .git
├── .gitignore    

It is a very simple structure:

1. a git repo at the top level.
2. a functions folder.
3. a circleci folder (more on this later).

Each lambda is a separate folder inside the functions folder. We will assume each function is complete in itself i.e. it can be independently deployed on AWS Lambda without any further dependencies. One important point: This structure is not optimized for code/library sharing across functions. We can easily workaround this by having a common folder with overlapping dependencies and use that during the build of each function. For simplicity, we will keep this setup out-of-scope of this tutorial.

Our CI/CD setup will require few resources which need to be setup one-time:

1. Lambda Execution Role on AWS

Log into AWS Console and create a IAM service role as per the docs here. This is the role with which our Lambdas will run. In case you already have a predefined role setup for your Lambdas, you can skip this step.

2. API Gateway on AWS

Log into AWS Console and go to API Gateway service page. Create a bare-bones API Gateway, like the following, per environment:

Each deployed function will be attached to a new route on this API Gateway e.g. a function called hello will be routed to <api-gateway-url>/hello and a function called bye will be routed to <api-gateway-url>/bye.

3. Add Repo in CircleCI

Log into CircleCI with Github/Bitbucket and you should see all your repos under Add Project:

Add your repo by clicking Set Up Project and in the next page just click Start Building (ignore all other instructions). This will setup all the hooks required in your repo so that CircleCI is triggered every-time there is a push. Currently, this will result in build failure as we haven't configured CircleCI yet. We will set this up in the next section.

4. Setup environment variables in CircleCI

We have to give few environment specific values and secrets to CircleCI build environment so that it can access the AWS resources. Head to your project settings in CircleCI:

Add the following environment variables:

#common
AWS_ACCOUNT_ID
AWS_REGION
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

#dev api gateway id
AWS_DEV_REST_API_ID

#stg api gateway id
AWS_STG_REST_API_ID

#prod api gateway id
AWS_PROD_REST_API_ID

CircleCI runs a build using the configuration present in .circleci folder of the git repo. Our .circleci folder contains 2 files: config.yml and deploy.sh

config.yml

The config.yml has metadata about the workflow like what are the steps to execute, on what branch to run, etc. At a high level this is what our config.yml looks like:

version: 2
jobs:
  build_dev:
    steps:
      - checkout
      - run:
          name: Setup Environment Variables
          command: |
            echo 'export AWS_REST_API_ID="$AWS_DEV_REST_API_ID"' >> $BASH_ENV
      - run:
          name: Install dependencies
          command: |
            apt-get update
            apt-get install -y jq zip python3-pip
            ...
      - run:
          name: Deploy functions
          command: |
            cd functions
            ../.circleci/deploy.sh echo
workflows:
  version: 2
  full:
    jobs:
      - build_dev:
          filters:
            branches:
              only: master

We have a workflow which calls a job build_dev only for the master branch. The build_dev job has few steps like setting up dependencies, building lambda, etc

This is the complete file for all 3 environments:

deploy.sh

The deploy.sh is a bash script that builds and deploys a folder (assumed to be a nodejs function here) to Lambda and links it to a route in the API Gateway. If you look at the config.yml file above, in the final step we are calling deploy.sh with the appropriate folder. We will only use the aws-cli in the script for complete control and observability.

At a high level, these are the steps that are being done in deploy.sh

1. Create a Lambda with the name <functionName>_<environment>, if doesn't exist already.
2. Zip the folder and upload the code to the above Lambda.
3. Create an alias for the Lambda with the GITSHA of the commit.
4. Create a route in the API Gateway, if doesn't exist already.
5. Link the alias with the route.

Here is the complete file:

We are all set. Once you have committed the .circleci folder with the above files and pushed, your functions will be built and deployed on Lambda with API Gateway. You can check the logs of each build on the CircleCI dashboard. Congratulations, you have just set up a git push workflow for your Lambdas.

PS: If you are using Hasura, then you can setup this workflow for your Remote Schemas and Event Triggers :)