Git Push to Deploy Lambdas using CircleCI
We want to setup a
git push workflow to deploy our functions on AWS Lambda:
$ git push origin master * branch master -> FETCH_HEAD 9e364190..0e14a1fd master -> origin/master Updating 9e364190..0e14a1fd $ #triggers a circleci job that deploys your serverless function $ curl aws-myapi-gateway.com/function "Hello world"
We can do this in 4 easy steps:
- Setup a repo with functions
- One-time setup of AWS and CircleCI
- Configure CircleCI
- Git push
Functions-as-a-service is a service offered by cloud platforms in which you can deploy functions and get API endpoints to invoke the same. Building on FaaS can be really simple and powerful for use-cases which do not require the need of a full-fledged web service. AWS Lambda is the most popular FaaS platform out there today. A function on AWS Lambda is also simply called: a lambda (we will use the terms interchangeably).
In this post, we will solve one of the major pain points of using Lambda in production i.e. CI/CD. There are many tools available in the market which help you build and deploy Lambdas safely but, in this post, we will showcase the way to do it using the traditional approach of using git (aka gitops style).
Our aim is to keep 3 environments in the cloud:
The CI/CD system will deploy our application, comprised of multiple functions, to each environment based on the branch on which the code is pushed:
The end output is that we will have an environment specific HTTP endpoint for each of our functions.
Before we jump straight to deploying our lambdas, lets talk a bit about code organization. Our deployment configuration will depend on this.
Here is the repo structure which we will be using in this blogpost:
. ├── functions | |── echo | | |── index.js | |── helloWorld | |── myFunc | | ... ├── .circleci | |── config.yml | |── deploy.sh |── .git ├── .gitignore
It is a very simple structure:
- a git repo at the top level.
- a functions folder.
- a circleci folder (more on this later).
Each lambda is a separate folder inside the functions folder. We will assume each function is complete in itself i.e. it can be independently deployed on AWS Lambda without any further dependencies. One important point: This structure is not optimized for code/library sharing across functions. We can easily workaround this by having a
common folder with overlapping dependencies and use that during the build of each function. For simplicity, we will keep this setup out-of-scope of this tutorial.
Our CI/CD setup will require few resources which need to be setup one-time:
1. Lambda Execution Role on AWS
Log into AWS Console and create a IAM service role as per the docs here. This is the role with which our Lambdas will run. In case you already have a predefined role setup for your Lambdas, you can skip this step.
2. API Gateway on AWS
Log into AWS Console and go to API Gateway service page. Create a bare-bones API Gateway, like the following, per environment:
Each deployed function will be attached to a new route on this API Gateway e.g. a function called
hello will be routed to
<api-gateway-url>/hello and a function called
bye will be routed to
3. Add Repo in CircleCI
Log into CircleCI with Github/Bitbucket and you should see all your repos under Add Project:
Add your repo by clicking
Set Up Project and in the next page just click
Start Building (ignore all other instructions). This will setup all the hooks required in your repo so that CircleCI is triggered every-time there is a push. Currently, this will result in build failure as we haven't configured CircleCI yet. We will set this up in the next section.
4. Setup environment variables in CircleCI
We have to give few environment specific values and secrets to CircleCI build environment so that it can access the AWS resources. Head to your project settings in CircleCI:
Add the following environment variables:
#common AWS_ACCOUNT_ID AWS_REGION AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY #dev api gateway id AWS_DEV_REST_API_ID #stg api gateway id AWS_STG_REST_API_ID #prod api gateway id AWS_PROD_REST_API_ID
Circle CI configuration
CircleCI runs a build using the configuration present in
.circleci folder of the git repo. Our
.circleci folder contains 2 files:
config.yml has metadata about the workflow like what are the steps to execute, on what branch to run, etc. At a high level this is what our config.yml looks like:
version: 2 jobs: build_dev: steps: - checkout - run: name: Setup Environment Variables command: | echo 'export AWS_REST_API_ID="$AWS_DEV_REST_API_ID"' >> $BASH_ENV - run: name: Install dependencies command: | apt-get update apt-get install -y jq zip python3-pip ... - run: name: Deploy functions command: | cd functions ../.circleci/deploy.sh echo workflows: version: 2 full: jobs: - build_dev: filters: branches: only: master
We have a workflow which calls a job
build_dev only for the
master branch. The
build_dev job has few steps like setting up dependencies, building lambda, etc
This is the complete file for all 3 environments:
deploy.sh is a bash script that builds and deploys a folder (assumed to be a nodejs function here) to Lambda and links it to a route in the API Gateway. If you look at the
config.yml file above, in the final step we are calling
deploy.sh with the appropriate folder. We will only use the
aws-cli in the script for complete control and observability.
At a high level, these are the steps that are being done in
- Create a Lambda with the name
<functionName>_<environment>, if doesn't exist already.
- Zip the folder and upload the code to the above Lambda.
- Create an alias for the Lambda with the
GITSHAof the commit.
- Create a route in the API Gateway, if doesn't exist already.
- Link the alias with the route.
Here is the complete file:
Git push and go
We are all set. Once you have committed the
.circleci folder with the above files and pushed, your functions will be built and deployed on Lambda with API Gateway. You can check the logs of each build on the CircleCI dashboard. Congratulations, you have just set up a git push workflow for your Lambdas.