Automate AWS Lambda function deployments to AWS CDK

This tutorial covers:

1. Defining your AWS CDK application and the AWS Lambda handler
2. Manually building and deploying your CDK application
3. Automating the deployments

This is the first tutorial in a two-part series. You can also learn how to use AWS CDK to automatically deploy REST APIs with Lambda authorizers.

When you build a cloud-based application, you can choose to deploy the resources using the GUI (Graphical User Interface) or CLI (Command Line Interface) provided by the cloud provider. This approach can work well with just a handful of resources, but as the complexity of your application increases, it can become difficult to manage the infrastructure manually.

Instead of deploying your cloud resources manually, you can use a solution like Terraform or AWS CDK that lets you manage your infrastructure code programmatically. With the power of expressive object-oriented programming languages, AWS CDK lets you use your existing skills and tools for developing a cloud infrastructure that speeds up the development process. Using AWS CDK eliminates the need for context switching because you can define both infrastructure code and runtime code using the same IDE and tools. AWS CDK also makes it easier to integrate your code with git workflows and allows you to use CI/CD pipelines for automating the deployment process.

In this tutorial, I will guide you through using AWS Cloud Development Kit (CDK) to deploy an AWS Lambda function that interacts with AWS S3 and AWS DynamoDB.

Prerequisites

For this tutorial, you will need to set up Node.js on your system to define your AWS CDK application and the AWS Lambda handler. You will also need to install AWS CLI and AWS CDK CLI on your system so that you can configure AWS credentials and manually build your CDK application. You will need an AWS account for deploying the application and a CircleCI account for automating the deployments.

Here is a list of everything you need in place to follow along with this tutorial:

• NodeJS and NPM
• AWS CLI
• Configured AWS credentials
• AWS CDK CLI
• AWS account
• A CircleCI account

Our tutorials are platform-agnostic, but use CircleCI as an example. If you don’t have a CircleCI account, sign up for a free one here.

Create a new AWS CDK project

First, create a new directory for your CDK project and navigate into it.

mkdir aws-cdk-lambda-circle-ci
cd aws-cdk-lambda-circle-ci

Next, use the CDK CLI to run the cdk init command, which creates a new CDK project using TypeScript. The app parameter specifies the template to use when initializing the project.

cdk init app --language typescript

Executing this command creates a new CDK project with a few files in it. Later on in the tutorial, I will explain the significance of some of these files and the constructs defined in them.

Note: CDK supports multiple languages such as TypeScript, Python, Java, and C#. You can choose to use any language that you are comfortable with.

Add a NodeJS Lambda function

In this section, you will define an AWS Lambda function using Node.js. The Lambda function demonstrates how you can save a CSV file to AWS S3 and add an entry to a DynamoDB table.

To get started, create a lambda directory at the root of the CDK project. Inside the lambda directory, add an index.js file for the lambda handler and a package.json file for defining the dependencies.

In the package.json file define the name of the NodeJS project and add a few dependencies which will be used by our handler. Here is what the file should contain:

{
    "name": "ddb-s3-lambda-function",
    "version": "0.1.0",
    "dependencies": {
        "csv-stringify": "^6.0.5",
        "fs": "0.0.1-security",
        "uuid": "^8.3.2"
    }
}

After you add the dependencies, run the npm install command in the lambda directory to install the packages.

Next, in the index.js file, define an empty function. You will be implementing this function later on in the tutorial.

exports.handler = async function (event) {

}

Now you can move on to implementation. Create a CSV file with dummy data and save it as a temp file. Add this code snippet to the bottom of the index.js file you created earlier:

// add imports at the top of the file
var fs = require('fs');
const {stringify} = require('csv-stringify');

function writeCsvToFileAndUpload(filePath, objectKey) {
    var data = getCsvData();
    var output = stringify(data);

    fs.writeFileSync(filePath, output);
    // we will add the uploadFile method later
    uploadFile(filePath, objectKey);
}

function getCsvData() {
    // return some CSV data
    return [
      ['1', '2', '3', '4'],
      ['a', 'b', 'c', 'd']
    ];
}

Next, define another function in the index.js file that takes a local file path and S3 object path and uploads the file to AWS S3.

// add imports at the top of the file
const AWS = require('aws-sdk');
AWS.config.update({ region: 'us-west-2' });

const s3 = new AWS.S3();
const BUCKET_NAME = process.env.BUCKET_NAME

const uploadFile = (fileName, objectKey) => {
    // Read content from the file
    const fileContent = fs.readFileSync(fileName);

    // Setting up S3 upload parameters
    const params = {
        Bucket: BUCKET_NAME,
        Key: objectKey,
        Body: fileContent
    };

    // Uploading files to the bucket
    s3.upload(params, function (err, data) {
        if (err) {
            throw err;
        }
        console.log(`File uploaded successfully. ${data.Location}`);
    });
    return objectKey;
};

This function reads the contents of the local file and uses the upload function of AWS.S3 SDK to upload the file.

Finally, add the implementation for the empty AWS Lambda handler that you created in the index.js file. The Lambda handler receives jobId in its event parameter. The handler first uploads the CSV file to AWS S3, and then updates the DynamoDB table with the jobId and the upload object’s S3 path.

//add import at the top of the file
const { v4: uuidv4 } = require('uuid');

var ddb = new AWS.DynamoDB();
const TABLE_NAME = process.env.TABLE_NAME

exports.handler = async function (event) {
    try {
        const uploadedObjectKey = generateDataAndUploadToS3()
        const jobId = event['jobId']
        var params = {
            TableName: TABLE_NAME,
            Item: {
                'jobId': { S: jobId },
                'reportFileName': { S: uploadedObjectKey }
            }
        };

        // Call DynamoDB to add the item to the table
        await ddb.putItem(params).promise();;
        return {
            "status": "success",
            "jobId": jobId,
            "objectKey": uploadedObjectKey
        }
    } catch (error) {
        throw Error(`Error in backend: ${error}`)
    }
}

const generateDataAndUploadToS3 = () => {
    var filePath = '/tmp/test_user_data.csv'
    const objectKey = `${uuidv4()}.csv`;
    writeCsvToFileAndUpload(filePath, objectKey)
    return objectKey
}

The handler uses the putItem method of the AWS.DynamoDB SDK to insert a new item in a DynamoDB table.

Define CDK Constructs for the application

Now that you have defined the AWS Lambda handler, you can define all the CDK constructs that will be used in your application. AWS CDK Constructs are cloud components that encapsulate the configuration detail and glue logic for using one or multiple AWS services. CDK provides a library of constructs for the most commonly used AWS services.

Generating the CDK project using the app template creates the lib/aws-cdk-lambda-circle-ci-stack.ts file. This file contains the AwsCdkLambdaCircleCiStack class. Use this file to define the CDK constructs.

//  The snippet shows the original contents for reference. You do not need to replace the file contents.
import { Stack, StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class AwsCdkLambdaCircleCiStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    // we will add all the constructs here
  }
}

Next, review what you need for your application to work. These tasks will be described in the next sections of this tutorial.

• Create an AWS S3 bucket to hold the CSV files uploaded by the AWS Lambda function.
• Create a DynamoDB table in which the jobId and object key will be updated by the AWS Lambda function.
• Define an AWS Lambda function that will use both the S3 and the DymamoDB table. Make sure that the AWS Lambda function has the BUCKET_NAME and TABLE_NAME as parameters.
• Ensure that the AWS Lambda function has adequate permissions to perform operations on the S3 bucket and the DymamoDB table.

Define an AWS S3 bucket

To define an AWS S3 bucket, add a CDK construct to create an AWS S3 bucket inside the constructor defined in the lib/aws-cdk-lambda-circle-ci-stack.ts file. AWS S3 bucket names are unique across all AWS accounts, so you will need to provide a unique name for your bucket.

import { Stack, 
  StackProps,
  //update the existing import to add aws_s3
  aws_s3 as s3
 } from 'aws-cdk-lib';

constructor(scope: Construct, id: string, props?: StackProps) {
  super(scope, id, props);

  // we will add all the constructs here
  // provide a unique name for your S3 bucket
  const circleCiGwpBucket = new s3.Bucket(this, "circle-ci-gwp-bucket", {
    bucketName: "<YOUR_BUCKET_NAME>",
  });
}

At this point, if you try deploying the stack, it will simply deploy a CloudFormation application and create an AWS S3 bucket for you.

Define a DynamoDB table

To define a DynamoDB table, add a CDK construct to create a DynamoDB table inside the constructor defined in the lib/aws-cdk-lambda-circle-ci-stack.ts file.

import {
  Stack,
  StackProps,
  aws_s3 as s3,
  //update the existing import to add aws_dynamodb
  aws_dynamodb as dynamodb
} from 'aws-cdk-lib';

constructor(scope: Construct, id: string, props?: StackProps) {
  // other code //
  //add the following construct after the existing code in the constructor
  const circleCiGwpTable = new dynamodb.Table(this, "CircleCIGwpTable", {
    tableName: "CircleCIGwpTable",
    partitionKey: { name: "jobId", type: dynamodb.AttributeType.STRING },
  });
}

The table name must be unique in your AWS account. If you already have a table named CircleCIGwpTable in your AWS account, update the tableName while defining the DynamoDB construct.

Notice that you have defined jobId as a primary partition key for the table. This will ensure that jobId values are unique in the table.

Define an AWS Lambda function

To define an AWS Lambda function, add a CDK construct to create an AWS Lambda function inside the constructor defined in the lib/aws-cdk-lambda-circle-ci-stack.ts file. The AWS Lambda function will use the NodeJS runtime and will use the code that we defined in the lambda directory. Also, the S3 bucket name and the DynamoDB table name will be passed to the function as environment variables.

import {
  Stack,
  StackProps,
  aws_s3 as s3,
  aws_dynamodb as dynamodb,
  //update the existing import to add aws_lambda and Duration
  aws_lambda as lambda,
  Duration
} from 'aws-cdk-lib';

constructor(scope: Construct, id: string, props?: StackProps) {
  // other code //
  //add the following construct after the existing code in the constructor
  const circleCiGwpLambda = new lambda.Function(
    this,
    "CircleCiGwpLambda",
    {
      runtime: lambda.Runtime.NODEJS_14_X,
      handler: "index.handler",
      timeout: Duration.seconds(30),
      code: lambda.Code.fromAsset("lambda/"),
      environment: {
        TABLE_NAME: circleCiGwpTable.tableName,
        BUCKET_NAME: circleCiGwpBucket.bucketName
      },
    }
  );
}

This code snippet specifies that the timeout for the AWS Lambda function is 30 seconds. The maximum execution time for a Lambda function can be up to 15 minutes.

Grant permissions to AWS Lambda function

Finally, grant adequate permissions to the AWS Lambda function. The Lambda function needs putObject permissions to the S3 object. Grant these permissions by adding the following construct after the existing code in the constructor of lib/aws-cdk-lambda-circle-ci-stack.ts:

circleCiGwpBucket.grantPut(circleCiGwpLambda);

The Lambda function also needs read/write permissions to the DynamoDB table. Add the following construct after the existing code in the constructor of lib/aws-cdk-lambda-circle-ci-stack.ts:

circleCiGwpTable.grantReadWriteData(circleCiGwpLambda);

You can define more complex IAM policies using the AWS CDK IAM module.

Deploy the CDK stack

Now that you have defined the CDK constructs in your stack, you can deploy the application to an AWS account. First deploy the project manually to make sure everything works. Then, once you verify that it is functional, you can automate the deployments using CircleCI.

Before deploying the project for the first time, you need to bootstrap the project using the cdk CLI. Bootstrapping the app provisions the resources that might be required by AWS CDK to deploy your application. These resources might include an S3 bucket for storing deployment-related files and IAM roles for granting deployment permissions. Issue this command from the root of the project:

cdk bootstrap

Make sure that you have the AWS credentials configured on your system. If the credentials are configured, CDK will use it automatically.

Next, deploy the application to the AWS account.

cdk deploy

Once you execute the command, you might be prompted to confirm the IAM role/policy changes that would be applied to your account. Deployment should work successfully if your application is set up correctly and you have all the prerequisites available on your system.

Automate application deployment using CircleCI

Now that you have deployed the CDK application manually using the command line, you can automate the workflow. Automating the workflow means that the infrastructure changes can be packaged and deployed automatically every time you push code to the main branch. You will need to complete the following tasks:

• Update .gitignore
• Update NPM scripts
• Add a configuration script
• Create a CircleCI project
• Set up environment variables

Update .gitignore

The code generated by the cdk init command contains a .gitignore file that ignores all .js files by default. Replace the contents of .gitignore with this code snippet:

!jest.config.js
*.d.ts
node_modules

# CDK asset staging directory
.cdk.staging
cdk.out

Update NPM scripts

The CircleCI deployment configuration uses NPM scripts for executing the deploy and diff commands. Add these scripts to the root level package.json file:

// update the aws-cdk-lambda-circle-ci/package.json file with the following scripts
{
  ... 
  "scripts": {
    ...
    // add the ci_diff and ci_deploy scripts
    "ci_diff": "cdk diff -c env=${ENV:-stg} 2>&1 | sed -r 's/\\x1B\\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g' || true",
    "ci_deploy": "cdk deploy -c env=${ENV:-stg} --require-approval never"
  },
  ...
}

Add configuration script

First, add a .circleci/config.yml script in the root of the project containing the configuration file for the CI pipeline. Add this code snippet to it:

version: 2.1

orbs:
  aws-cli: circleci/aws-cli@2.0.6
executors:
  default:
    docker:
      - image: 'cimg/node:14.18.2'
    environment:
      AWS_REGION: 'us-west-2'
jobs:
  build:
    executor: 'default'
    steps:
      - aws-cli/setup:
          aws-access-key-id: AWS_ACCESS_KEY
          aws-secret-access-key: AWS_ACCESS_SECRET
          aws-region: AWS_REGION_NAME
      - checkout
      - run:
          name: 'install_lambda_packages'
          command: |
            cd lambda/authorizer && npm install
            cd ../../
            cd lambda/processJob && npm install
            cd ../../
      - run:
          name: 'build'
          command: |
            npm install
            npm run build
      - run:
          name: 'cdk_diff'
          command: |
            if [ -n "$CIRCLE_PULL_REQUEST" ]; then
              export ENV=stg
              if [ "${CIRCLE_BRANCH}" == "develop" ]; then
                export ENV=prd
              fi 
              pr_number=${CIRCLE_PULL_REQUEST##*/}
              block='```'
              diff=$(echo -e "cdk diff (env=${ENV})\n${block}\n$(npm run --silent ci_diff)\n${block}")
              data=$(jq -n --arg body "$diff" '{ body: $body }') # escape
              curl -X POST -H 'Content-Type:application/json' \
                -H 'Accept: application/vnd.github.v3+json' \
                -H "Authorization: token ${GITHUB_TOKEN}" \
                -d "$data" \
                "https://api.github.com/repos/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/issues/${pr_number}/comments"
            fi
      - run:
          name: 'cdk_deploy'
          command: |
            if [ "${CIRCLE_BRANCH}" == "main" ]; then
              ENV=prd npm run ci_deploy
            elif [ "${CIRCLE_BRANCH}" == "develop" ]; then
              ENV=stg npm run ci_deploy
            fi 

The CI script uses CircleCI’s aws-cli orb for setting AWS configuration – the access key and secret. The build job has a few different steps that install the packages, calculate the diff, and deploy the changes. The cdk_diff step executes only on pull requests and adds a comment on the PR that summarizes the infrastructure changes.

The cdk_deploy command checks the branch and deploys only on the prd or stg environment. The cdk_deploy command executes the ci_deploy script defined in the package.json file.

The pipeline configuration will take care of building, packaging, and deploying the CDK stack to the specified AWS account. Commit the changes and push them to the Github repository.

Note: Be sure to replace the AWS_REGION with your own region, if it is different.

Create a CircleCI project for the application

Next, set up the repository as a CircleCI project using the CircleCI console. On the Projects tab of the console, search for the GitHub repo name. Click Set Up Project for your project.

You will be prompted to manually add a new configuration file or use an existing one. You have already pushed the required configuration file to the codebase, so select the Fastest option and enter the name of the branch hosting your configuration file. Click Set Up Project to continue.

Completing the setup will automatically trigger the pipeline. The pipeline will fail on its first run since we haven’t defined the environment variables.

Set up environment variables

Click Project settings from the project dashboard, then click the Environment variables tab. Click Add environment variable. You should have created an AWS access key and secret as mentioned in the prerequistes for this tutorial. Add those values as AWS_ACCESS_KEY and AWS_ACCESS_SECRET respectively. Also, set the environment variable for AWS_REGION_NAME to a region where you wish to deploy your application.

Once the environment variables are configured, run the pipeline again. This time it should build successfully.

Conclusion

This tutorial showed you how AWS CDK makes it easier to manage infrastructure-related code. With AWS CDK you can provision resources for your application using languages familiar to you. AWS CDK allows you to use logical statements and object-oriented techniques while defining your application. Also, AWS CDK makes the infrastructure code testable using industry-standard protocols.

This tutorial walked you through the very common use case of defining an AWS Lambda function with package dependencies that interact with other AWS services. I hope you agree how straightforward it is to define the stack with all the AWS services and grant fine-grained permissions to it using the steps outlined here. You can check out the full source code used in this tutorial on GitHub. The GitHub project can also be used as a template for you if you are trying to define a similar kind of stack.

---

Vivek Kumar Maskara is a Software Engineer at JP Morgan. He loves writing code, developing apps, creating websites, and writing technical blogs about his experiences. His profile and contact information can be found at maskaravivek.com.

Read more posts by Vivek Maskara