Automate AWS Lambda function deployments to AWS CDK

This is the first tutorial in a two-part series. You can also learn how to use AWS CDK to automatically deploy REST APIs with Lambda authorizers.

While building a cloud-based application, you have a choice to either deploy the resources manually using the graphical user interface (GUI), using the command line interface (CLI) provided by the cloud provider, or to do it manually. The manual approach could work well if you just have a handful of resources, but as the complexity of your application increases, it can lead to costly bottlenecks and mistakes.

Instead of deploying your cloud resources manually, you could use an infrastructure-as-code solution like Terraform or AWS CDK that lets you manage your infrastructure code programmatically. AWS CDK helps you use your existing skills and tools for developing a cloud infrastructure. Not only does the power of expressive object-oriented programming languages speed up the development process, it eliminates the need for context switching. That’s because you can define both infrastructure code and runtime code using the same IDE and tools. AWS CDK also makes it easier to integrate your code with Git workflows and allows you to use CI/CD pipelines for automating the deployment process.

In this tutorial, you will learn how to use AWS Cloud Development Kit (CDK) to deploy an AWS Lambda function that interacts with AWS S3 and AWS DynamoDB.

Prerequisites

For this tutorial, you will be using NodeJS to define your AWS CDK application and the AWS Lambda handler. You will also need to install AWS CLI and AWS CDK CLI on your system to configure AWS credentials and to manually build your CDK application.

Refer to this list to set up everything required for this tutorial:

• Install NodeJS and NPM.
• Install AWS CLI.
• Configure AWS credentials.
• Install AWS CDK CLI.
• Create an AWS account.
• Create a CircleCI account

Our tutorials are platform-agnostic, but use CircleCI as an example. If you don’t have a CircleCI account, sign up for a free one here.

Create a new AWS CDK project

To create a new CDK project, you need the CDK CLI installed on your system. First, create a new directory for your CDK project and open it:

mkdir aws-cdk-lambda-circle-ci
cd aws-cdk-lambda-circle-ci

Next, using the CDK CLI run the cdk init command to create a new CDK project using Typescript. The app parameter specifies the template that you are going to use for initializing the project.

cdk init app --language typescript

Executing this command creates a new CDK project with a few files in it. Later in the tutorial, I will explain the significance of some of these files and the constructs defined in them.

Note: CDK supports multiple languages such as Typescript, Python, Java, and C#. You can choose to use any language that you prefer.

Add a NodeJS Lambda function

In this section, you will define an AWS Lambda function using NodeJS. The Lambda function will demonstrate how you can save a CSV file to AWS S3 and add an entry to a DynamoDB table.

Create a lambda directory at the root of the CDK project. Inside the lambda directory add an index.js file for the lambda handler and a package.json file for defining the dependencies.

In the package.json file, define the name of the NodeJS project and add a few dependencies which will be used by your handler.

{
    "name": "ddb-s3-lambda-function",
    "version": "0.1.0",
    "dependencies": {
        "csv-stringify": "^6.0.5",
        "fs": "0.0.1-security",
        "uuid": "^8.3.2"
    }
}

After you add the dependencies, run the npm install command in the lambda directory to install the packages.

Next, in the index.js file define an empty function (you will implement this later).

exports.handler = async function (event) {

}

Create a CSV file with demo data and save the CSV as a temp file. At the bottom of the index.js file you created earlier, add:

// add imports at the top of the file
var fs = require('fs');
const {stringify} = require('csv-stringify');

function writeCsvToFileAndUpload(filePath, objectKey) {
    var data = getCsvData();
    var output = stringify(data);

    fs.writeFileSync(filePath, output);
    // we will add the uploadFile method later
    uploadFile(filePath, objectKey);
}

function getCsvData() {
    // return some CSV data
    return [
      ['1', '2', '3', '4'],
      ['a', 'b', 'c', 'd']
    ];
}

Next, define another function in the index.js file that takes a local file path and an S3 object path, and uploads the file to AWS S3:

// add imports at the top of the file
const AWS = require('aws-sdk');
AWS.config.update({ region: 'us-west-2' });

const s3 = new AWS.S3();
const BUCKET_NAME = process.env.BUCKET_NAME

const uploadFile = (fileName, objectKey) => {
    // Read content from the file
    const fileContent = fs.readFileSync(fileName);

    // Setting up S3 upload parameters
    const params = {
        Bucket: BUCKET_NAME,
        Key: objectKey,
        Body: fileContent
    };

    // Uploading files to the bucket
    s3.upload(params, function (err, data) {
        if (err) {
            throw err;
        }
        console.log(`File uploaded successfully. ${data.Location}`);
    });
    return objectKey;
};

The function reads the contents of the local file and uses the upload function of AWS.S3 SDK to upload the file.

Add the implementation for the empty AWS Lambda handler that you created in the index.js file. The Lambda handler receives jobId in its event parameter. The handler first uploads the CSV file to AWS S3 and then updates the DynamoDB table with the jobId and the upload object’s S3 path.

//add import at the top of the file
const { v4: uuidv4 } = require('uuid');

var ddb = new AWS.DynamoDB();
const TABLE_NAME = process.env.TABLE_NAME

exports.handler = async function (event) {
    try {
        const uploadedObjectKey = generateDataAndUploadToS3()
        const jobId = event['jobId']
        var params = {
            TableName: TABLE_NAME,
            Item: {
                'jobId': { S: jobId },
                'reportFileName': { S: uploadedObjectKey }
            }
        };

        // Call DynamoDB to add the item to the table
        await ddb.putItem(params).promise();;
        return {
            "status": "success",
            "jobId": jobId,
            "objectKey": uploadedObjectKey
        }
    } catch (error) {
        throw Error(`Error in backend: ${error}`)
    }
}

const generateDataAndUploadToS3 = () => {
    var filePath = '/tmp/test_user_data.csv'
    const objectKey = `${uuidv4()}.csv`;
    writeCsvToFileAndUpload(filePath, objectKey)
    return objectKey
}

The handler uses the putItem method of the AWS.DynamoDB SDK to insert a new item in a DynamoDB table.

Define CDK constructs for the application

Now that you have defined your AWS Lambda handler, it’s time to define all the CDK constructs that will be used in your application. AWS CDK Constructs are cloud components encapsulating the configuration detail and gluing logic for using one or multiple AWS services. CDK provides a library of constructs for most of the commonly used AWS services.

When you generate the CDK project using the app template, the lib/aws-cdk-lambda-circle-ci-stack.ts file is created for you containing the AwsCdkLambdaCircleCiStack class. You will be defining the CDK constructs in this file.

//  The snippet shows the original contents for reference. You do not need to replace the file contents.
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class AwsCdkLambdaCircleCiStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // we will add all the constructs here
  }
}

Define in the stack:

• An AWS S3 bucket to hold the CSV files uploaded by the AWS Lambda function
• A DynamoDB table in which the jobId and object key will be updated by the AWS Lambda function.
• An AWS Lambda function that will use both the S3 and the DymamoDB table. Also, you need to make sure that the AWS Lambda function receives the BUCKET_NAME and TABLE_NAME as parameters.
• Give the AWS Lambda function adequate permissions to perform operations on the S3 bucket and the DymamoDB table.

Define an AWS S3 bucket

Add a CDK construct to create an AWS S3 bucket inside the constructor defined in the lib/aws-cdk-lambda-circle-ci-stack.ts file. AWS S3 bucket names are unique across all AWS accounts so you will need to provide a unique name for your bucket.

import * as cdk from 'aws-cdk-lib';
import * as s3 from 'aws-cdk-lib/aws-s3';
import { Construct } from 'constructs';

constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

  // we will add all the constructs here
  // provide a unique name for your S3 bucket
  const circleCiGwpBucket = new s3.Bucket(this, "circle-ci-gwp-bucket", {
    bucketName: "<YOUR_BUCKET_NAME>",
  });
}

At this point, if you try deploying the stack, it will simply deploy a CloudFormation application and create an AWS S3 bucket for you.

Define a DynamoDB table

To define a DynamoDB table, add a CDK construct to create a DynamoDB table inside the constructor defined in the lib/aws-cdk-lambda-circle-ci-stack.ts file.

import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import { Construct } from 'constructs';

constructor(scope: Construct, id: string, props?: cdk.StackProps) {
  // other code //
  //add the following construct after the existing code in the constructor
  const circleCiGwpTable = new dynamodb.Table(this, "CircleCIGwpTable", {
    tableName: "CircleCIGwpTable",
    partitionKey: { name: "jobId", type: dynamodb.AttributeType.STRING },
  });
}

Note: The table name must be unique in your AWS account. If you already have a table named CircleCIGwpTable in your AWS account, update the tableName while defining the DynamoDB construct.

Notice that jobId is defined as a primary partition key for the table. This will ensure that jobId values are unique in the table.

Define an AWS Lambda function

To define an AWS Lambda function, add a CDK construct to create an AWS Lambda function inside the constructor defined in the lib/aws-cdk-lambda-circle-ci-stack.ts file. The AWS Lambda function will use the NodeJS runtime and will use the code that you defined in the lambda directory. Also, the S3 bucket name and the DynamoDB table name will be passed to the function as environment variables.

import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { Construct } from 'constructs';

constructor(scope: Construct, id: string, props?: cdk.StackProps) {
  // other code //
  //add the following construct after the existing code in the constructor
  const circleCiGwpLambda = new lambda.Function(
    this,
    "CircleCiGwpLambda",
    {
      runtime: lambda.Runtime.NODEJS_14_X,
      handler: "index.handler",
      timeout: cdk.Duration.seconds(30),
      code: lambda.Code.fromAsset("lambda/"),
      environment: {
        TABLE_NAME: circleCiGwpTable.tableName,
        BUCKET_NAME: circleCiGwpBucket.bucketName
      },
    }
  );
}

The timeout for the AWS Lambda function is defined to be 30 seconds. The maximum execution time for a Lambda function is 15 minutes.

Grant permissions to AWS Lambda function

Finally, grant adequate permissions to the AWS Lambda function. The Lambda function needs putObject permissions to the S3 object. Grant these permissions following the existing code in the constructor of lib/aws-cdk-lambda-circle-ci-stack.ts:

circleCiGwpBucket.grantPut(circleCiGwpLambda);

The function also needs read/write permissions to the DynamoDB table. After the existing code in the constructor of lib/aws-cdk-lambda-circle-ci-stack.ts, add:

circleCiGwpTable.grantReadWriteData(circleCiGwpLambda);

You can define more complex IAM policies using the AWS CDK IAM module.

Deploy the CDK stack

Now that you have defined the CDK constructs in your stack, you can go ahead and deploy the application to an AWS account. You will first deploy the project manually to make sure everything works. Once you verify it, you can automate the deployments using CircleCI.

Before deploying for the first time, you need to bootstrap the project using the cdk CLI. Bootstrapping the app provisions the resources that might be required by AWS CDK to deploy your application. These resources might include an S3 bucket for storing deployment-related files and IAM roles for granting deployment permissions. From the root of the project, run:

cdk bootstrap

Make sure that you have the AWS credentials configured on your system. If the credentials are configured, CDK will use it automatically. If they aren’t, click the link in the prerequistes for instructions.

Next, deploy the application to the AWS account:

cdk deploy

Once you execute the command, you might be prompted to confirm the IAM role/policy changes that would be applied to your account. Deployment should work successfully if your application is set up correctly and you have all the prerequisites available on your system.

Automate application deployment using CircleCI

You were able to deploy the CDK application manually using the command line, but now you can automate the workflow. Automation packages and deploys infrastructure changes every time you push code to the main branch. To automate the deployments:

• Update .gitignore
• Update NPM scripts
• Add configuration script
• Create a CircleCI project
• Set up environment variables

Update .gitignore

The code generated by the cdk init command contains a .gitignore file that ignores all .js files by default. Make sure to replace the contents of .gitignore with this:

!jest.config.js
*.d.ts
node_modules

# CDK asset staging directory
.cdk.staging
cdk.out

Update NPM scripts

Your CircleCI deployment configuration uses NPM scripts for executing the deploy and diff commands. In the root level package.json file, add these scripts:

// update the aws-cdk-lambda-circle-ci/package.json file with the following scripts
{
  ... 
  "scripts": {
    ...
    // add the ci_diff and ci_deploy scripts
    "ci_diff": "cdk diff -c env=${ENV:-stg} 2>&1 | sed -r 's/\\x1B\\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g' || true",
    "ci_deploy": "cdk deploy -c env=${ENV:-stg} --require-approval never"
  },
  ...
}

Add configuration script

First, add a .circleci/config.yaml script in the root of the project containing the configuration file for the CI pipeline. Add this to the config file:

version: 2.1

orbs:
  aws-cli: circleci/aws-cli@2.0.6
executors:
  default:
    docker:
      - image: 'cimg/node:14.18.2'
    environment:
      AWS_REGION: 'us-west-2'
jobs:
  build:
    executor: 'default'
    steps:
      - aws-cli/setup:
          aws-access-key-id: AWS_ACCESS_KEY
          aws-secret-access-key: AWS_ACCESS_SECRET
          aws-region: AWS_REGION_NAME
      - checkout
      - run:
          name: 'install_lambda_packages'
          command: |
            cd lambda/authorizer && npm install
            cd ../../
            cd lambda/processJob && npm install
            cd ../../
      - run:
          name: 'build'
          command: |
            npm install
            npm run build
      - run:
          name: 'cdk_diff'
          command: |
            if [ -n "$CIRCLE_PULL_REQUEST" ]; then
              export ENV=stg
              if [ "${CIRCLE_BRANCH}" == "develop" ]; then
                export ENV=prd
              fi 
              pr_number=${CIRCLE_PULL_REQUEST##*/}
              block='```'
              diff=$(echo -e "cdk diff (env=${ENV})\n${block}\n$(npm run --silent ci_diff)\n${block}")
              data=$(jq -n --arg body "$diff" '{ body: $body }') # escape
              curl -X POST -H 'Content-Type:application/json' \
                -H 'Accept: application/vnd.github.v3+json' \
                -H "Authorization: token ${GITHUB_TOKEN}" \
                -d "$data" \
                "https://api.github.com/repos/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/issues/${pr_number}/comments"
            fi
      - run:
          name: 'cdk_deploy'
          command: |
            if [ "${CIRCLE_BRANCH}" == "main" ]; then
              ENV=prd npm run ci_deploy
            elif [ "${CIRCLE_BRANCH}" == "develop" ]; then
              ENV=stg npm run ci_deploy
            fi 

The CI script uses CircleCI’s aws-cli orb for setting AWS configuration (the access key and secret). The build job also has a few different steps that install the packages, calculate the diff, and deploy the changes. The cdk_diff step executes only on pull requests and adds a comment on the PR summarizing the infrastructure changes.

The cdk_deploy command checks the branch and deploys on the prd or stg environment. Notice that the cdk_deploy command executes the ci_deploy script defined in the package.json file.

Your pipeline configuration will take care of building, packaging, and deploying the CDK stack to the specified AWS account.

Commit the changes and push your project to GitHub.

Note: Be sure to replace the AWS_REGION in the example with your region if it is different.

Create a CircleCI project for the application

Next, set up the repository as a CircleCI project using the CircleCI console. On the Circle CI console, click Projects tab, search for the GitHub repo name and click Set Up Project for your project.

You will be prompted to choose whether you want to manually add a new configuration file or use an existing one. Since you have already pushed the required configuration file to the codebase, select the Fastest option. Enter the name of the branch hosting your configuration file. Click Set Up Project to continue.

Completing the setup will automatically trigger the pipeline. The pipeline will fail in its first run. That’s to be expected because you haven’t defined the environment variables yet.

Set up environment variables

Click Project settings from the project dashboard and click the Environment variables tab. Click the Add environment variable button to add a new key value. Add the AWS access key and secret you created earlier in the tutorial as AWS_ACCESS_KEY and AWS_ACCESS_SECRET. Also, set the environment variable for AWS_REGION_NAME to the region that you’ll deploy your application to.

Once the environment variables are configured, run the pipeline again. This time it should build successfully.

Conclusion

In this tutorial, you learned how AWS CDK makes it easier to manage infrastructure-related code. With AWS CDK you can provision resources for your application using languages already familiar to you. AWS CDK allows you to use logical statements and use object-oriented techniques while defining your application. AWS CDK also makes the infrastructure code testable using industry-standard protocols. This tutorial walked you through a very common use case of defining an AWS Lambda function with package dependencies that interact with other AWS services. You learned to define the stack with all the AWS services and grant fine-grained permissions to it.

You can check out the full source code used in this tutorial on Github. The Github project can also be used as a template for you if you are trying to define a similar kind of stack.

---

Vivek Kumar Maskara is a Software Engineer at JP Morgan. He loves writing code, developing apps, creating websites, and writing technical blogs about his experiences. His profile and contact information can be found at maskaravivek.com.

Read more posts by Vivek Maskara