TutorialsDec 20, 20185 min read

Building CI/CD pipelines using the CircleCI AWS ECR Orb

Developer Advocate, CircleCI

CircleCI has recently released a new product called orbs which are designed to get you up and running quickly on CircleCI. You can now easily integrate your DevOps tools with trusted orbs provided by our technology partners.

In this post I’ll demonstrate and explain the AWS ECR Orb and its usage within the CircleCI configuration file, leveraging CircleCI Workflows, and pushing an image to Docker Hub and a specified AWS ECR.

What are CircleCI orbs?

Orbs are packages of CircleCI configuration that can be shared across projects. Orbs allow you to make a single bundle of jobs, commands, and executors that can reference each other and can be imported into a CircleCI build configuration and invoked in their own namespace. Orbs are registered with CircleCI, with revisions expressed using the semver pattern. CircleCI orbs are hosted and maintained on the CircleCI Orbs Registry which is a centralized repository for CircleCI orbs.

Prerequisites

Before you get started you’ll need to have these things:

GitHub account
Fork or clone the orbs-ecr-python repo locally
Docker Hub account
AWS account
AWS IAM User Account with Access Keys & Secrets
Create a new AWS ECR Repository with the name orbs-ecr-python

After you have completed all the prerequisites, you’re ready to proceed to the next section.

Configure CircleCI Project Environment Variables

Our project’s CI/CD pipeline will deploy Docker images to multiple container repositories so you need to set some environment variables within the CircleCI Project Settings:

Click Add Project on the CircleCI Dashboard in the left menu
Find and click the project’s name in the projects list and Click Set Up Project on the right side
Click the Project cog button in the CircleCI Dashboard in the top right area
In the Build Settings section, click Environment Variables
Click Add Variable

In the Add an Environment Variable dialog box you’re going to define several environment variables needed for this build. Below is a list of the environment variables that must be defined:

Name: AWS_ECR_ACCOUNT_URL Value: Your AWS ECR Registry URL
Name: AWS_ACCESS_KEY_ID Value: Your AWS IAM Account's Access Key ID
Name: AWS_SECRET_ACCESS_KEY Value: Your AWS IAM Account's Secret Access Key
Name: DOCKER_LOGIN Value: Your Docker Hub User Name
Name: Docker_PWD Value: Your Docker Hub Password

Setting these environment variables correctly is critical to the build successfully completing. Ensure that these variables and their values are properly set before continuing to the next section.

Using the CircleCI AWS ECR Orb

Below is the config.yml file that defines this project’s pipeline. In the following sections I’ll explain the various element of this config.yml file.

version: 2.1

orbs:
  aws-ecr: circleci/aws-ecr@0.0.4

workflows:
  build_test_deploy:
    jobs:
      - build_test
      - docker_hub_build_push_image:
          requires:
            - build_test
      - aws-ecr/build_and_push_image:
          region: us-east-1
          account-url: ${AWS_ECR_ACCOUNT_URL}
          repo: ${CIRCLE_PROJECT_REPONAME}
          tag: ${CIRCLE_BUILD_NUM}
          requires:
            - build_test
jobs:
  build_test:
    docker:
      - image: circleci/python:2.7.14
    steps:
      - checkout
      - run:
          name: Setup VirtualEnv
          command: |
            virtualenv helloworld
            . helloworld/bin/activate
            pip install --no-cache-dir -r requirements.txt
      - run:
          name: Run Tests
          command: |
            . helloworld/bin/activate
            python test_hello_world.py
  docker_hub_build_push_image:
    docker:
      - image: circleci/python:2.7.14
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: false
      - run:
          name: Build and push Docker image to Docker Hub
          command: |
            echo 'export TAG=0.1.${CIRCLE_BUILD_NUM}' >> ${BASH_ENV}
            echo 'export IMAGE_NAME=${CIRCLE_PROJECT_REPONAME}' >> ${BASH_ENV}
            source ${BASH_ENV}
            docker build -t ${DOCKER_LOGIN}/${IMAGE_NAME} -t ${DOCKER_LOGIN}/${IMAGE_NAME}:${TAG} .
            echo ${DOCKER_PWD} | docker login -u ${DOCKER_LOGIN} --password-stdin
            docker push ${DOCKER_LOGIN}/${IMAGE_NAME}

Specifying workflows and orbs

orbs:
  aws-ecr: circleci/aws-ecr@0.0.4

The orbs: key specifies that an orb will be used in this pipeline. The aws-ecr: keys defines an internal name used within the config. The circleci/aws-ecr@0.0.4 value specifies and associates the actual orb to be used and referenced by the aws-ecr: key. These orb statements could be considered as import statements found in other languages and frameworks.

workflows:
  build_test_deploy:
    jobs:
      - build_test
      - docker_hub_build_push_image:
          requires:
            - build_test
      - aws-ecr/build_and_push_image:
          region: us-east-1
          account-url: ${AWS_ECR_ACCOUNT_URL}
          repo: ${CIRCLE_PROJECT_REPONAME}
          tag: ${CIRCLE_BUILD_NUM}
          requires:
            - build_test

Workflow definitions

The workflows: key specifies a list of workflows which are composed of build jobs and orbs. This segment specifies a workflow called build_test_deploy:

jobs:
  - build_test
  - docker_hub_build_push_image:
      requires:
        - build_test

In this segment a jobs: key is specified and lists all of the jobs and orbs executed within this pipeline. The first job in this list is build_test which doesn’t have job dependency.

Orb definitions

The next job in the workflow list, docker_hub_build_push_image:, refers to a job that will be discussed later, but make note of the requires: key which specifies that the docker_hub_build_push_image: job is dependent on the build_test job passing. Otherwise the entire build will fail and the remaining jobs in the workflow will not execute.

- aws-ecr/build_and_push_image:
    region: us-east-1
    account-url: ${AWS_ECR_ACCOUNT_URL}
    repo: ${CIRCLE_PROJECT_REPONAME}
    tag: ${CIRCLE_BUILD_NUM}
    requires:
      - build_test

The above segment shows the aws-ecr/build_and_push_image: key which specifies the execution of the AWS ECR Orb. The AWS ECR Orb has parameters that require values which are assigned built-in enviroment variables in this example. See the AWS ECR Orb docs for more details on this orb. This specific AWS ERC Orb has a dependency on the build_test: job successfully completing before executing the build_and_push_image method of this orb.

Job definitions

I’ve discussed the workflows: and orbs: elements of the config which are references to jobs defined in the primary jobs: elements of this config. The following segment of code shows all of the jobs defined in this configuration syntax.

jobs:
  build_test:
    docker:
      - image: circleci/python:2.7.14
    steps:
      - checkout
      - run:
          name: Setup VirtualEnv
          command: |
            virtualenv helloworld
            . helloworld/bin/activate
            pip install --no-cache-dir -r requirements.txt
      - run:
          name: Run Tests
          command: |
            . helloworld/bin/activate
            python test_hello_world.py
  docker_hub_build_push_image:
    docker:
      - image: circleci/python:2.7.14
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: false
      - run:
          name: Build and push Docker image to Docker Hub
          command: |
            echo 'export TAG=0.1.${CIRCLE_BUILD_NUM}' >> ${BASH_ENV}
            echo 'export IMAGE_NAME=${CIRCLE_PROJECT_REPONAME}' >> ${BASH_ENV}
            source ${BASH_ENV}
            docker build -t ${DOCKER_LOGIN}/${IMAGE_NAME} -t ${DOCKER_LOGIN}/${IMAGE_NAME}:${TAG} .
            echo ${DOCKER_PWD} | docker login -u ${DOCKER_LOGIN} --password-stdin
            docker push ${DOCKER_LOGIN}/${IMAGE_NAME}

In the above segment, two jobs are defined build_test: and docker_hub_build_push-image: which demonstrates raw configuration syntax. The build_test: job instantiates a Python container, installs app dependencies, and runs the app’s unit tests.

The docker_hub_build_push_image: job sets environment variables used for naming and tagging the image, builds the Docker image based on the Dockerfile, and pushes the built image to Docker Hub.

This entire configuration demonstrates how to define and implement workflows, orbs, and jobs, which offer robust and powerful capabilities in configuration syntax. After successfully executing this pipeline you should end up with a tested Docker image in Docker Hub and your AWS ECR repository.

Summary

As you can see the jobs: segment is a bit more verbose compared to the AWS ECR Orb definition. Orbs are designed to encapsulate functionality, provide consistency and reduce the amount repeated code in pipeline configurations, therefore rendering less verbose configuration syntax.

With orbs, you can share your preferred CI/CD set-up across teams and projects, and integrate tools and third-party solutions easily with just a few lines. Orbs can be written by members of the development community to solve common problems and to help with managing configuration. Sharing and re-using orbs for common use cases frees teams to solve the interesting, unique problems that differentiate their business while offering others within the community solutions that accelerate their CI/CD workflows.

I look forward to the future development of orbs and the awesome streamlined functionality they bring to CI/CD pipelines. If you browse the orb registry and don’t find an orb that fits the functionality that you require, then I encourage you to author your own custom orb. Be aware that at the moment all orbs hosted on the CircleCI Orb Registry are open and publicly available. Private orbs are currently not supported in the registry but this feature is on the orbs road map and will be available in the future.