I’ve been using CircleCI version 1.0 to build, test and upload Scala application packages to S3 for over 2 years. The CircleCI 1.0 template for the Scala apps are easy to understand and implement quickly which was a huge benefit over the older CI/CD solutions I was using. In July 2017, CircleCI 2.0 was released.

CircleCI 2.0 features dramatically reduced build times and gives users more flexibility regarding the build environments than 1.0. Having said all this, there is a cost in upgrading, in that you need to migrate your 1.0 configuration file to the new 2.0 schema.

In this post I’ll walk you through how I migrated a Scala application from 1.0 to 2.0. Below is my CircleCI 1.0 yaml file for a Scala-based app named samplescala. The source code for the samplescala app can be found in this github repo. This post assumes that your project’s AWS Permission settings are configured with valid AWS keys that are permitted to read and write to an S3 bucket. The examples in this post upload build packages to the specified S3 bucket.

CircleCI 1.0

File: circle.yml

machine:
  environment:
    ARTIFACT_BUILD: $CIRCLE_PROJECT_REPONAME-$CIRCLE_BUILD_NUM.zip
dependencies:
  cache_directories:
    - "~/.ivy2/cache"
    - "~/.sbt"
    - "~/.m2"
test:
    override:
    - echo "No tests to run"
deployment:
    override:
        - sbt update dist
    commands:
        - mv target/universal/samplescala.zip $CIRCLE_ARTIFACTS/$ARTIFACT_BUILD
        - aws s3 cp $CIRCLE_ARTIFACTS/$ARTIFACT_BUILD s3://samplescala.blogs/builds/ --metadata {\"git_sha1\":\"$CIRCLE_SHA1\"}

This 1.0 circle.yml file does the following:

  • ARTIFACT_BUILD: Specifies an environment variable that represents a file name for the final package. The filename is composed of the CircleCI system-related Environment Variables which are used later in the process.
  • dependencies/cache_directories: specifies the cached sbt, maven and ivy directories which helps in speeding up the build process.
  • test: for this example we won’t run a test, but if you have tests to run, you would execute your test command in the test/override section.
  • deployment/override: The sbt update dist command actually builds the samplescala package and produces an application distribution file named samplescala.zip in the target/universal/ directory.
  • deployment/commands: these commands perform 2 functions
    1. Renames & moves the samplescala.zip file to the $CIRCLE_ARTIFACTS/ directory with a new name composed of $ARTIFACT_BUILD variable.
    2. The final command uploads the package file located in $CIRCLE_ARTIFACTS/$ARTIFACT_BUILD to the AWS S3 Bucket s3://samplescala.blogs/builds/

Once the package file exists in the S3 bucket, it can be used in the deployment process.

Migrate samplescala to CircleCI Version 2.0

Now we’re going to walk through migrating the samplescala configuration file to 2.0.

Before we get started, you need to create new 2.0-related assets in your project’s git repository. CircleCI 2.0 requires you to create a new directory in the repo’s root and a yaml file within this new directory. The new assets must follow these naming schema’s directory: .circleci/ file: config.yml.

mkdir .circleci/
touch .circleci/config.yml

These commands create a directory named .circleci & the next command creates a new file named config.yml within the .circleci directory. Again you must use the names .circleci for the dir and config.yml. Learn more about the version 2.0 prerequisites here.

config.yml file

Open the newly created config.yml in your favorite text editor and paste the following CircleCI 2.0 schema into the file. Below is the complete 2.0 conversion of our 1.0 schema:

version: 2
jobs:
  build:
    working_directory: ~/samplescala
    docker:
      - image: openjdk:8
    environment:
      - SBT_VERSION: 1.0.4
    steps:
      - run: echo 'export ARTIFACT_BUILD=$CIRCLE_PROJECT_REPONAME-$CIRCLE_BUILD_NUM.zip' >> $BASH_ENV
      - run:
          name: Get sbt binary
          command: |
                    apt update && apt install -y curl
                    curl -L -o sbt-$SBT_VERSION.deb https://dl.bintray.com/sbt/debian/sbt-$SBT_VERSION.deb
                    dpkg -i sbt-$SBT_VERSION.deb
                    rm sbt-$SBT_VERSION.deb
                    apt-get update
                    apt-get install -y sbt python-pip git
                    pip install awscli
                    apt-get clean && apt-get autoclean
      - checkout
      - restore_cache:
          key: sbt-cache
      - run:
          name: Compile samplescala dist package
          command: cat /dev/null | sbt clean update dist
      - store_artifacts:
          path: target/universal/samplescala.zip
          destination: samplescala
      - save_cache:
          key: sbt-cache
          paths:
            - "~/.ivy2/cache"
            - "~/.sbt"
            - "~/.m2"
      - deploy:
          command: |
              mv target/universal/samplescala.zip $CIRCLE_ARTIFACTS/$ARTIFACT_BUILD
              aws s3 cp $CIRCLE_ARTIFACTS/$ARTIFACT_BUILD s3://samplescala.blogs/builds/ --metadata {\"git_sha1\":\"$CIRCLE_SHA1\"}

CircleCI 2.0 provides loads of benefits, but there is a small learning curve. As you can see from the 2.0 schema file there are obvious schema changes in 2.0. We’ll briefly cover the new config.yml schema and explain the changes.

All version 2.0 config.yml files must start with his line:

version: 2

The next key in the schema is the jobs & build keys. These keys are required and represent the default entry point for a run.

version 2

jobs:
    build:

The build section hosts the remainder of the schema which executes our commands. This will be explained below.

  build:
    working_directory: ~/samplescala
    docker:
      - image: openjdk:8
    environment:
      - SBT_VERSION: 1.0.4

The docker/image key represents the Docker image you want to use for the build. In this case, we want to use the official openjdk:8 image from Docker Hub because it has the native Java compiler we need for our Scala project.

The environment/SBT_VERSION is an environment variable that specifies the version of sbt to download in later commands which is required to compile the Scala app.

    environment:
      - SBT_VERSION: 1.0.4
    steps:
      - run: echo 'export ARTIFACT_BUILD=$CIRCLE_PROJECT_REPONAME-$CIRCLE_BUILD_NUM.zip' >> $BASH_ENV
      - run:
          name: Get sbt binary
          command: |
                    apt update && apt install -y curl
                    curl -L -o sbt-$SBT_VERSION.deb https://dl.bintray.com/sbt/debian/sbt-$SBT_VERSION.deb
                    dpkg -i sbt-$SBT_VERSION.deb
                    rm sbt-$SBT_VERSION.deb
                    apt-get update
                    apt-get install -y sbt python-pip git
                    pip install awscli
                    apt-get clean && apt-get autoclean

The steps/run keys specify the types of actions to perform. The run keys represent the actions to be executed.

- run: echo 'export ARTIFACT_BUILD=$CIRCLE_PROJECT_REPONAME-$CIRCLE_BUILD_NUM.zip' >> $BASH_ENV

This echo command defines the $ARTIFACT_BUILD environment variable and sets it to a build filename. This environment variable is equivalent to the environment section of the 1.0 schema and is used later in the schema.

Version 1.0 Schema
  environment:
    ARTIFACT_BUILD: $CIRCLE_PROJECT_REPONAME-$CIRCLE_BUILD_NUM.zip

The next run command executes multiple commands within the openjdk container. Since we’re executing multiple commands we’ll be defining a multi-line run/command which is designated by the pipe | character, as shown below. When using the multi-line option, one line represents one command.

          command: |
                    apt update && apt install -y curl
                    curl -L -o sbt-$SBT_VERSION.deb https://dl.bintray.com/sbt/debian/sbt-$SBT_VERSION.deb
                    dpkg -i sbt-$SBT_VERSION.deb
                    rm sbt-$SBT_VERSION.deb
                    apt-get update
                    apt-get install -y sbt python-pip git
                    pip install awscli
                    apt-get clean && apt-get autoclean

The 2.0 version of our samplescala schema requires us to download required dependencies and install them into the container. Below is an explanation of the example multi-line command:

  • Updates the container OS and installs curl.
  • Downloads the Simple Build Tool (sbt) compiler version specified in the $SBT_VERSION variable.
  • Installs the sbt compiler package.
  • Deletes the sbt.deb file after install.
  • Updates the OS package listing.
  • Installs python-pip and git client.
  • Installs the awscli package which is the AWS Command Line Interface needed to perform the S3 uploads.
  • Removes all the unnecessary install packages to minimize container size.

The following keys represent actions performed after the multi-line command is executed:

      - checkout
      - restore_cache:
          key: sbt-cache
      - run:
          name: Compile samplescala dist package
          command: cat /dev/null | sbt clean update dist
      - store_artifacts:
          path: target/universal/samplescala.zip
          destination: samplescala
      - save_cache:
          key: sbt-cache
          paths:
            - "~/.ivy2/cache"
            - "~/.sbt"
            - "~/.m2"

Below is an explanation of the preceding example:

  • checkout: basically git clones the project repo from github into the container
  • restore_cache/key: specifies the name of the cache files to restore. The key name is specified in the save_cache key that is found later in the schema. If the key specified is not found then nothing is restored and continues to process.
  • run/command cat /dev/null | sbt clean update dist: executes the sbt compile command that generates the package .zip file.
  • store_artifacts/path: specifies the path to the source file to copy to the ARTIFACT zone in the image.
  • save_cache/path: saves the specified directories for use in future builds when specified in the restore_cache keys.

The final portion of the 2.0 schema are the deploy/command keys which move and rename the compiled samplescala.zip to the $CIRCLE_ARTIFACTS/ directory. The file is then uploaded to the AWS S3 bucket specified.

  - deploy:
      command: |
        mv target/universal/samplescala.zip $CIRCLE_ARTIFACTS/$ARTIFACT_BUILD
        aws s3 cp $CIRCLE_ARTIFACTS/$ARTIFACT_BUILD s3://samplescala.blogs/builds/ --metadata {\"git_sha1\":\"$CIRCLE_SHA1\"}

The deploy/command is another multi-line execution and is the equivalent to the deployment/commands section from the 1.0 schema shown below:

deployment:
    override:
        - sbt update dist
    commands:
        - mv target/universal/samplescala.zip $CIRCLE_ARTIFACTS/$ARTIFACT_BUILD
        - aws s3 cp $CIRCLE_ARTIFACTS/$ARTIFACT_BUILD s3://samplescala.blogs/builds/ --metadata {\"git_sha1\":\"$CIRCLE_SHA1\"}

The CircleCI 2.0 Scala application migration requires some schema adjustments and execution tweaks, but the gains are really worth the migration efforts. As I previously mentioned, there is a small learning curve in familiarization with the 2.0 schema so running through the 1.0 > 2.0 Migration docs is highly recommended.