Running builds in continuous integration/continuous deployment (CI/CD) pipelines is a great way to automate your repetitive deployment and testing tasks. However, if you have tons of tests, build steps, or other slow setup tasks, it can heavily bog down your build and drive the feedback duration through the roof! Long feedback cycles are friends to no one and slow down development lifecycle, so the next logical step is to find ways to reduce the duration wherever possible. Some of us may run tests within our build steps in parallel or perhaps take unnecessary shortcuts in pursuit of shaving some time, though these solutions may not always work optimally in the end. Let’s take a look at how we can harness the power of CircleCI 2.0’s workflows and how running jobs in parallel can unlock a whole new world of speed for our teams and us.
Paralelli-wha?
For those who may be unfamiliar, most common build steps, like tests, are run sequentially: that is, each step runs on its own and only happens after the previous one. If we broke our long, single build process into different steps and ran them all at the same time, those steps would be said to run in parallel or concurrently. A build that has five steps that each takes one minute to run would require five minutes to complete when run sequentially. If we were to take each of those five steps and run them all at the same time, our build would only need one minute to run! That is a gigantic increase compared to running sequentially.
To help illustrate the difference between what sequential and parallelism look like and the time they would take, here is what a build’s timeline could look like:
Sequential
0:00------1:00------2:00------3:00------4:00------5:00
Step1-----Step2-----Step3-----Step4-----Step5-----
Parallel
0:00------1:00
Step1-----
Step2-----
Step3-----
Step4-----
Step5-----
Much faster!
Sequential beginnings
Before we get into the wonderful world of workflows, let’s take a look at a demo project that has some simulated common build steps. We will be spending most of our time working with our CircleCI 2.0 config.yml and seeing how things behave on the dashboard. We will start by giving the config a review:
version: 2
jobs:
build:
docker:
- image: circleci/ruby:2.4.1
working_directory: ~/repo
steps:
- checkout
- restore_cache:
keys:
- v1-dependencies-{{ checksum "Gemfile.lock" }}
- v1-dependencies-
- run:
name: Dependencies
command: |
bundle install --jobs=4 --retry=3 --path vendor/bundle
- save_cache:
name: Save Cache
paths:
- ./vendor/bundle
key: v1-dependencies-{{ checksum "Gemfile.lock" }}
- run:
name: Stage Deploy
command: make build_stage
- run:
name: Database
command: |
bundle exec rake simulated_db_create
bundle exec rake simulated_db_load
- run:
name: RSpec
command: |
bundle exec rake simulated_rspec
- run:
name: Cucumber Non-integrated
command: |
bundle exec rake cucumber
- run:
name: Selenium UI Checks
command: |
bundle exec rake simulated_selenium
- run:
name: Deploy Prod
command: |
bundle exec rake simulated_deploy
Our build job above is responsible for everything we need to do: managing our gem dependencies, deploying to our staging environment, spinning up a local database for unit tests/non-integrated tests, running those tests, running Selenium UI checks, and if all pass, deploying to production. It is a very straightforward set up that makes sure we have dotted our i’s and crossed our t’s before we deploy, though it is sequential and therefore slow.
Times like these
Our job above clocks in at 6:43 and while that is not horrible, we want the fastest feedback we can, and we are here to learn about parallelism, so let’s get to it!
Our longest running steps are Selenium and Cucumber, taking 3:00 and 2:04 respectively. In Ruby, there is a parallel_tests gem that we can use to run our Cucumber tests in parallel which should be an easy fix for that two minutes we are waiting.
What an improvement in time! That small change of adding in the parallelization has dropped our overall time to 5:23! The downside is, though, that when we look at the console output during our tests, it looks like this:
Where our build output was once neat, easily followable, and could be helpful for debugging, it is now a jumbled mess that is not useful aside from seeing that something is happening. The examples section should only contain strings, integers, or floats for the particular scenarios actively running, but since all scenarios are running at the same time, their output is being displayed at the same time, too. This undesirable outcome is a prime example of how some parallelization routes can decrease our run times at the expense of ease of use.
To contrast, here is what the output looks like without running Cucumber with the parallel_tests gem:
Our 3:00 Selenium job still accounts for the vast majority of our build time, and we will handle that ahead with our refactor.
The parallel zone
If jobs are a collection of steps, then workflows can be thought of as a collection of jobs. Similar to our example of sequential vs. parallel, let’s take a look at a config only using a single build job compared to one using workflows. We will use our example from earlier of five jobs that each take one minute to run and make it a real setup.
version: 2
jobs:
build:
docker:
- image: circleci/ruby:2.4
steps:
- run: sleep(60) && echo 1
- run: sleep(60) && echo 2
- run: sleep(60) && echo 3
- run: sleep(60) && echo 4
- run: sleep(60) && echo 5
Here is what things look like in the dashboard:
As noted above, a workflow is a collection of jobs, so here is what our project looks like refactored to utilize the parallelization of workflows:
version: 2
jobs:
one:
docker:
- image: circleci/ruby:2.4
steps:
- echo 1
two:
docker:
- image: circleci/ruby:2.4
steps:
- echo 2
three:
docker:
- image: circleci/ruby:2.4
steps:
- echo 3
four:
docker:
- image: circleci/ruby:2.4
steps:
- echo 4
five:
docker:
- image: circleci/ruby:2.4
steps:
- echo 5
workflows:
version: 2
build:
jobs:
- one
- two
- three
- four
- five
While the above refactor is not optimized for DRYness, it shows a very basic refactoring to the config to use workflows. Each of the original steps moved into individual jobs in the upper section, and then in the lower section, we have told CircleCI that we want them to run as a workflow collection.
So what does this get us? It takes us from our sequential build steps clocking in at 5:02 down to a total build time of 1:03!
This looks pretty similar to our sequential jobs, so what is the difference? This is how CircleCI displays jobs within workflows. While it looks similar, each of these is an individual job with steps, environments, and so on, that were all run at the same time. Now that we have completed our crash course in workflows, let’s take a look at a more elaborate workflow setup and get back to our example.
Bringing it all together
Let’s see what the awesome power of workflows can do for our project together. As a brief recap, our initial project ran everything sequentially and did nothing to try to run subtests in parallel. The builds of that version took about 10 minutes.
Running our jobs in parallel via workflows slashes our build times to around 1:50!
From 10 minutes to under 2 with some refactoring to take advantage of parallel jobs! We didn’t even run all jobs in parallel, either! Here is our project with its newly refactored parallel jobs in a workflow:
“Woah” ~ Keanu Reeves
Each one of the entries above is a job, and each job has a collection of steps.
The workflow above illustrates a handful of different CircleCI concepts:
- Shared resources - Reusing dependencies/data across multiple jobs to save time.
- Gating jobs behind others - Requiring that other jobs finish before others ensures that our dependencies have downloaded or other requirements have completed their configurations.
- Small, standalone jobs - Breaking out our Selenium and Cucumber tests into smaller chunks allows us to get the power of parallelism, while still getting the useful data we need, allowing as many jobs to pass as possible in the case of failures, and also allowing our reruns to be faster using workflow’s handy “Rerun from failed” option.
Now let’s take a look at our refactored config file. Note that this has been abbreviated. You can find the entire config in a gist file here.
aliases:
- &restore_gem_cache
... # repetitive data from the previous config has been omitted for length
- &save_gem_cache
...
- &bundle_install
...
- &attach_workspace
attach_workspace:
at: ~/data
- &dependencies
- checkout
- *attach_workspace
- restore_cache: *restore_gem_cache
- run: *bundle_install
- save_cache: *save_gem_cache
- persist_to_workspace:
root: .
paths:
- vendor/bundle
- &db_setup
...
- &database
- checkout
- *attach_workspace
- run: *bundle_install
- run: *db_setup
- &run_rspec
...
- &rspec_steps
- checkout
- *attach_workspace
- run: *bundle_install
- run: *run_rspec
- &cucumber
name: Cucumber
command: |
echo "cucumber feature - ${FEATURE}"
bundle exec cucumber ${FEATURE}
- &selenium
...
- &selenium_steps
- checkout
- *attach_workspace
- run: *bundle_install
- run: *selenium
- &cucumber_steps
- checkout
- *attach_workspace
- run: *bundle_install
- run: *cucumber
- &stage_deploy
...
- &deploy_stage
- checkout
- run: apk add --update make
- run: *stage_deploy
- &prod_deploy
...
- &deploy_prod
- checkout
- *attach_workspace
- run: *bundle_install
- run: *prod_deploy
- &deps
- "Dependencies"
- &deps_and_deploy
- "Dependencies"
- "Stage Deploy"
- &deps_and_db
- "Dependencies"
- "Database"
- &all_test
- "RSpec"
- "Cucumber Number Addition"
...
- "Selenium Firefox"
...
- &deploy_environment
working_directory: ~/data
docker:
- image: alpine:latest
- &test_environment
working_directory: ~/data
docker:
- image: ruby:2.4
version: 2
workflows:
version: 2
master:
jobs:
- "Stage Deploy"
- "Dependencies"
- "Database":
requires: *deps
- "RSpec":
requires: *deps_and_db
- "Cucumber Number Addition":
requires: *deps_and_db
...
- "Selenium Firefox":
requires: *deps_and_deploy
...
- "Deploy Prod":
requires: *all_test
jobs:
"Stage Deploy":
<<: *deploy_environment
steps: *deploy_stage
"Dependencies":
<<: *test_environment # All following jobs will use this environment, however its been excluded for length
steps: *dependencies
"Database":
steps: *database
"RSpec":
steps: *rspec_steps
"Cucumber Number Addition":
environment:
FEATURE: features/number_addition.feature
steps: *cucumber_steps
...
"Selenium Firefox":
environment:
platform: firefox
version: latest
steps: *selenium_steps
...
"Deploy Prod":
steps: *deploy_prod
There is a lot going on in our config file now, so let’s go through it all so that we are comfortable. Some are YAML features, and others are CircleCI & workflow features.
YAML
Aliases
Aliases are used throughout the config as variables or pointers to information that we want to prevent ourselves from typing over and over. They are all declared in the upper section of our config led by an ampersand(&) and referenced by using an asterisk(*). One could also use the features of CircleCI 2.1 to write reuseable executors instead of using YAML aliases.
Hash merging Hash merging is used to combine two collections of key-value data in our config. This technique is used above to add in our desired Docker container.
CircleCI & workflow features
Persisting data across jobs Workflows have a workspace that can be used to store data to be used by jobs that come later on. In our example, we are persisting our dependencies so that we do not have to continue downloading them. Persisting data coupled with caching makes dependency handling fast!
Gating As mentioned above, we can gate jobs behind ones that manage our dependencies. We are also using gating to make sure that all our tests past before deploying to production. If nothing is gated, it all runs at the same time.
Environment variables We use assigning environment variables to be used in commands across both the Cucumber and Selenium jobs to pass in exactly what we want to run for each job. Instead of having a jumbled mess in the Cucumber tests, they are each run on their own, and the output is pristine.
Similarly, we have refactored our one single Selenium job into multiples that are focused on specific platform combinations to test. This way, too, if one platform fails, the others will continue without issue.
What a trip!
We started in Sequentialville and ended up in Parallel Paradise all thanks to CircleCI’s workflows and parallel jobs. Our builds now run nearly 74% faster, and we are able to reap the benefits of faster feedback because of it. Less waiting and more accomplishing! Make your incremental steps smaller and enable yourself to more safely undo by being able to run your builds much more frequently!
How much can your team can slash their build times by running your jobs in parallel?
Gemini Smith is a continuously learning software engineer with a passion for testing, communication, and improving the software development lifecycle. Primarily writing in Go, she also contributes to the software and testing communities via public speaking, advocacy, consulting, and mentorship.
