I recently gave a talk at DockerCon and it was focused on showing the audience how to implement a continuous integration/continuous deployment (CI/CD) pipeline into their codebase with very little effort. In this post I’ll walk through the demo code and CircleCI config that I used in my talk and demonstrate how to implement CI/CD pipelines into your code base.
This post will cover:
- A simple unit test for a Python flask application
- How to implement a CI/CD pipeline in the codebase using a CircleCI config file in the project
- Building a Docker image
- Pushing the Docker image to Docker Hub
- Kicking off a deployment script which will run the application in Docker container on a Digital Ocean server
Prerequisites
Before we get started, you’ll need:
- A Docker Hub account
- Set the project environment variables which specify your Docker Hub username and password in the CircleCI dashboard.
- SSH access to a cloud server. You can add a SSH key to your account via the CircleCI portal. For this post I’ll be using a Digital Ocean server but you can use whichever server/cloud provider you desire.
- You’ll also need to create a deployment script on the host server that will be used to deploy this application here is an example deployment script deploy_app.sh.
After you have all the pre-requisites complete you’re ready to proceed to the next section.
The app
For this post I’ll be using a simple Python Flask and you can find the complete source code for this project here and git clone
it locally. The app is a simple web server that renders html when a request is made to it. The Flask application lives in the hello_world.py
file:
from flask import Flask
app = Flask(__name__)
def wrap_html(message):
html = """
<html>
<body>
<div style='font-size:120px;'>
<center>
<image height="200" width="800" src="https://infosiftr.com/wp-content/uploads/2018/01/unnamed-2.png">
<br>
{0}<br>
</center>
</div>
</body>
</html>""".format(message)
return html
@app.route('/')
def hello_world():
message = 'Hello DockerCon 2018!'
html = wrap_html(message)
return html
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000)
The key takeaway in this code the message
variable within the hello_world()
function. This variable specifies a string value, and the value of this variable will be tested for a match in a unit test.
Testing code
All code must be tested to ensure that high quality, stable code is being released to the public. Python comes with a testing framework named unittest and I’ll be using that for this post. We now have a complete Flask application and it needs a companion unit test that will test the application and ensure it’s functioning as designed. The unit test file test_hello_world.py
is the unit test for our hello_world.py app. Now, let’s walk through the code.
import hello_world
import unittest
class TestHelloWorld(unittest.TestCase):
def setUp(self):
self.app = hello_world.app.test_client()
self.app.testing = True
def test_status_code(self):
response = self.app.get('/')
self.assertEqual(response.status_code, 200)
def test_message(self):
response = self.app.get('/')
message = hello_world.wrap_html('Hello DockerCon 2018!')
self.assertEqual(response.data, message)
if __name__ == '__main__':
unittest.main()
import hello_world
import unittest
Import the hello_world
application using the import
statement which gives the test access to the code in the hello_world.py
. Next import the unittest
modules and start defining test coverage for the application.
class TestHelloWorld(unittest.TestCase):
The TestHelloWorld is instantiated from the base class unittest.Test
which is the smallest unit of testing. It checks for a specific response to a particular set of inputs. unittest provides a base class, TestCase, which may be used to create new test cases.
def setUp(self):
self.app = hello_world.app.test_client()
self.app.testing = True
setUp()
is a class level method called to prepare the test fixture. This is called immediately before calling the test method. In this example, we create and define a variable named app
and instantiate it as app.test_client()
object from the hello_world.py code.
def test_status_code(self):
response = self.app.get('/')
self.assertEqual(response.status_code, 200)
test_status_code()
is a method and it specifies an actual test case in code. This test case makes a get
request to the flask application and captures the app’s response in the response
variable. The self.assertEqual(response.status_code, 200)
compares the value of the response.status_code
result to the expected value of 200
which signifies the get
request was successful. If the server responds with a status_code other than 200, the test will fail.
def test_message(self):
response = self.app.get('/')
message = hello_world.wrap_html('Hello DockerCon 2018!')
self.assertEqual(response.data, message)
test_message()
is another method that specifies a different test case. This test case is designed to check the value of the message
variable that is defined in the hello_world()
method from the hello_world.py code. Like the previous test a get call is made to the app and the results are captured in a response
variable. The following line :
message = hello_world.wrap_html('Hello DockerCon 2018!')
The message
variable is assigned the resulting html from the hello_world.wrap_html()
helper method which is defined in the hello_world app. The string Hello DockerCon 2018
is supplied to the wrap_html()
method which is then injected & returned in html. The test_message()
will verify that the message variable in the app will match the expected string in this test case. If the strings don’t match, then the test will fail.
CI/CD pipelines
Now that we’re clear on the application and its unit tests, it is time to implement a CI/CD pipeline into the codebase. Implementing a CI/CD pipeline using CircleCI is very simple. Before continuing, make sure you do the following:
Implementing a CI/CD pipeline
Once your project is set up in the CircleCI platform, any commits pushed upstream will be detected and CircleCI will execute the job defined in your config.yml
file.
You will need to create a new directory in the repo’s root and a yaml file within this new directory. The new assets must follow these naming schema - directory: .circleci/
file: config.yml
in your project’s git repository. This directory and file basically define your CI/CD pipeline adn configuration for the CircleCI platform.
Configuration files
The config.yml is where all of the CI/CD magic happens. Below is an example of the file used in the example file and I’ll briefly explain what’s going on within the syntax:
version: 2
jobs:
build:
docker:
- image: circleci/python:2.7.14
environment:
FLASK_CONFIG: testing
steps:
- checkout
- run:
name: Setup VirtualEnv
command: |
echo 'export TAG=0.1.${CIRCLE_BUILD_NUM}' >> $BASH_ENV
echo 'export IMAGE_NAME=python-circleci-docker' >> $BASH_ENV
virtualenv helloworld
. helloworld/bin/activate
pip install --no-cache-dir -r requirements.txt
- run:
name: Run Tests
command: |
. helloworld/bin/activate
python test_hello_world.py
- setup_remote_docker:
docker_layer_caching: true
- run:
name: Build and push Docker image
command: |
. helloworld/bin/activate
pyinstaller -F hello_world.py
docker build -t ariv3ra/$IMAGE_NAME:$TAG .
echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
docker push ariv3ra/$IMAGE_NAME:$TAG
- run:
name: Deploy app to Digital Ocean Server via Docker
command: |
ssh -o StrictHostKeyChecking=no root@hello.dpunks.org "/bin/bash ./deploy_app.sh ariv3ra/$IMAGE_NAME:$TAG"
The jobs:
key represents a list of jobs that will be run. A job encapsulates the actions to be executed. If you only have one job to run then you must give it a key name build:
you can get more details about jobs and builds here.
The build:
key is composed of a few elements:
docker:
steps:
The docker:
key tells CircleCI to use a Docker executor, which means our build will be executed using Docker containers.
image: circleci/python:2.7.14
specifies the Docker image that the build must use.
steps:
The steps:
key is a collection that specifies all of the commands that will be executed in this build. The first action that happens the - checkout
command that basically performs a git clone of your code into the build environment.
The - run:
keys specify commands to execute within the build. Run keys have a name:
parameter where you can label a grouping of commands. For example name: Run Tests
groups the test related actions which helps organize and display build data within the CircleCI dashboard.
Note: Each run
block is equivalent to separate/individual shells or terminals. Commands that are configured or executed will not persist in later run blocks. Use the $BASH_ENV
workaround in the tips & tricks section of the docs.
- run:
name: Setup VirtualEnv
command: |
echo 'export TAG=0.1.${CIRCLE_BUILD_NUM}' >> $BASH_ENV
echo 'export IMAGE_NAME=python-circleci-docker' >> $BASH_ENV
virtualenv helloworld
. helloworld/bin/activate
pip install --no-cache-dir -r requirements.txt
The command:
key for this run block has a list of commands to execute. These commands set the $TAG
& IMAGE_NAME
custom environment variables that will be used throughout this build. The remaining commands set up the python virtualenv & installs the Python dependencies specified in the requirements.txt
file.
- run:
name: Run Tests
command: |
. helloworld/bin/activate
python test_hello_world.py
In this run block, the command executes tests on our application and if these tests fail the entire build will fail and will require the developers to fix their code and recommit.
- setup_remote_docker:
docker_layer_caching: true
This run block specifies the setup_remote_docker: key which is a feature that enables building, running and pushing images to Docker registries from within a Docker executor job. When docker_layer_caching is set to true, CircleCI will try to reuse Docker Images (layers) built during a previous job or workflow. That is, every layer you built in a previous job will be accessible in the remote environment. However, in some cases your job may run in a clean environment, even if the configuration specifies docker_layer_caching: true.
Since we’re building a Docker image for our app and pushing that image to Docker Hub the setup_remote_docker:
is required.
- run:
name: Build and push Docker image
command: |
. helloworld/bin/activate
pyinstaller -F hello_world.py
docker build -t ariv3ra/$IMAGE_NAME:$TAG .
echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
docker push ariv3ra/$IMAGE_NAME:$TAG
The Build and push Docker image run block specifies the commands that package the application into a single binary using pyinstaller then continues on to the Docker image building process.
docker build -t ariv3ra/$IMAGE_NAME:$TAG .
echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
docker push ariv3ra/$IMAGE_NAME:$TAG
These commands build the docker image based on the Dockerfile
included in the repo. Dockerfile is the instruction on how to build the Docker image.
FROM python:2.7.14
RUN mkdir /opt/hello_word/
WORKDIR /opt/hello_word/
COPY requirements.txt .
COPY dist/hello_world /opt/hello_word/
EXPOSE 80
CMD [ "./hello_world" ]
The echo $DOCKER_PWD | docker login -u $DOCKER_LOGIN --password-stdin
command uses the $DOCKER_LOGIN and $DOCKER_PWD env variables set in the CircleCI dashboard as credentials to login & push this image to Docker Hub.
- run:
name: Deploy app to Digital Ocean Server via Docker
command: |
ssh -o StrictHostKeyChecking=no root@hello.dpunks.org "/bin/bash ./deploy_app.sh ariv3ra/$IMAGE_NAME:$TAG"
The final run bock deploys our new code to a live server running on the Digital Ocean platform. Make sure that you’ve created a deploy script on the remote server. The ssh command access the remote server and executes the deploy_app.sh
script on the the server and specifies: ariv3ra/$IMAGE_NAME:$TAG which specifies the image to pull & deploy from Docker Hub.
After the job successfully completes, the new application should be running on the target server you specified in you config.yml.
Summary
In review this post should guide you through implementing a CI/CD pipeline into your code. Though this example is built using python technologies the general build, test and deployment concepts can easily be implemented in whatever language or framework you desire. The examples in this post are also simple but you can expand on them and tailor them to your pipelines. CircleCI has great documentation so don’t hesitate to research our docs site and if you really get stuck you can also reach out to the CircleCI community via the https://discuss.circleci.com/ community/forum site.
Read more: