CI/CD systems follow a multi-tiered environments pattern: development, testing, staging, and production release are all part of this process. Each setting in this cycle could have a variety of set ups and configurations. As a result, having to set up separate configurations for different environments could be inconvenient and burdensome.
In this tutorial, we will take a look at what Docker is and how it has freed developers from set-up problems and port clashes. I will go over how to “Dockerize” a Python application. Then I will guide you through setting up a pipeline to build an image and push it to the Docker Hub after tests on it pass.
Prerequisites
The following are required to complete this tutorial:
- Python installed on your system.
- A CircleCI account.
- A GitHub account and understanding of some Github actions such as
commitandpush. - A Docker Hub account.
- A basic understanding of how pipelines work. With these in place, you can begin.
Our tutorials are platform-agnostic, but use CircleCI as an example. If you don’t have a CircleCI account, sign up for a free one here.
Why use Docker?
Imagine you are a developer testing a project. You are using Python 2.7 while you are working on the project but for some reason, you are going to use Python 3.9 in production. Also, the Devops team are currently using Python 3.6 to host applications for the organization.
These version inconsistencies are likely to make the development process somewhat painful. They are also likely to create a headache for Devops teams as they to try to keep up with the different versions of every application that needs to be hosted. Luckily, Docker offers a solution. First, let me demistify some concepts that are associated with the use of Docker.
Docker is a platform that uses containerization technology to allow developers to quickly create, share, and run applications in the state that they were created in.
A Docker container is a runtime environment for an application that consists of packaged-up code and all its dependencies. To spin up a Docker container you use Docker images.
A Docker image is an executable (template) that contains everything needed for an application to run properly, including code, dependencies, and virtual environments.
Docker images are created using a special file known as Dockerfile. A Dockerfile is a document that contains a set of instructions for Docker to follow when creating an image.
Docker eliminates the need to worry about the machine configuration because it bootstraps all that within the container itself. This ensures that the applications run consistently across all machines that have Docker set up on them.
Unlike virtual machines, using Docker results in easy management of microservice architecture development and deployment. This not only leads to lean organizations but also decoupled systems. Decoupled systems minimize the chances that a failure will occur, making them less risky than a monolithic application.
Docker promotes compatibility and maintainability across platforms, as well as simplicity, rapid deployment, and security.
Dockerizing your Python application
“Dockerizing” an application consists of these steps:
- Setting up an API to build an image and
- Creating a Dockerfile
Setting up an API to build an image
To demonstrate the Docker process in this section of the tutorial, use FastAPI RestAPI locally. The API has already been created for you. Start by cloning the repository using this command:
git clone https://github.com/CIRCLECI-GWP/dockerhub-automated-circleci-deployments.git;
cd dockerhub-automated-circleci-deployments;
This clones the GitHub repository into a directory called dockerhub-automated-circleci-deployments, and then goes to it.
Inside the dockerhub-automated-circleci-deployments directory, create a virtual environment and activate it using the commands provided for your operating system.
## Windows OS ##
# create a venv
python -m venv venv
# activate venv
venv\Scripts\activate
## Linux/Mac OS ##
#create a venv
python3 -m venv venv
# activate venv
source venv/bin/activate
Once the virtual environment is created, you can install the dependencies using this command:
pip install -r requirements.txt
Run the application:
uvicorn app.main:app --reload
Now that you have validated that your API runs successfully, go ahead and create a Dockerfile for your application.
Creating a Dockerfile
As mentioned earlier, a Dockerfile is a cookbook for creating images. Docker will read the instructions from the Dockerfile and construct an image when you run the command docker build.
In this case, you need the Docker file to:
- Download the
Python 3.10.2version - Find your packages within the application
- Perform the installation Once the installation is successful, you need the image to spin up your application within the container.
Add this to your Dockerfile:
FROM python:3.10.2
WORKDIR /usr/src/app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
In this Docker configuration, the WORKDIR command specifies the working directory, which is the absolute path to your directory. The COPY command copies the requirements.txt file to the directory where your application is located. The RUN command installs the packages that you need to run your application. Finally, CMD specifies the command that will be executed when the image is run.
You may be wondering what the host and the ports are used for in the application RUN command. Docker will use the uvicorn as the server and your application entry port. The host and the port will ensure that you can access the application outside the container.
If you wanted to build the image locally to verify that your Dockerfile is working, you can run this command:
docker build -t fastapi-app .
Next, you will need to create a new repository on GitHub, commit and then push all the changes to the repository first. Then you can start on the process of deploying your Docker containers to Docker Hub using CircleCI.
Using CircleCI to deploy a Docker image to Docker Hub
Docker Hub is a repository for Docker images, similar to GitHub. It makes it easier to search and share Docker images with other developers.
Setting up CircleCI
To ensure integration of your application to CircleCI, create a configuration file that will tell CircleCI how to initialize your repository and run tests. From the root directory, create a new directory called .circleci/. In this new directory, create a new file called config.yml. Add this to your config.yml file:
version: 2
jobs:
build:
docker:
- image: cimg/python:3.10.2
steps:
- checkout
- run:
name: Install pip packages
command: |
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
- run:
name: Test with pytest
command: |
. venv/bin/activate
pytest
workflows:
version: 2
build-master:
jobs:
- build
This configuration defines the Python 3.10.2 image. Then, once the image has been downloaded, the configuration sets up the virtual environment, installs all the application dependencies, and then runs your tests.
After adding our CircleCI configuration file, commit and push your changes to your remote GitHub repository.
Now you can create a CircleCI project from the dashboard.
Click Set Up Project to start building. You will be prompted to use the configuration file within your project’s repository. Enter the name of the branch where the configuration file is. In this case, it is the main branch.
Click Set Up Project to complete the process.
Now that you have run tests on CircleCI, your job is half done! Next, you need to deploy your application to Docker Hub. You will need to add an option to first log in to Docker Hub, then build an image each time the tests pass.
Pushing any image to Docker Hub will always require authentication. You first need to configure your Docker Hub USER_ID, PASSWORD and image name on the CirclecI dashboard. Go to the CircleCI dashboard. From the Settings section, select Environment Variables.
Add the environment variables using this format:
DOCKER_HUB_USER_IDis your Docker Hub ID.DOCKER_HUB_PASSWORDis your Docker Hub password.
Add the Docker image name as an environment variable with the environment name IMAGE_NAME. Assign it a unique name as a value.
Currently, your CircleCI configuration only runs your tests. You will need to modify it so that it first runs the tests, then builds a Docker image and pushes it to Docker Hub if all the tests pass.
Add a deploy job to your config.yml file like this:
version: 2
jobs:
build:
docker:
- image: cimg/python:3.10.2
steps:
- checkout
- run:
name: Install pip packages
command: |
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
- run:
name: Test with pytest
command: |
. venv/bin/activate
pytest
deploy:
docker:
- image: cimg/base:2022.06
steps:
- checkout
- setup_remote_docker:
version: 19.03.13
- run:
name: Build and push to Docker Hub
command: |
docker build -t $DOCKER_HUB_USER_ID/$IMAGE_NAME:latest .
echo "$DOCKER_HUB_PASSWORD" | docker login -u "$DOCKER_HUB_USER_ID" --password-stdin
docker push $DOCKER_HUB_USER_ID/$IMAGE_NAME:latest
workflows:
version: 2
build-master:
jobs:
- build
- deploy:
requires:
- build
This configuration file contains a workflow with two jobs:
- The
buildjob builds and tests the code. - The
deployjob builds and pushes your Docker image to Docker Hub.
In the added configuration deploy step, the deploy job retrieves the code before launching a remote Docker engine. When creating Docker images for deployment, you must use the setup_remote_docker option. This option creates a separate and remote environment for each build, for security purposes. This environment is specifically set up to run Docker commands.
Once that’s done, you can start running Docker commands for building and tagging your image:
docker build -t $DOCKER_HUB_USER_ID/$IMAGE_NAME:latest .
This command builds a Docker image and tags it with the :latest tag. The image name is prefixed with the Docker Hub username you set up as an environment variable in the CircleCI dashboard, followed by the image’s actual name, (also an environment variable you set).
Once you have a built image, sign into your Docker Hub account using CircleCI and the stored environment variables.
echo "$DOCKER_HUB_PASSWORD" | docker login -u "$DOCKER_HUB_USER_ID" --password-stdin
This is what the command in the configuration file does. Now that you have an image and are logged into Docker Hub, your image is pushed to Docker Hub using this command defined in the configuration file:
docker push $DOCKER_HUB_USER_ID/$IMAGE_NAME:latest
To ensure the integrity of your pipeline, there is a condition that requires your build and test job to pass before deploying to Docker Hub. This happens in the final configuration block under the workflow:
workflows:
version: 2
build-master:
jobs:
- build
- deploy:
requires:
- build
Finally, we need to commit and push all of our changes to GitHub. Once we do this, we can go ahead and check the CircleCI dashboard to see the progress of our tests and deployment.
And Voila! both our deploy and build steps are green and checking the deploy job, our image was built successfully and pushed to Docker hub.
To verify that our image was pushed to Docker Hub, we can go to the Docker Hub website and check the image’s status. In our case, our image is named circleci-automated-dockerhub-image and we can see it on the Docker hub dashboard as below:
With this, we have not only verified that our CircleCI configuration works, but also verified that our image was successfully pushed to Docker Hub. While it feels like a lot, we have been able to create a CI/CD process using CircleCI and Docker Hub saving ourselves lots of time that would otherwise have been spent on manual Docker Hub deployment.
Conclusion
In this tutorial, we have learned how we can use CirclecI to build and deploy Docker images to Docker Hub. We have also learned how we can build a Docker image using a defined Dockerfile in our project.
In the final section of the tutorial we also covered how to create a pipeline that builds an image and pushes it to Docker Hub only when our tests pass. I hope you enjoyed reading this tutorial as we did creating it. Until the next one, stay sharp, and keep learning!
Waweru Mwaura is a software engineer and a life-long learner who specializes in quality engineering. He is an author at Packt and enjoys reading about engineering, finance, and technology. You can read more about him on his web profile.
