Most CI/CD systems follow a multi-tiered environments pattern. Development, testing, staging, and production release are all part of this process. Each setting in this cycle could have a variety of setups and configurations. As a result, having to set up separate configurations for different environments can be inconvenient, burdensome, and risky.
In this tutorial, you will learn how Docker frees developers from setup problems and port clashes. First, you will learn how to “Dockerize” a sample Python application using a custom Dockerfile. Then you will create a CI/CD pipeline to automatically build and test your Docker image every time you update the underlying code. If all tests pass, your pipeline will automatically push your Dockerized Python application to Docker Hub.
Prerequisites
The following are required to complete this tutorial:
- Python installed on your system.
- A CircleCI account.
- A GitHub account and understanding of some Github actions such as
commitandpush. - A Docker Hub account.
-
A basic understanding of how pipelines work.
With these in place, you can begin.
Why use Docker?
Imagine you are a developer testing a project. You are using Python 2.7 while you are working on the project. For some reason, you are going to use Python 3.9 in production. Also, the Devops team are currently using Python 3.6 to host applications for the organization.
These version inconsistencies are likely to make the development process somewhat painful. They are also likely to create a headache for DevOps teams as they to try to keep up with the different versions of every application that needs to be hosted. Luckily, Docker offers a solution. First, let me demystify some concepts about Docker.
Docker is a platform that uses containerization technology to allow developers to quickly create, share, and run applications in the state that they were created in.
A Docker container is a runtime environment for an application that consists of packaged-up code and all its dependencies. To spin up a Docker container you use Docker images.
A Docker image is an executable (template) that contains everything needed for an application to run properly, including code, dependencies, and virtual environments.
Docker images are created using a special file known as Dockerfile. A Dockerfile is a document that contains a set of instructions for Docker to follow when creating an image.
Docker eliminates the need to worry about the machine configuration because it bootstraps it within the container itself. This ensures that the applications run consistently across all machines that have Docker set up on them.
Unlike virtual machines, Docker containers provide easy management of microservice architecture development and deployment. This not only leads to lean organizations but also decoupled systems. Decoupled systems minimize the chances that a failure will occur, making them less risky than a monolithic application.
Docker promotes compatibility and maintainability across platforms, as well as simplicity, rapid deployment, and security.
Dockerizing your Python application
“Dockerizing,” or containerizing, a Python application consists of these steps.
- Setting up a Python application (in this case, you will use a basic API created for this tutorial)
- Creating a Dockerfile with the necessary instructions to containerize the application
- Running the
docker buildcommand to create a Docker image from the Dockerfile
Setting up the sample Python API
To demonstrate the Docker process in this section of the tutorial, you will use FastAPI RestAPI locally. The API has already been created for you. Start by cloning the repository using this command:
git clone https://github.com/CIRCLECI-GWP/python-app-dockerhub.git
cd python-app-dockerhub
This clones the GitHub repository into a directory called python-app-dockerhub, and then goes to it.
Inside the python-app-dockerhub directory, create a virtual environment and activate it using the commands provided for your operating system.
## Windows OS ##
# create a venv
python -m venv venv
# activate venv
venv\Scripts\activate
## Linux/Mac OS ##
#create a venv
python3 -m venv venv
# activate venv
source venv/bin/activate
Once the virtual environment is created, you can install the dependencies using this command:
pip install -r requirements.txt
Run the application:
uvicorn app.main:app --reload
Now that you have validated that your API runs successfully, create a Dockerfile for your application.
Creating a Dockerfile
As mentioned earlier, a Dockerfile is a cookbook for creating images. Docker will read the instructions from the Dockerfile and construct an image when you run the command docker build.
In this case, you need the Docker file to:
- Download the
Python 3.10.2version - Find your packages within the application
- Perform the installation
Once the installation is successful, you need the image to spin up your application within the container.
Add this to your Dockerfile:
FROM python:3.10.2
WORKDIR /usr/src/app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
In this Docker configuration, the WORKDIR command specifies the working directory, which is the absolute path to your directory. The COPY command copies the requirements.txt file to the directory where your application is located. The RUN command installs the packages that you need to run your application. Finally, CMD specifies the command that will be executed when the image is run.
You may be wondering what the host and the ports are used for in the application RUN command. Docker will use the uvicorn as the server and your application entry port. The host and the port will ensure that you can access the application outside the container.
If you want to build the image locally to verify that your Dockerfile is working, you can run this command:
docker build -t fastapi-app .
Next, you will need to create a new repository on GitHub, commit and then push all the changes to that repository. Then you can start deploying your Docker containers to Docker Hub using CircleCI.
Using CI/CD to deploy a Docker image to Docker Hub
Docker Hub is a repository for Docker images, similar to GitHub. It makes it easier to search and share Docker images with other developers.
If you are regularly building and deploying Docker images for your projects, automating these processes with CI/CD can significantly streamline your workflow. Automation ensures that every code commit triggers the creation of a Docker image, which is then tested and deployed systematically without manual intervention. This not only saves time but also reduces the chances of errors and inconsistencies between development, testing, and production environments.
In this section, you’ll learn how to set up a CircleCI CI/CD pipeline for fast, error-free Docker builds and deployments. If you haven’t yet signed up for your free CircleCI account, be sure to do that before proceeding.
Setting up CircleCI
To integrate your application with CircleCI, you will need to create a configuration file. The config file will tell CircleCI how to initialize your repository and run tests.
From the root directory, create a new directory called .circleci/. In this new directory, create a new file called config.yml. Add this to your config.yml file:
version: 2.1
orbs:
python: circleci/python@2.1.1
jobs:
build-and-test:
executor: python/default
steps:
- checkout
- python/install-packages:
pkg-manager: pip
- run:
name: Install dependencies
command: |
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
- python/install-packages:
args: pytest
pkg-manager: pip
pypi-cache: false
- run:
name: Test with pytest
command: pytest
workflows:
build-master:
jobs:
- build-and-test
This configuration uses the circleci/python@2.1.1 orb to install all dependencies and then runs the tests defined in test_main.py using Pytest.
After adding the CircleCI configuration file, commit and push your changes to your remote GitHub repository.
Now you can create a CircleCI project from the dashboard.
Click Set Up Project to start building. You will be prompted to use the configuration file within your project’s repository. Enter the name of the branch where the configuration file is. In this case, it is the main branch.
Click Set Up Project to complete the process.
Now that you have run tests on CircleCI, your job is half done! Next, you need to deploy your application to Docker Hub. You will need to add an option to first log in to Docker Hub, then build an image each time the tests pass.
Pushing any image to Docker Hub will always require authentication. You first need to configure your Docker Hub USER_ID, PASSWORD and image name on the CirclecI dashboard. Go to the CircleCI dashboard. From the Settings section, select Environment Variables.
Add the environment variables using this format:
DOCKER_HUB_USER_IDis your Docker Hub ID.DOCKER_HUB_PASSWORDis your Docker Hub password.
Add the Docker image name as an environment variable with the environment name IMAGE_NAME. Assign it a unique name as a value.
Your CircleCI configuration only runs your tests. You will need to modify it so that it runs the tests first, then builds a Docker image and pushes it to Docker Hub if all the tests pass.
Add a deploy job to your config.yml file like this:
version: 2.1
orbs:
python: circleci/python@2.1.1
jobs:
build-and-test:
executor: python/default
steps:
- checkout
- python/install-packages:
pkg-manager: pip
- run:
name: Install dependencies
command: |
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
- python/install-packages:
args: pytest
pkg-manager: pip
pypi-cache: false
- run:
name: Test with pytest
command: pytest
deploy:
docker:
- image: cimg/base:2024.02
steps:
- checkout
- setup_remote_docker
- run:
name: Build and push to Docker Hub
command: |
docker build -t $DOCKER_HUB_USER_ID/$IMAGE_NAME:latest .
echo "$DOCKER_HUB_PASSWORD" | docker login -u "$DOCKER_HUB_USER_ID" --password-stdin
docker push $DOCKER_HUB_USER_ID/$IMAGE_NAME:latest
workflows:
build-master:
jobs:
- build-and-test
- deploy:
requires:
- build-and-test
This configuration file contains a workflow with two jobs:
- The
build-and-testjob builds and tests the code. - The
deployjob builds and pushes your Docker image to Docker Hub.
In the added configuration deploy step, the deploy job retrieves the code before launching a remote Docker engine. When creating Docker images for deployment using the Docker execution environment, you will need to use the setup_remote_docker option. This option creates a separate and remote environment for each build, for security purposes. This environment is specifically set up to run Docker commands.
Once that’s done, you can start running Docker commands for building and tagging your image:
docker build -t $DOCKER_HUB_USER_ID/$IMAGE_NAME:latest .
This command builds a Docker image and tags it with the :latest tag. The image name is prefixed with the Docker Hub username you set up earlier as an environment variable in the CircleCI dashboard. That is followed by the image’s actual name, which is also an environment variable you set.
Once you have a built image, sign into your Docker Hub account using CircleCI and the stored environment variables.
echo "$DOCKER_HUB_PASSWORD" | docker login -u "$DOCKER_HUB_USER_ID" --password-stdin
This is what the command in the configuration file does. Now that you have an image and are logged into Docker Hub, your image is pushed to Docker Hub using this command defined in the configuration file:
docker push $DOCKER_HUB_USER_ID/$IMAGE_NAME:latest
To ensure the integrity of your pipeline, there is a condition that requires your build and test job to pass before deploying to Docker Hub. This happens in the final configuration block under the workflow:
workflows:
version: 2
build-master:
jobs:
- build-and-test
- deploy:
requires:
- build-and-test
Finally, you need to commit and push all of your changes to GitHub. Once you do this, check the CircleCI dashboard to review the progress of your tests and deployment.
And voilà! both your deploy and build-and-test steps are green! Check the deploy job, and you will find that the image was built successfully and pushed to Docker Hub.
Now verify that your image was pushed to Docker Hub. Go to the Docker Hub website and check the image’s status. In this case, your image is named pythonuniquedockerimage.
You have not only verified that your CircleCI configuration works, but also verified that your image was successfully pushed to Docker Hub. You have been able to create a CI/CD process using CircleCI and Docker Hub, saving yourself time that would have been spent on manual Docker Hub deployment.
Conclusion
In this tutorial, you learned how to Dockerize your Python applications and to build a Docker image using a defined Dockerfile in a project. You also learned how to use CircleCI to automatically build, test, and deploy Docker images to Docker Hub, making your Docker workflows faster and more reliable.
I hope you enjoyed following this tutorial. Until the next one, stay sharp, and keep learning!
Waweru Mwaura is a software engineer and a life-long learner who specializes in quality engineering. He is an author at Packt and enjoys reading about engineering, finance, and technology. You can read more about him on his web profile.
