This is the first of a two-part series on package management in CI/CD, written by Armando Canals, co-founder at packagecloud.
Overview
In this post, we’re going to explain some fundamental concepts around software packages and package management and how to combine them with CI/CD to build a software delivery pipeline.
We’ll go over software packages and repositories, continuous delivery, and automating the building, testing and release stages of the software development process.
What is a software package?
A software package is a distributable file installed on servers, computers, and personal devices.
Think of .EXE files on Windows, .dmg files on Mac, or .rpm and .deb files on Linux/Debian systems — these are all software packages used by different platforms.
These software packages come in an array of different formats, and as many different ways to describe their metadata. Still, we can define a few steps that are common for most software packages:
* Write code
* Define metadata (name, version, dependencies, etc.)
* Archive code and metadata for distribution
The last step mentioned above, archiving code, creates a software package for release. This archive, or artifact, is distributed to repositories at the release stage which we’ll go over later in the post.
Automating builds, tests, and releases of software packages
To create a delivery pipeline, we want to automate as many manual processes as we can. Implementing automated releases in our project will help to reduce friction and human-error from the software development process.
Let’s go over some of the concepts, processes, and tools we’re going to touch on when creating a software delivery pipeline.
Continuous integration
Continuous integration is the practice of a team of software developers agreeing to integrate into a shared codebase frequently. Usually, a version-control system is used, like git or Subversion, along with running automated unit tests at each integration to the codebase.
We’re going to be using the git version control system and CircleCI as our continuous integration server.
We want each git commit to:
1. run any build tasks necessary to run tests.
2. run automated tests to ensure code quality.
By running these steps at each integration, we can detect any errors in these phases as soon as possible and fix them.
Continuous delivery
Continuous delivery is a logical next step of continuous integration. It describes a way to quickly get software from development to release by making software deployments simple, repeatable tasks.
Building software packages
The actual package building process varies per package type. Package types range from operating-system level packages to packages running front-end code in browsers–each one built differently.
Each package type has either a tool or a specific process, to build the package from source code into an archive for distribution. For example, npm comes with the npm pack command which will archive the codebase into a tarball for use on an npm registry.
For our example, we want to handle the necessary tasks to build the package from inside our CI server. This process will allow us to release the package to a repository in the release stage.
Automating releases
To trigger a build that will generate a new release, we need to tell our CI server that a release is ready. To do this, we use the git tag feature as a trigger to tell our CI server that a build is ready for release.
For example, the first command below would tag the latest commit, and the second would push the newly created tag to the remote git repository, triggering our build in CI:
git tag v0.1.0
git push origin v0.1.0
Pushing tags to the remote repo would trigger the deployment workflow of our CircleCI configuration which we’ll go over later in a later section.
As demonstrated above, a tagged commit pushed to the remote repository with a version number would trigger a build in our continuous integration system. This process, would archive the code into a package format (Deb, RPM, JAR, etc.) and then distribute it to an artifact/package repository.
Build the software package once
When testing code in different environments (DEV, STAGING, PRODUCTION), the package should be the same throughout each one. Building packages for each environment in a CI system could introduce hard-to-find bugs and create extra work, so it’s ideal to build any packages once, early in the testing process and upload them to a repository for use at later stages of testing, and through to production.
Distributing software packages
Once a release is ready, it needs to be made available to computers in a secure, reliable way. For us, this means publishing software packages to a package repository.
CircleCI allows us to publish our packages by supporting the use of command-line tools, and the ability to configure repository credentials securely.
Repositories
Package repositories are a mix of metadata and package objects that, in conjunction with package managers, are used to deliver up-to-date software to computers. These repositories are warehouses of packages along with their metadata and can be public or private.
Some examples of public software repositories would be the registry for npm and NodeJS packages npmjs.org or the Maven Central repository for Java/Maven artifact types.
For our example, we’ll be using packagecloud, which hosts private and public package repositories for many different package types and works seamlessly with different package managers.
Packagecloud generates the repository metadata used by the various package managers from operating-system level packages, to JavaScript packages used in the browser.
Package managers
Package managers are tools to interact with the package repositories containing software. They’re used by developers to search, install, and manage packages within the repositories.
For example, if you work with Debian-based operating systems, you’re likely familiar with the Advanced Packaging Tool - APT, or if you use RedHat-based systems, it’s possible you’re used to working with the yum/dnf tool.
Another example would be if you’re a JavaScript developer and you use npm for NodeJS packages, or a Python programmer using pip for the Python programming language — the list of package managers is extensive and ranges from operating-system level packages to JavaScript code running in the browser.
Conclusion
Integrating the moving parts of your software development process into your CI/CD systems can significantly reduce cycle times and human-error when developing software. By understanding the different parts of the software pipeline and automating the software delivery processes, teams can work faster and more efficiently.
In our next post, we’re going to build a fully-functional software delivery pipeline using CircleCI to build the package, run tests, and upload a working package to a repository on packagecloud.
