In our previous posts, we explored the benefits of job orchestration, or whether you run your jobs one at a time or in parallel; and multi-executor workflows, a feature which lets you run jobs across different platforms, languages and resource classes in one workflow. Now, we’ll explore various ways to further control your workflows, using branch and tag filtering, and different approval options.
What is filtering?
There are times when it’s not ideal to build your entire workflow on every push. For example, wanting to deploy only from a particular branch (i.e. master), or if you have a large monorepo and want to save time. One way to get around this is with filtering.
There are two types of filters that exist in workflows: filtering by branch and filtering by tag.
Branch and tag filtering are methods for controlling when a job should run. With filtering, you can deploy or run tests for the branches or tags that you define. That means you can set up a filter that says, “Run this job only when it meets this condition”. With filters, you can create rules and only run jobs based on those criteria. Those criteria can be defined under the job configuration itself. For example: if you want to deploy to production from your default branch, you can use filtering to filter all the other branches and keep them from running. You can create conditions and rules to run your deployment jobs from only the branches or tags that you choose.
Note: branch and tag filters can operate on both a job-specific and a workflow-specific level. For an example of workflow-level filtering, see our CircleCI images.
Example 1: Datadog
See DataDog’s config.yml here.
Think of a tag like a bookmark. Your git commit history is happening all the time, and you can place tags at key locations that you might want to come back to. DataDog is using tags to mark their releases, so both their team and their customers can keep track of the versioned releases and release notes. Tags help keep a history of things that are important.
In this example, they are using tags to run all of the jobs except their publish_master job, which runs only for their master branch. When they push, they push a git tag, and that triggers a workflow and its subsequent jobs. If the tag matches the pattern defined in the regular expression of that tag in the config, then the workflow and its subsequent jobs are executed.
Example 2: Google Cloud
See Google Cloud’s config.yml here.
In this example, the team working on Google Cloud is doing a regular expression match on tags for all their jobs. When running a job, teams can not only define a branch-level filter, but also a tag-level filter. In the case of Google Cloud, they are using regular expression to match their tags: the tags that match that regular expression are the ones that are going to build. The jobs that don’t match those expressions will not build.
Example 3: Aeternity
See Aeternity’s config.yml here.
Here, the Aeternity team is using the scheduling functionality to run workflows at different times for different branches. In this example, they are running their integration_deploy job at night while also running system tests at midnight. For jobs that you want to run at a particular time, and also long-running jobs, it can be a wise choice to schedule them to run when the workday is over, and you know you’ll have sufficient bandwidth. For example, if you had two containers and you were to run these jobs midday, your team might find themselves waiting on containers, and you never want your team waiting on machines.
Example 4: Azure
See Azure’s config.yml here.
This team is running two different workflows: workflows for their PRs, and workflows for their master branch. In the workflow called build_and_test_PR, they are using the manual approval feature in order to gate the jobs which are running across different Kubernetes environments. In this way, they’re not running the jobs on each commit–they’re running the jobs only after manual approval. Once these jobs are approved, they’ll run across all their different Kubernetes environments. Using manual approvals can be good for long-running jobs or very resource-heavy jobs. Additionally, approvals can be handy when calling third party services or trying to protect compute. In these instances, you can set up your job so that an approver from the QA or product team needs to give the go-ahead.
We’ve shown you how examples of various ways workflows can be set up to take advantage of various job orchestration setups, running workflows on multiple platforms and across executors, as well as controlling the flow of running jobs. Next in our series, we have a look inside our own workflows, as we show you how we built our Docker convenience images.
