Imagine yourself in this situation: You are a motivated and skilled DevOps or DevEx engineer. You have a plan to implement automated, complete CI/CD pipelines. You know how to do it, and you know how the extra productivity and automation will benefit your team and the whole company. But the project is never approved, because of security concerns.
Many organizations, especially those in regulated industries, have strict requirements for releasing their software, and rightfully so. Who can trigger a release, and when, under what conditions, and what specific checks are required before release can go ahead must be tightly controlled, auditable, and reversible.
These kinds of complicated processes often require a dedicated delivery manager to collect and present all relevant feature briefs, test data, security assessment, and rollback plans to a committee of decision makers in the company. That group can give the delivery manager the green light to deploy, often also on a specific schedule.
There is another approach. You can set up effective and fully automated CI/CD pipelines that include all the checks a human delivery manager and approval committee can complete. This tutorial will show you ways to create fine grained access control for your pipelines. You can use these methods for strictly regulated internal company projects for the enterprise, as well as for popular open source projects.
Prerequisites
This tutorial assumes you have experience with CI/CD pipelines, and ideally with CircleCI. To implement the steps covered you will also need admin access to your GitHub and CircleCI organizations (and accounts for both services).
If you do not have administrator access to a GitHub organization, you can create a free one in GitHub and use it with a free CircleCI account.
Access control in CI/CD pipelines in enterprise projects
For an enterprise organization it is vital that existing checks are followed and communicated well. The process often includes manual checks of supporting documentation and the potential impact to business before a release can continue. I will guide you through implementing a pipeline that executes a number of verification steps automatically, and continues with deployment to production only after the dedicated delivery manager manually approves the process.
To implement access control you will need to set up a few things:
- Protected branches in GitHub
- Security groups in GitHub
- Contexts in CircleCI
- Approval jobs in CircleCI
Protected branches
Setting up a protected branch is the first step to setting up access control. A protected branch prohibits team members from pushing code directly to that branch, and instead forces all changes to go through the pull request (PR) process. Commonly you would protect the branch you use to create releases from; main for example. You can toggle protected branches in the settings for your repo.
You can also specify rules and exceptions, such as review requirements, checks that need to pass before merging, history rules, and much more.
In my example I set up my main branch as a protected branch that requires a review and PR to pass.
Setting up security groups
If a user without the correct context privileges triggers a CircleCI pipeline (for example, with a commit), then the job that requires that context will be marked as 403 (unauthorized).
In our organization we will set up two security groups: development-leads and delivery-managers
All developers in your organization can push to their own branches and issue pull requests, but only team leaders should be required to review any new pull requests. Only when the team leader approves can changes be merged into the protected main branch. Then, when delivery managers get the green light to ship, they are the ones to trigger releases of software. In this tutorial, we will create a separate security group for them.
CircleCI will use these security groups with contexts to limit which jobs can be executed by whom, so that only the delivery managers can trigger them manually.
Using CircleCI contexts
Contexts in CircleCI serve a double role. One role limits the scope of secrets shared with a job while the other controls access.
Only a job that has a context specified to it in a workflow is able to use the context’s secrets as environment variables. Contexts are shared across the entire organization in CircleCI, so they can be reused in multiple projects. You can set them up using the organization settings menu on the left hand side of the CircleCI dashboard.
For access control, you can specify security groups to any context. For this tutorial,
we will create a context called release and specify the delivery-mgrs security group to it.
Now you can set up your pipeline. The sample project uses the following flow:
- Tests
- Security scans
- Dev environment deployment
- Approval job for the delivery manager
- Production deployment
workflows:
build-test-deploy:
jobs:
- test
- security-scan:
context: security
- deploy-app:
name: deploy-app-dev
env: dev
context: deployment-dev
requires:
- test
- security-scan
filters:
branches:
only:
- main
- approve-for-prod:
type: approval
requires:
- deploy-app-dev
- deploy-app:
name: deploy-app-prod
env: prod
requires:
- approve-for-prod
context: release-prod
The pipeline uses contexts for environment variables like our API keys, but also uses the release-prod context. The release-prod context is gated so it can only be used by a delivery manager.
Setting up approval jobs
The final piece of the puzzle is the approval job: approve-for-prod. Technically, anyone can log in and approve this job. However, the deploy-app-prod job following it uses the release-prod context, which only accepts delivery managers’ credentials. If someone without those credentials approves it, the release job will fail with an unauthorized error, and the whole pipeline will terminate unsuccessfully.
Other helpful bits and pieces
One additional control we have set up in our workflow is the branch filter. The development deployment will occur only on the main branch because of that filter.
The main branch is labelled as protected so the only way to get the code onto the main branch is by passing the automated checks and code review, potentially by a lead engineer.
You could also use dedicated QA or Security review steps that require manual approval. Build them the same way as part of the chain.
Considerations for open source projects
So far we covered the enterprise use case for access control. Most OSS projects do not have the same strict compliance requirements as enterprises. In contrast these projects need to have their contributions opened to a much wider group of people. That means allowing many more people to trigger pipelines.
Often the company that initiated and owns a popular OSS project continues to employ the core contributors. They will probably be joined by other regular contributors and maintainers that are not part of that company. And then there is everyone else - anyone who occasionally might contribute a fix or a feature.
For that scenario, you can set up three flows:
- Company internal flow. Members contribute and also administer everything. Company employees are likely the only people allowed to release new versions of software. We can call them “inner core contributors”.
- Semi-internal flow. These members are part of the organization and will be able to push directly to the main OSS repository. They will be able to use CircleCI for everything, including debugging jobs with SSH and triggering pipelines. Call them “outer core contributors”.
- Standard flow. The rest of the contributors in the community who can issue PRs, otherwise work on their own. We will call them “community contributors”.
For community contributors. the flow is standard: make a fork, then open a pull request. You can toggle CircleCI to build pull requests that will automatically verify their changes.
You should still have a protected branch, and required checks.
For the groups of core contributors, you may want to include some secrets as well. Secrets require some more work. Also, passing secrets via forks is inherently insecure. Anyone could extract any secrets you allow to pass by modifying the pipeline in their PR. This could include API keys to certain services, or credentials to where your project is distributed.
One option is to avoid passing secrets to forks altogether. You will have to push any code after reviewing it into your own org, where it can be executed, and then merged into your default branch. There is a helpful article that explains the process.
The other option is to allow passing secrets to forks, but to prevent these secrets from being shared unless the code has been thoroughly reviewed and approved by a team member. You can set that up in the CircleCI project settings.
If you toggle secrets you will need to set up an approval job. Put any secrets in a context that will be accessed only by people who belong to that context.
Note: If you use this method, make sure that all contexts in your organization specify the security group. If you do not, a malicious actor could guess your context name in a PR that gets executed and extract environment variables from it.
With your inner and outer core of contributors, you can set up access controls in very much the same way. Maybe only the inner core can release new versions, but anyone in the outer core can merge any pull requests. Or maybe you just have one set of core maintainers who can do everything. The details are up to you!
Conclusion
In this article we have looked at ways to protect your CI/CD pipelines further by setting up access controls to cordon off parts of pipelines (and the associated credentials) to a limited scope of people. This is a versatile approach and works for both enterprise organizations as well as OSS projects.
We used security groups, protected branches, contexts, and approval jobs in CircleCI to achieve that effect, for both OSS and proprietary projects. Experiment with the best configuration for your organization and release your applications safely and securely.
