GitHub and Bitbucket Integration
This document provides an overview of using GitHub, GitHub Enterprise, or Bitbucket Cloud with CircleCI in the following sections:
- Best Practices for Keys
- Renaming Orgs and Repositories
- Enable Your Project to Check Out Additional Private Repositories
- Controlling Access Via a Machine User
- Permissions Overview
- Deployment Keys and User Keys
- Establishing the authenticity of an SSH host
To use CircleCI you need to be using either GitHub or Bitbucket for your VCS. When you add a project to CircleCI, the following GitHub or Bitbucket Cloud settings are added to the repository using the permissions you gave CircleCI when you signed up:
- A deploy key that is used to check out your project from GitHub or Bitbucket Cloud.
- A service hook (or “push hook”) that is used to notify CircleCI when you push to GitHub or Bitbucket Cloud.
CircleCI builds push hooks by default. So, builds are triggered for all push hooks for the repository and PUSH is the most common case of triggering a build.
There are some additional, less common cases where CircleCI uses hooks, as follows:
- CircleCI processes PR hooks (Pull Request Hooks) to store PR information for the CircleCI app. If the Only Build Pull Requests setting is set then CircleCI will only trigger builds when a PR is opened, or when there is a push to a branch for which there is an existing PR. Even if this setting is set we will always build all pushes to the project’s default branch.
- If the Build Forked Pull Requests setting is set, CircleCI will trigger builds in response to PRs created from forked repos.
It is possible to edit the webhooks in GitHub or Bitbucket Cloud to restrict events that trigger a build. Editing the webhook settings lets you change which hooks get sent to CircleCI, but doesn’t change the types of hooks that trigger builds. CircleCI will always build push hooks and will build on PR hooks (depending on settings), but if you remove push hooks from the webhook settings CircleCI won’t build. Refer to the GitHub Edit a Hook document or the Atlassian Manage Webhooks document for details.
Refer to CircleCI documentation of Workflows filters for how to build tag pushes.
Add a .circleci/config.yml File
After you create and commit a
.circleci/config.yml file to your GitHub or Bitbucket Cloud repository CircleCI immediately checks your code out and runs your first job along with any configured tests. For example, if you are working on a Rails project using Postgres specifications and features you might configure the following job run step:
jobs: build: docker: - image: circleci/ruby:2.4.1-jessie auth: username: mydockerhub-user password: $DOCKERHUB_PASSWORD # context / project UI env-var reference steps: - run: | bundle install bundle exec rake db:schema:load bundle exec rspec spec bundle exec cucumber
CircleCI runs your tests on a clean container every time so that your code is never accessible to other users and the tests are fresh each time you push. Watch your tests update in real-time on your dashboard or get status when CircleCI sends you a notification email after the job finishes. Status badges also appear on GitHub or Bitbucket Cloud as shown in the following screenshot for a commit from user keybits:
Integrated status also appears on the pull request screen, to show that all tests have passed:
Best Practices for Keys
- Use Deploy Keys whenever possible.
- When Deploy Keys cannot be used, Machine User Keys must be used, and have their access restricted to the most limited set of repos and permissions necessary.
- Never use non-Machine user keys (keys should be associated with the build, not with a specific person).
- You must rotate the Deploy or User key as part of revoking user access to that repo.
- After revoking the user’s access in github, delete keys in GitHub.
- Delete the keys in the CircleCI project.
- Regenerate the keys in CircleCI project.
- Ensure no developer has access to a build in a repo with a User Key that requires more access than they have.
Renaming Orgs and Repositories
If you find you need to rename an org or repo that you have previously hooked up to CircleCI, best practice is to follow these steps:
- Rename org/repo in VCS.
- Head to the CircleCI application, using the new org/repo name, for example,
- Confirm that your plan, projects and settings have been transferred successfully.
- You are then free to create a new org/repo with the previously-used name in your VCS, if desired.
Note: If these steps are not followed, it is possible, that you may lose access to your org or repo settings, including environment variables and contexts.
Enable Your Project to Check Out Additional Private Repositories
If your testing process refers to multiple repositories, CircleCI will need a GitHub user key in addition to the deploy key because each deploy key is valid for only one repository while a GitHub user key has access to all of your GitHub repositories.
Provide CircleCI with a GitHub user key in your project’s Project Settings > SSH keys. Scroll down the page to User Key and click Authorize with Github. CircleCI creates and associates this new SSH key with your GitHub user account for access to all your repositories.
User key security
CircleCI will never make your SSH keys public.
Remember that SSH keys should be shared only with trusted users and that anyone that is a GitHub collaborator on a project employing user keys can access your repositories, so only entrust a user key to someone with whom you would entrust your source code.
User key access-related error messages
Here are common errors that indicate you need to add a user key.
Python: During the
pip install step:
ERROR: Repository not found.
Ruby: During the
bundle install step:
Permission denied (publickey).
Controlling Access Via a Machine User
For fine-grained access to multiple repositories, it is best practice to create a machine user for your CircleCI projects. A machine user is a GitHub user that you create for running automated tasks. By using the SSH key of a machine user, you allow anyone with repository access to build, test, and deploy the project. Creating a machine user also reduces the risk of losing credentials linked to a single user.
To use the SSH key of a machine user, follow the steps below.
Note: To perform these steps, the machine user must have admin access. When you have finished adding projects, you can revert the machine user to read-only access.
Create a machine user by following the instructions on GitHub.
Log in to GitHub as the machine user.
Log in to CircleCI. When GitHub prompts you to authorize CircleCI, click the Authorize application button.
From the Add Projects page, follow all projects you want the machine user to have access to.
On the Project Settings > Checkout SSH keys page, click the Authorize With GitHub button. This gives CircleCI permission to create and upload SSH keys to GitHub on behalf of the machine user.
Click the Create and add XXXX user key button.
Now, CircleCI will use the machine user’s SSH key for any Git commands that run during your builds.
- Get a user’s email address
- Get a list of a user’s repos
- Add an SSH key to a user’s account
Admin Permissions, needed for setting up a project
- Add deploy keys to a repo
- Add service hooks to a repo
Note: CircleCI only asks for permissions that are absolutely
necessary. However, CircleCI is constrained by the specific
permissions each VCS provider chooses to supply. For example, getting
a list of all user’s repos – public and private – from GitHub
which is write-level access. GitHub does not provide a read-only
permission for listing all a user’s repositories.
If you feel strongly about reducing the number of permissions CircleCI uses, consider contacting your VCS provider to communicate your concerns.
Permissions for Team Accounts
This section provides an overview of the possible team and individual account choices available to meet various business needs:
If an individual has a personal GitHub account, they will use it to log in to CircleCI and follow the project on CircleCI. Each ‘collaborator’ on that repository in GitHub is also able to follow the project and build on CircleCI when they push commits. Due to how GitHub and Bitbucket store collaborators, the CircleCI Team page may not show a complete list. For an accurate list of contributors, please refer to your GitHub or Bitbucket project page.
If an individual upgrades to a GitHub Team account they can add team members and may give admin permissions on the repo to those who run builds. The owner of the team GitHub account (org) must go to the CircleCI Add Project, click the link to GitHub’s application permissions screen, and select Authorize CircleCI to enable members of the org to follow the project from their account. A team account with two members is $25 per month instead of $7 per month for a personal account.
An individual Bitbucket account is free for private repos for teams of up to five. An individual may create a Bitbucket team, add members and give out admin permissions on the repo as needed to those who need to build. This project would appear in CircleCI for members to follow without additional cost.
How to Re-enable CircleCI for a GitHub Organization
This section describes how to re-enable CircleCI after enabling third-party application restrictions for a GitHub organization. Go to GitHub Settings and in the “Organization access” section either:
- “Request access” if you are not an admin for the organization in question (an admin will have to approve the request) or
- “Grant access” if you are an admin
After access is granted, CircleCI should behave normally again.
GitHub recently added the ability to approve third party application access on a per-organization level. Before this change, any member of an organization could authorize an application (generating an OAuth token associated with their GitHub user account), and the application could use that OAuth token to act on behalf of the user via the API with whatever permissions were granted during the OAuth flow.
Now OAuth tokens will, by default, not have access to organization data when third party access restrictions are enabled. You must specifically request access on a per organization basis, either during the OAuth process or later, and an organization admin must approve the request.
You can enable third party access restrictions by visiting the organization settings page on GitHub, and clicking “Setup application access restrictions” button in the “Third-party application access policy” section.
If you enable these restrictions on an organization for which CircleCI has been running builds then we will stop receiving push event hooks from GitHub (thus not building new pushes), and API calls will be denied (causing, for instance, re-builds of old builds to fail the source checkout.) To get CircleCI working again you have to grant access to the CircleCI application.
The account and permissions system we use is not as clear as we would like and as mentioned we have a much improved system in development with users as first class citizens in CircleCI.
Deployment Keys and User Keys
What is a deploy key?
When you add a new project, CircleCI creates a deployment key on the web-based VCS (GitHub or Bitbucket) for your project. A deploy key is a repo-specific SSH key. If you are using GitHub as your VCS then GitHub has the public key, and CircleCI stores the private key. The deployment key gives CircleCI access to a single repository. To prevent CircleCI from pushing to your repository, this deployment key is read-only.
If you want to push to the repository from your builds, you will need a deployment key with write access. The steps to create a deployment key with write access depend on your VCS. See below for GitHub-specific instructions.
What is a user key?
A user key is a user-specific SSH key. Your VCS has the public key, and CircleCI stores the private key. Possession of the private key gives the ability to act as that user, for purposes of ‘git’ access to projects.
Creating a GitHub Deploy Key
In this example,
the GitHub repository is
and the CircleCI project is https://circleci.com/gh/you/test-repo.
Create an SSH key pair by following the GitHub instructions. When prompted to enter a passphrase, do not enter one.
Caution: Recent updates in
ssh-keygendon’t generate the key in PEM format by default. If your private key does not start with
-----BEGIN RSA PRIVATE KEY-----, enforce PEM format by generating the key with
ssh-keygen -m PEM -t rsa -C "email@example.com"
https://github.com/you/test-repo/settings/keys, and click “Add deploy key”. Enter a title in the “Title” field, then copy and paste the public key you created in step 1. Check “Allow write access”, then click “Add key”.
Go to your project settings, click on SSH Keys, and “Add SSH key”, and add the private key you created in step 1. In the “Hostname” field, enter “github.com”, and press the submit button.
In your config.yml, add the fingerprint using the
version: 2 jobs: deploy-job: steps: - add_ssh_keys: fingerprints: - "SO:ME:FIN:G:ER:PR:IN:T"
When you push to your GitHub repository from a job, CircleCI will use the SSH key you added.
Creating a Bitbucket User Key
Bitbucket does not currently provide CircleCI with an API to create user keys. However, it is still possible to create a user key by following this workaround:
In the CircleCI application, go to your project’s settings.
Navigate to the Checkout SSH Keys page.
Right-click the Create
<username>user key button and select the Inspect option to open the browser inspector.
In the developer console, select the Network tab.
In the developer console, click the
checkout-keywith a 201 status and copy the
public_keyto your clipboard.
Add the key to Bitbucket by following Bitbucket’s guide on setting up SSH keys.
.circleci/config.yml, add the fingerprint using the
version: 2 jobs: deploy-job: steps: - add_ssh_keys: fingerprints: - "SO:ME:FIN:G:ER:PR:IN:T"
How are these keys used?
When CircleCI builds your project, the private key is installed into the
directory and SSH is subsequently configured to communicate with your version
control provider. Therefore, the private key is used for:
- checking out the main project
- checking out any GitHub-hosted submodules
- checking out any GitHub-hosted private dependencies
- automatic git merging/tagging/etc.
For this reason, a deploy key isn’t sufficiently powerful for projects with additional private dependencies.
What about security?
The private keys of the checkout keypairs CircleCI generates never leave the CircleCI systems (only the public key is transmitted to GitHub) and are safely encrypted in storage. However, since the keys are installed into your build containers, any code that you run in CircleCI can read them. Likewise, developers that can SSH in will have direct access to this key.
Isn’t there a difference between deploy keys and user keys?
Deploy keys and user keys are the only key types that GitHub supports. Deploy keys are globally unique (for example, no mechanism exists to make a deploy key with access to multiple repositories) and user keys have no notion of scope separate from the user associated with them.
To achieve fine-grained access to more than one repo, consider creating what GitHub calls a machine user. Give this user exactly the permissions your build requires, and then associate its user key with your project on CircleCI.
Establishing the authenticity of an SSH host
When using SSH keys to checkout repositories, it may be neccesary to add the
fingerprints for GitHub or BitBucket to a “known hosts” file
~/.ssh/known_hosts) so that the executor can verify that the host it’s
connecting to is authentic. The
checkoutjob step does this automatically, so
the following command will need to be used if you opt to use a custom checkout
mkdir -p ~/.ssh echo 'github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== bitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw== ' >> ~/.ssh/known_hosts
SSH keys for servers can be fetched by running
ssh-keyscan <host>, then
adding the key that is prefixed with
ssh-rsa to the
known_hosts file of your
job. You can see this in action here:
➜ ~ ssh-keyscan github.com # github.com:22 SSH-2.0-babeld-2e9d163d github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== # github.com:22 SSH-2.0-babeld-2e9d163d # github.com:22 SSH-2.0-babeld-2e9d163d ➜ ~ ✗