Well-designed secrets management is a delicate balancing act between security and usability. Secrets must be easily accessible to the right users when building and deploying, but they must also at the same time be well-secured and easy to rotate. This article will cover how to thread this needle by integrating CircleCI with HashiCorp Vault and retrieving secrets using short-lived OpenID Connect (OIDC) authentication tokens.

Secrets management antipatterns

In the past, many teams chose usability over security. They embedded secrets in their CI/CD platform configuration or even checked them into version control, often in a “set and forget” fashion. This required very little effort and was often done as a temporary measure. But temporary measures tend to become permanent, and these secrets often remained in place far longer than intended. They became a potential attack vector – how quickly could they be rotated if, say, source code were leaked or an engineer left the company? Even on teams with regular manual secret rotation schedules, rotations were often late, incomplete, or simply never happened at all. Though the secrets were conveniently accessible, the company’s security posture was dangerously compromised.

Read more: The Path to Platform Engineering

At the other end of the spectrum, some teams locked down all access to secrets behind inconvenient solutions such as password managers that could only be accessed manually or manual MFA for every interaction with a service. Not only did this slow down development speed, but it also ironically often weakened security as well, since many engineers simply copied credentials locally or used backdoor accounts to circumvent the cumbersome official process. Though secrets were in theory quite secure, in practice they were often illicitly stored in insecure locations and developer agility was hindered to boot.

OIDC authentication offers a middle way between these two extremes that preserves both usability and platform security. CircleCI’s OIDC support allows developers to use ephemeral authentication tokens in their build, test, and deploy jobs, eliminating the risk inherent in long-lived credentials. Implementing OIDC enhances security and reduces friction across a project’s entire CI/CD pipeline, leading to faster and more efficient development. We’ve covered the basics of OIDC and how to authenticate with AWS and Google Cloud using CircleCI OIDC tokens in a previous blog post. Today we’ll cover how to use OIDC to authenticate with HashiCorp Vault.

What is HashiCorp Vault?

HashiCorp Vault is an identity-based secrets and encryption management system. As technical organizations increasingly move towards GitOps and infrastructure-as-code practices, they often run into the difficult question of how to securely store and access information that is too sensitive to check in to version control and too widely-used to leave lying around on someone’s workstation. Vault enables developers and platform engineers to conveniently store, access, and rotate secrets in a secure, centralized system. Using OIDC to authenticate from CircleCI to Vault adds a further layer of security since your engineers no longer need to store long-lived credentials outside of Vault itself.

In this tutorial, we will cover how to authenticate to your existing Vault cluster using a CircleCI’s OIDC token.

Step 0: Before You Begin

You will need the following to begin:

You should also gather the following info from the CircleCI UI:

  • Your organization Name and ID: Open the CircleCI webapp and navigate to Organization Settings > Overview on the CircleCI web app to find both
  • Your project ID: Open the CircleCI web app and navigate to your project’s page and click on Project Settings > Overview
  • A personal access token: See instructions here for how to create one.

Note: CircleCI must be able to connect to your Vault instance. If the instance is publicly accessible, you can restrict access by whitelisting CircleCI’s IP ranges so that only CircleCI machines can send traffic. If the instance is on a private network, you could run your Vault authentication job on a CircleCI Runner inside the private network.

Step 1: Configuring Vault

You will need to enable the JWT authentication method in your Vault instance. Log in to Vault and enable the JWT auth method:

export VAULT_ADDR="your vault instance address"
export VAULT_TOKEN="your vault token"
vault login $VAULT_TOKEN
vault auth enable jwt

Next, configure the JWT auth method to accept OIDC tokens from your CircleCI organization:

vault write auth/jwt/config \
    oidc_discovery_url="https://oidc.circleci.com/org/<your org id>" \
    bound_issuer="https://oidc.circleci.com/org/<your org id>"

Verify the config with this command:

curl --header "X-Vault-Token: $VAULT_TOKEN" "$VAULT_ADDR/v1/auth/jwt/config" | jq

Next, create a Vault policy for your Vault role. For this tutorial, you will grant CircleCI read access to all secrets under the path secret/circleci-demo/

vault policy write circleci-demo-policy - <<EOF
# Grant CircleCI project <your project name> RO access to secrets under the 'secret/data/circleci-demo/*' path
path "secret/data/circleci-demo/*" { 
  capabilities = ["read", "list"] 
}
EOF

Verify the policy creation with the following command:

vault policy read circleci-demo-policy

Create a Vault role under the JWT auth method named circleci-demo for CircleCI to assume. The attributes of this role tell Vault where the OIDC token will come from, what permissions should be granted to the token bearer, and any additional claims that should be included in the token. These additional claims can be used in the Vault role’s bound_claims field to restrict role access to specific projects and/or jobs with access to specific contexts.

In this tutorial, you will use the project_id additional claim to restrict access to only your tutorial project. You can find more detail on CircleCI OIDC token claims in our OIDC documentation.

vault write auth/jwt/role/circleci-demo -<<EOF
{
  "role_type": "jwt",
  "user_claim": "sub",
  "user_claim_json_pointer": "true",
  "bound_claims": {
    "oidc.circleci.com/project-id": "<your project id>"
  },
  "policies": ["default", "circleci-demo-policy"],
  "ttl": "10m"
}
EOF

Verify the role creation with this command:

vault read auth/jwt/role/circleci-demo

Finally, let’s create some secrets to access from CircleCI. First, enable a new secrets engine mounted at secret/:

vault secrets enable -version=2 -path=secret kv

Next, create some secrets:

vault kv put secret/circleci-demo/demo-secrets \
  username="paul.atreides" \
  password="lis@n-al-g4ib"

Verify that the new secrets were created with this command:

vault kv get secret/circleci-demo/demo-secrets

Step 2: Configuring CircleCI

Creating a CircleCI Context for Vault Connection Secrets

Instead of checking our Vault endpoint and role name into version control where they could be inadvertently leaked, we can store them securely in a CircleCI context. In this tutorial, we will walk through creating and populating a context using the CircleCI CLI.

Note: If you are using a CircleCI standalong org (e.g. if you are using GitLab as your VCS), you will need to create a context and populate the variables via the CircleCI UI instead of the CLI.

Run the following command to set up the CircleCI CLI. Enter you personal access token when prompted.

circleci setup

Once the CLI is configured, create a context named circleci-vault-demo using the command below. Your VCS should be either gh (Github) or bb (Bitbucket).

export CIRCLE_VCS=<your vcs>
export CIRCLE_ORG_NAME=<your org name>
circleci context create $CIRCLE_VCS $CIRCLE_ORG_NAME circleci-vault-demo

Verify that the new context was created with this command:

circleci context list $CIRCLE_VCS $CIRCLE_ORG_NAME

Next, add the Vault connection secrets to your new context using the commands below. You will be prompted to enter the secret after each command, so be sure to have the values ready. Example values are shown in the table below.

circleci context store-secret $CIRCLE_VCS $CIRCLE_ORG_NAME circleci-vault-demo VAULT_ADDR
circleci context store-secret $CIRCLE_VCS $CIRCLE_ORG_NAME circleci-vault-demo VAULT_ROLE
Context variable name           Value
VAULT_ADDR URL of your vault instance, including port (e.g. https://vault.example.com:8200)
VAULT_ROLE Vault role that CircleCI will assume (this will be circleci-demo if you followed the steps above)

Verify that the secrets have been created with this command:

circleci context show $CIRCLE_VCS $CIRCLE_ORG_NAME circleci-vault-demo

Write a CircleCI config that uses OIDC to authenticate with Vault

Now we’ll write a CircleCI config file that uses a CircleCI OIDC token to authenticate with Vault, uses Vault’s auto-auth functionality to retrieve the secrets we created, and then finally exports them as environment variables for use within a CircleCI job.

In the repository you created for this tutorial, create a file at .circleci/config.yml and add the code below.

version: 2.1
commands: 
  install-vault:
    steps:
      - run:
          name: Install Vault and prereqs
          command: |
            vault -h && exit 0 || echo "Installing vault"
            # only runs if vault command above fails
            cd /tmp
            wget https://releases.hashicorp.com/vault/1.12.2/vault_1.12.2_linux_amd64.zip
            unzip vault_1.12.2_linux_amd64.zip
            sudo mv vault /usr/local/bin        
            vault -h
  vault-auto-auth:
    description: "Use Vault auto auth to load secrets"
    steps:
      - run:
          name: Auto-authenticate with Vault
          command: |
            # Write the CircleCI provided value to a file read by Vault
            echo $CIRCLE_OIDC_TOKEN > .circleci/vault/token.json
            # Substitute the env vars in our context to render the Vault config file
            sudo apt update && sudo apt install gettext-base
            envsubst < .circleci/vault/agent.hcl.tpl > .circleci/vault/agent.hcl
            # This config indicates which secrets to collect and how to authenticate     
            vault agent -config=.circleci/vault/agent.hcl
      - run:
          name: Set Environment Variables from Vault
          command: |
            # In order to properly expose values in Environment, we _source_ the shell values written by agent
            source .circleci/vault/setenv
jobs:
  setup-vault-and-load-secrets:
    docker: 
      - image: cimg/base:2023.01
    steps:
      - checkout
      - install-vault
      - vault-auto-auth
      - run: 
          name: Use secrets retrieved from Vault in a subsequent step
          command: |
            echo "Username is $SECRET_DEMO_USERNAME, password is $SECRET_DEMO_PASSWORD"
workflows:
  vault: 
    jobs:
      - setup-vault-and-load-secrets:
          context:
            - circleci-vault-demo
# VS Code Extension Version: 1.5.1

Next, we will create a templated Vault agent config file at .circleci/vault/agent.hcl.tpl using the code below. This file will tell the Vault agent which Vault server to connect to and how it should authenticate.

pid_file = "./pidfile"
exit_after_auth = true
vault {
  address = "${VAULT_ADDR}"
  retry {
    num_retries = -1
  }
}
auto_auth {
  method "jwt" {
    exit_on_err = true
    config = {
      role = "${VAULT_ROLE}"
      path = ".circleci/vault/token.json"
      remove_jwt_after_reading = false
    }
  }
  sink "file" {
    config = {
      path = "/tmp/vault-token"
    }
  }
}
template_config {
  exit_on_retry_failure = true
}
template {
  source      = ".circleci/vault/secrets.ctmpl"
  destination = ".circleci/vault/setenv"
}

Finally, we will create a Consul template that will tell the Vault agent what to do with the secrets it retrieves. In this tutorial, we will simply export them as environment variables for use in our pipeline. Create a file at .circleci/vault/secrets.ctmpl with the code below.

# Export the secrets as env vars for use in this job 
# Also writes them to $BASH_ENV so that they'll be available as env vars in subsequent jobs for this pipeline
{{ with secret "secret/circleci-demo/demo-secrets" }}
    export SECRET_DEMO_USERNAME="{{ .Data.data.username }}"
    export SECRET_DEMO_PASSWORD="{{ .Data.data.password }}"
    echo "export SECRET_DEMO_USERNAME=\"{{ .Data.data.username }}\"" >> $BASH_ENV
    echo "export SECRET_DEMO_PASSWORD=\"{{ .Data.data.password }}\"" >> $BASH_ENV
{{ end }}

Push your changes. This will trigger a pipeline in CircleCI. Run the following command from your local repo directory:

circleci open

This will open your project in the CircleCI webapp. If everything has been configured correctly, you should see the final step print the secrets that were retrieved from Vault as shown below:

Success!

Conclusion

In this article, we covered how to use OIDC authentication to securely store and access secrets in HashiCorp Vault from CircleCI. OIDC authentication eliminates the need to store long-lived credentials outside of a secure system, strengthening your organization’s security posture without hindering development speed. To learn more about OIDC authentication on CircleCI, visit our documentation or check out the following articles: