In our article on managing static credentials, we discussed the necessity of secrets — the passwords, tokens, and API keys that connect digital services together — and the importance of keeping them secure so that your infrastructure and data are kept safe from intrusion and misuse.

For organizations delivering software at scale, managing credentials across multiple teams and projects can quickly become tedious and error-prone, creating bottlenecks and unnecessary risk. Long-lived static credentials are particularly vulnerable, as they typically provide access to a wide range of resources and services, which can make it difficult to implement the principle of least privilege and limit the blast radius in the event of a security incident.

In this tutorial, we’ll explore how platform engineers can improve pipeline security by implementing OpenID Connect (OIDC) tokens to authenticate with a central secrets store and using custom OIDC claims for fine-grained access control. To demonstrate, we’ll show you how to connect your CircleCI pipelines to HashiCorp Vault with OIDC, and how to use Vault policies and custom OIDC claims to enforce role-based access control (RBAC).

Benefits of using OIDC and custom claims for role-based credential management

One of the core responsibilities faced by platform engineers is to empower development teams through streamlined, secure, self-service workflows, and building efficient CI/CD pipelines is the foundation of this work. To automate software delivery processes, CI/CD pipelines need access to a range of protected environments and services. To avoid the risks associated with granting pipelines broad access to sensitive credentials, platform engineers can implement short-term authentication using OIDC.

Read more: The Path to Platform Engineering

OIDC is an identity layer built on top of the OAuth 2.0 authorization protocol. It allows services to authenticate and authorize access to resources using temporary access tokens — often only for the duration of a single request.

Authenticating using OIDC means platform engineers can give developers access to the resources they need without the risk of exposing long-lived static credentials in the process. Through the use of custom OIDC claims, engineers can restrict which teams, repositories, or projects can access credentials and can quickly invalidate and rotate keys to respond to security incidents.

The principle of least privilege states that users and services should only be granted access to the resources they need to perform an assigned task — rather than having blanket access to resources. Using RBAC to restrict access to secrets in CI pipelines by project, user, branch, or tag allows platform engineers to adhere to this principle. If suspicious behavior is detected, access to the secure vault can be revoked, and all of the affected secrets can be quickly identified and then immediately rotated, preventing attackers from using shared and overly permissive credentials to move laterally through your network.

HashiCorp Vault can securely store static credentials and provides RBAC functionality through OIDC claims — giving platform engineers fine-grained control over which users or CI/CD workflows can authenticate to which environments and services.

In the following sections, we’ll describe how to centralize credentials management and implement the principle of least privilege in your pipelines using HashiCorp Vault and OIDC tokens with custom claims.

What you need to use OIDC to connect CircleCI to HashiCorp Vault

To connect CircleCI to HashiCorp Vault, you will need the following already set up:

If you’re new to OIDC and the OAuth 2.0 protocol that it is built on, it is worth getting to know them. OIDC and OAuth are the industry standards for authentication and authorization, with services such as AWS, Google Cloud, and Azure supporting them as a means of securing access to connected resources.

Setting up role-based OIDC authentication with CircleCI and Vault

CircleCI is configured using the .circleci/config.yml file, which can be edited either through the CircleCI web interface or through your Git repository. Vault is installed and configured via its command-line interface, and there is a graphical web interface to help you easily manage your secrets once you’re up and running.

Configuring Vault and CircleCI

CircleCI’s flexible CI/CD platform allows you to integrate almost any third-party service through orbs (reusable packages of pipeline configuration), support for command-line operations executed on a variety of operating systems and architectures, and our API and webhooks.

To get started setting up Vault and configuring your CircleCI pipelines to connect to it using OIDC, follow our tutorial on integrating HashiCorp Vault with CircleCI. This tutorial shows you how to set up a Vault instance and retrieve secrets from it using OIDC authentication by calling the Vault CLI from within a CircleCI pipeline.

Success after retrieving secrets from Vault

Setting up policies in Vault

Once you have CircleCI connected to Vault using OIDC, you can start implementing role-based access for your workflow tasks. This will give you granular control over which projects, users, and automated CI/CD pipelines can access secrets.

After following the OIDC tutorial linked above, you will have a single policy and a role that refers to the CircleCI project ID to grant access to secrets in Vault. Vault stores objects in a hierarchical path structure, and policies are used to grant or deny access to specified paths.

The next step to expanding the use of RBAC in CircleCI is to create additional policies in your Vault configuration, each granting access to the specific Vault paths that store the secrets required for different tasks in each of your CI/CD pipelines.

The shell command below creates an example Vault policy that, when added to a role, will grant it access to all secrets stored under the path secret/data/circleci-rbac-demo:

vault policy write circleci-rbac-demo-policy - <<EOF
# Grant users with this role access to secrets under the 'secret/data/circleci-rbac-demo/*' path
path "secret/data/circleci-rbac-demo/*" { 
  capabilities = ["read", "list"] 
}
EOF

You can list all of the policies that you have created by running:

vault policy list

Using OIDC claims in CircleCI for role-based access in Vault

In OIDC, claims are made by the authorized party to the application with attributes about the user that can be used to determine access. In this case, Vault will use the information provided by CircleCI’s OIDC claims to assign a role.

Multiple policies can be assigned to a role, so you can create roles for each CircleCI project or pipeline that connects to Vault, and then add and remove policies to the role as required to manage its access to Vault secrets. This allows for granular control of which users and services can access secrets.

In the example below, a Vault role is created to restrict access to a secret based on the branch of a project’s Git repository. CircleCI pipelines can be triggered for each commit, and can be set to only run on commit to a specific branch. Restricting secret access to only the relevant branch means that other pipelines that may be set up for development environments cannot access production resources.

vault write auth/jwt/role/circleci-rbac-demo -<<EOF
{
  "role_type": "jwt",
  "user_claim": "sub",
  "user_claim_json_pointer": "true",
  "bound_claims": {
    "oidc.circleci.com/vcs-ref": "refs/heads/prod"
  },
  "policies": ["default", "circleci-rbac-demo-policy"],
  "ttl": "10m"
}
EOF

This role, named circleci-rbac-demo, uses the oidc.circleci.com/vcs-ref claim provided by CircleCI during OIDC authorization. The format of this claim is refs/heads/branch_name, so to restrict the actions taken by pipelines running from the prod branch, Vault will check if the custom claim value matches refs/heads/prod. If it is a match, it assigns the policies default and circleci-rbac-demo-policy. CircleCI provides the default OIDC claims when making a request, as well as several additional claims that contain CircleCI-specific information that can be used to assign a role.

The OIDC with Vault tutorial stores the role that CircleCI will use to connect with Vault in the environment variable VAULT_ROLE. You can list all of the roles you have created by using the Vault list command:

vault list auth/jwt/role

Once you have set up a few roles with access to different secrets, you can test your RBAC implementation by committing to different branches of your repository. In a repository that uses the example policies and roles provided above, pipelines not running from the prod branch should fail based on the implemented RBAC.

Unsuccessful pipeline execution

Rotating secrets stored in Vault

Though centrally managing your credentials with Vault will improve their security, they should still be regularly rotated. An example of this is provided below as a Bash script that utilizes the Vault command-line interface and can be tailored to your specific requirements:

#!/bin/bash

# The Vault CLI is required for this script 
# https://developer.hashicorp.com/vault/downloads

# Set the token to access Vault. 
# The variable name VAULT_TOKEN is the default that
# is read by Vault if it is not already authenticated
# Tokens can be generated from the Vault CLI 
# https://developer.hashicorp.com/vault/docs/commands/token/create
VAULT_TOKEN=$(vault token create -period=30m -field=token -tls-skip-verify)

# Secrets will be read from a CSV file containing
# a comma-separated username/password
SECRETS_FILE="./secrets.csv"
VAULT_KV_PATH="secret/circleci-demo/demo-secrets"

# Get the username/password values by splitting 
# the contents of the secrets file at the comma (,)
USERNAME="$(cat $SECRETS_FILE | cut -d',' -f1)"
PASSWORD="$(cat $SECRETS_FILE | cut -d',' -f2)"

# Update the secret stored in Vault 
# https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2
vault kv put -tls-skip-verify  "$VAULT_KV_PATH" \
	username="$USERNAME" \
	password="$PASSWORD"

This script connects to your Vault instance and calls the CLI write command to rotate out a username and password stored in the vault’s KV (Key-Value) engine, taking new values from the file secrets.csv. This script can be called from a cron job or adapted to run in a scheduled pipeline. HashiCorp Vault also has an HTTP API that can be called instead of using the command-line interface if you are calling it from a web service or other environment.

Once your key rotation processes are in place, they can be called on a schedule and invoked on demand in response to an ongoing incident. Of particular note is Vault’s ability to rotate the encryption keys it uses to secure the data it stores, allowing you to lock down your Vault instance and require that all clients re-authenticate in the event of suspicious activity.

If the services you authenticate with are supported, you can use Vault’s dynamic secrets functionality instead of manually rotating keys. This removes the need for storing credentials altogether, instead retrieving ephemeral secrets from the third-party service when they are required and passing them on to the requester. Dynamic secrets are revoked immediately after use, making them the most secure option for authenticating and authorizing access to sensitive resources.

Conclusion

Organizations are under constant threat of attack through an ever-growing number of vectors. Malware, physical breaches, social engineering, extortion, software vulnerabilities, and poor secrets handling can all lead to bad actors being able to damage your business through your digital infrastructure.

A platform engineer’s job is to ensure that development teams have the tools they need to work effectively and that these tools do not unnecessarily increase the organization’s risk exposure. Following secrets management best practices is essential. Even for teams with impeccable security practices, the software supply chain presents its own risks, requiring that organizations have the tools and policies in place to quickly respond.

Short-lived OIDC tokens solve many of the security problems presented by static secrets. By incorporating role-based OIDC authentication with your secure secrets vault, you can achieve fine-grained control over sensitive credentials and reduce the risks associated with storing and using long-lived secrets. To get started securing your organization’s credentials and take control of your software delivery pipelines, sign up for a free CircleCI account today.