Static credential management for platform engineers

Cloud CI/CD is a force multiplier for development teams, especially those working remotely. Automated CI/CD takes load off of developers, allowing them to focus on building better products. Hosted CI/CD adds further benefit to this, ensuring that this newfound capacity isn’t spent managing the testing and deployment infrastructure, and that remote team members have easy access to CI/CD tools.

This is a boon for platform engineers, but there is one major concern when offloading vital development and deployment tasks to third parties — credential management. Credentials are the secret passwords, API keys, tokens, and other sensitive information that your application uses to connect to databases and third-party services. Unintended disclosure of these secrets due to accidental leakage, security breaches, or simply guesswork or brute force can lead to further data breaches, network intrusion, or abuse of infrastructure resources.

In this tutorial, we’ll discuss how platform engineers can implement best practices for managing static or long-lived credentials, starting with how to use CircleCI contexts, run workflows within restricted contexts, and rotate secrets stored as environment variables from the CircleCI CLI or API. Organizations interested in further reducing their risk exposure can also learn how to set up and manage short-lived OpenID Connect (OIDC) authentication tokens.

Why credential management is important for platform engineers and their teams

In large organizations, platform engineers support several development teams, each with their own projects spanning cloud, desktop, and mobile app deployments.

Read more: The Path to Platform Engineering

The platform team’s job is to ensure these development teams operate at peak efficiency by simplifying, standardizing, and streamlining their development processes. This involves automating complex infrastructure management tasks, creating reliable self-serve workflows for testing and deployment, and managing build, test, and deployment toolchains. A robust CI/CD pipeline is an essential part of this process. But in order for your pipelines to work seamlessly with your other tools, you need a programmatic way to manage access credentials and inject them into your build pipelines, without exposing them to unnecessary risk.

Storing credentials securely and rotating them regularly are vital for the security of any organization. CircleCI includes a number of features designed to help you securely connect your CI/CD pipelines to external services, including OIDC with role assumption/web identity, using OIDC to retrieve static credentials from a central vault, and storing static credentials in a CircleCI context.

OIDC with role assumption or web identities is the current industry standard for securely connecting online services. No long-lived secrets are involved, and the remote system is in charge of establishing trust between parties. OIDC is still in the early stages of adoption, and many services do not currently support it. However, static credentials can still be secured by storing them in a vault that’s accessed using OIDC, allowing them to be centrally managed and easily rotated. Where neither of these is practical, teams can use CircleCI contexts to store and isolate secrets for use in CI/CD pipelines.

Prerequisites for rotating static credentials in CircleCI

The rest of this article will demonstrate how to rotate static credentials stored in CircleCI contexts, providing a starting point to a more secure and easier-to-manage secrets management process for your CI/CD pipelines. It assumes that you have the following already set up:

• A CircleCI organization and a user with administrative privileges
• One or more projects in your CircleCI organization

Users of the CircleCI free tier who have signed in with GitHub or Bitbucket are able to act as the admins of their own organizations by immediately importing their Git repositories as CircleCI projects.

If you need to add projects from other sources, visit Account Integrations in the CircleCI user settings. If your organization uses a separate account to host repositories, you may also need to grant access to projects for other users.

Setting up contexts to manage project secrets in CircleCI

CircleCI contexts allow you to secure and share environment variables across your projects. This is the preferred method for handling static credentials across multiple projects in CircleCI, allowing for centralized management and rotation rather than individually managing secrets on a per-project basis. Below, we will run through the steps to create and use a context in a project, and then demonstrate how to programmatically rotate the secrets in the context using the CircleCI CLI.

We’ve created a GitHub repository that you can fork and import into CircleCI to try this out. This is a simple Fastify app that requires HTTP Basic authentication with a username and password. The app includes a single Node.js tap test that checks whether an HTTP 200 status code is returned. The username and password for the HTTP Basic authentication are supplied by a .env file: https://www.npmjs.com/package/dotenv.

This repository includes a CircleCI pipeline configuration that pulls secrets from a specified context, writes them to the .env file, and then runs the included test to ensure that everything is working.

Once you’ve cloned the example project or imported your own, you’ll need to create a context and update the list of context names in your CircleCI pipeline’s config.yml to match. Once you’ve created the context, click on it to edit it — you will want to add project restrictions so that only the projects that need access to the environment variables in the context can access it. Then add two environment variables and some appropriate values: HTTP_USERNAME and HTTP_PASSWORD.

CircleCI will pull the values of the environment variables from the contexts available to the project. To demonstrate how context project restrictions work, you can return to the context settings, revoke access for the project, and then try to run the pipeline again — it will fail, as it requires the presence of the now-inaccessible context variables to function.

How to automatically rotate secrets stored in CircleCI contexts

Regularly rotating static secrets is an important part of any credentials management strategy. This practice provides a base level of protection against accidental disclosure — if an API key or credential pair leaks unnoticed, they will only be useful for a limited amount of time. Ideally, they will be automatically rotated before they can be used by bad actors. Having the tools in place to readily rotate secrets also allows for a more rapid response in the case of a supply chain incident or other security breach.

CircleCI contexts can be managed with the CircleCI command-line interface (CLI). This allows for command-line and programmatic management of contexts, including environment variables.

To demonstrate, below is an example Bash script that takes two secrets from a text file and stores them as environmental variables in a CircleCI context. Before using this script, you will need to authenticate the CLI with CircleCI. Once you have authenticated, you’ll need the path to the context you wish to manage, which you can find using the list contexts command:

circleci context list

Once you have this information, you can tailor the script below to update your secrets:

#!/bin/bash

# Secrets will be read from a CSV file 
# containing a comma-separated username/password
SECRETS_FILE="./secrets.csv"

# Specify the path to the context that will be updated
VCS_TYPE="github"
ORGANIZATION_NAME="your_org_name"
CONTEXT_NAME="example_context"

# Get the username/password values by splitting
# the contents of the secrets file at the comma (,)
USERNAME="$(cat $SECRETS_FILE | cut -d',' -f1)"
PASSWORD="$(cat $SECRETS_FILE | cut -d',' -f2)"

# Update the environment variable in the CircleCI context
# Existing environment variables will be overwritten
echo "$USERNAME" | circleci context store-secret $VCS_TYPE $ORGANIZATION_NAME $CONTEXT_NAME "USERNAME"
echo "$PASSWORD" | circleci context store-secret $VCS_TYPE $ORGANIZATION_NAME $CONTEXT_NAME "PASSWORD"

This script could be automated by cron on a local machine, added to a CircleCI project and invoked via a scheduled pipeline, or triggered on demand to rapidly respond to an ongoing security threat. It is also just as easy to perform the same task using the CircleCI API if you are running in a different environment.

Scheduled and on-demand rotation using a CircleCI pipeline

To automate secrets rotation using a CircleCI pipeline, you’ll need to create a separate repository for the configuration so that it can be invoked without performing other testing or build tasks related to other projects. This example CircleCI configuration calls the above script when the pipeline is run:

workflows:
rotate-secrets:
    jobs:
    - rotate-secrets-script:
        # Set the contexts this job will run in - this must match the name of the context in your organization Settings
        context:
            - example_context

jobs:
rotate-secrets-script:
    # Run in a basic Ubuntu environment https://circleci.com/developer/images/image/cimg/base
    docker:
    - image: cimg/base:2023.04
    steps:
    # Install the latest version of the CircleCI CLI https://github.com/CircleCI-Public/circleci-cli
    - run:
        command: curl -fLSs https://raw.githubusercontent.com/CircleCI-Public/circleci-cli/main/install.sh | sudo bash
        name: Install CircleCI CLI
    - run:
        command: circleci setup --no-prompt --token=$CIRCLECI_API_TOKEN
        name: Configure CircleCI CLI
    # Get the code from the repository
    - checkout
    # Run script
    - run:
        command: sh rotate.sh
        name: Run secrets rotation script

To run this pipeline automatically on a schedule, add a scheduled trigger to the project in the CircleCI web interface.

You can schedule your pipeline to rotate secrets as often as your business requires, up to multiple times per hour depending on your security needs.

In case of an active security incident, you can also invoked your secrets rotation pipeline on demand. From the CircleCI dashboard, navigate to the project, select a branch, and click the Trigger Pipeline button.

The result of triggering the pipeline can be seen below — the Last Updated value for the updated secrets in the context has changed:

Usually, API keys and credentials are generated by the service they provide access to. For example, AWS IAM (Identity and Access Management) credentials are provided as a .csv file or in CLI/API-generated output. These credentials should never be committed to version control, so the above scripts are provided only as an example. You will need to update them to retrieve credentials from your own secure storage, for example, via SFTP or by executing your pipelines on a machine with access to your local resources using a self-hosted runner. These requirements can make rotating keys impractical — something addressed by implementing OIDC and using it to access a purpose-built secrets management solution such as HashiCorp Vault.

Improved credential management helps platform engineers increase security and productivity

Automated CI/CD is a modern necessity, allowing lean, agile development teams to rapidly iterate and innovate. CircleCI provides robust, cloud-hosted CI/CD pipelines, meaning that this liberating functionality comes with no infrastructure overhead.

This does require placing some amount of trust in your CI/CD platform along with the other third-party services you rely on. CircleCI works hard to earn this trust, providing full transparency regarding our security practices and tools to ensure that you can securely grant your pipelines access to external systems.

This article covered how to use CircleCI’s API and CLI tools to conveniently manage and rotate long-lived, static credentials stored in CircleCI contexts. CircleCI also supports OIDC role-based access — ephemeral secrets that are more secure than long-lived credentials. Our tutorial on role-based credential management with OIDC covers how to set this up with HashiCorp Vault so that you can progress from simple secret rotation to using a centralized secure store for your static credentials.