Dynamically rendering config templates for secrets management

It’s often necessary to inject secrets into your build or deployment process so that the deployed service can interact with other services. This can be straightforward if you’re only deploying to a single environment. When deploying to multiple environments, though, you might need to dynamically inject different secrets depending on the environment to which you’re deploying.

Say you have a virtual machine (VM) that is part of your application. That VM needs a secret API token embedded in a configuration file to communicate with a third-party service. To keep your production and nonproduction environments isolated, you have created a “prod” API token for your production servers and a “dev” API token for your non-production servers. Now, you don’t want to hardcode these tokens into a config file and check it into your version control system (VCS), for two reasons. First, because checking secrets into source control is poor security practice, and second, because you don’t want to maintain separate “prod” and “dev” versions of this config file. A better solution would be to generate the config file with the correct token on the fly and inject it into the CI/CD pipeline job. But this introduces several new challenges:

1. How to securely store and access the secrets
2. How to dynamically render the config templates
3. How to inject the templates once rendered

We’ll go over some solutions to each of these challenges in the next sections of this post.

Storing and accessing secrets securely

The challenge here is twofold: First, how can you store your secrets securely? Second, how can you make them accessible to the processes in your CI/CD pipeline that will render the templates?

Managing secrets with CircleCI contexts

CircleCI has a built-in secret store accessible via our contexts feature. Secrets recorded in contexts have the following attributes:

• They are stored in Hashicorp Vault, encrypted using AES256-GCM96, and cannot be accessed by CircleCI employees (link).
• They can be automatically rotated using the CircleCI CLI.
• Their keys and values cannot be modified after creation.
• Their values cannot be revealed outside of a CircleCI pipeline job.

In the configuration for a CircleCI pipeline job, you can specify one or more contexts to load. The secrets in these contexts will then be made available as environment variables for the duration of that job. You could then have your template rendering engine insert the data from these environment variables into your config file templates to render a config file appropriate for the target deployment environment.

Using third-party secret stores

Another option is to use third-party secret stores such as SecretHub, AWS Secrets Manager, or Hashicorp Vault. These are more flexible because their secrets can be used with services other than CircleCI. However, to use their secrets in CircleCI, you will need to script your own method of pulling these secrets and of making them available to your template rendering engine. You may be able to leverage CircleCI’s partner and community orbs to help integrate some of these tools.

Dynamically rendering configuration templates

Your choice of template rendering engine will depend on several factors, including:

• The tools you are using for builds and/or deployments
• The languages used in your application
• The templating languages that your team is most comfortable using

If you use Terraform, you might use its string templating syntax. If you use Python or Ansible, you might use Mako or Jinja templates. In both of these examples, you could save some time by using CircleCI pre-built images with Terraform, Ansible, or Python pre-installed for jobs that include template rendering. You could also check out any relevant orbs, such as the Jinja orb.

Once you’ve checked your config file templates into your VCS and added steps to your pipeline’s deploy job to render the templates using your chosen template rendering engine, you just ensure that the secrets that you will inject into the templates are available to your template rendering engine. As mentioned in the previous section, CircleCI contexts make this easy. If you are using a third-party secret store, you’ll need to do some scripting to securely retrieve the secrets and pass them to the template rendering engine.

Injecting rendered templates into the pipeline

So you’ve rendered your templates in your pipeline and now you need to inject them into your pipeline jobs. This part might sound easy: just render the templates and insert them into your build artifacts, or push them to infrastructure created during your deployment

However, it’s not always practical to render your templates in the same job that will use the rendered templates. Imagine that you are rendering Jinja-templated config files that you want to inject into a Docker image build. The most convenient CircleCI executor for rendering Jinja templates is a Docker container running CircleCI’s Python convenience image. The most efficient executor for building Docker images might be a machine executor (in other words, a VM). Each job runs in a clean container or VM, so the templates rendered in the Docker container would not be available to the machine executor running the Docker build. You could combine both tasks into a single pipeline job that runs on one of the two executors, but this would make your pipeline slower and more complex.

Fortunately, CircleCI allows you to cache dependencies between jobs in a workspace. You can render your templates, cache them in a workspace, and then attach that workspace to any subsequent jobs that need to use those templates. This enables you to use the executor that is the most effective (in terms of config effort and compute time) for each job.

Dynamically rendering config templates solves the problem of securely storing secrets while keeping them accessible to the processes in your CI/CD pipeline. In this article, I’ve covered how to securely store and access secrets using CircleCI contexts or third-party secret stores. I also covered choosing a template rendering engine and how to use that engine to inject secrets into configuration templates. Applying best practices for secrets management using dynamically rendered templates will help keep your CI/CD pipelines running successfully and securely.