1. cci-x/docker-registry-image-cache@0.2.12

cci-x/docker-registry-image-cache@0.2.12

Sections
Save and restore a docker image cache using a registry. This orb provides commands which may be used as an alternative to the CircleCI 'docker_layer_caching' feature. It is intended to be used with 'machine' executors or jobs which use 'setup_remote_docker'. Using this orb This orb requires a docker registry which will be used to push and pull images. Popular docker registries include Docker Hub, Google Container Registry, AWS Elastic Container Registry, and Quay.io. If you already use one of these registries in your job you are all set. The registry you choose for the cache may be different from the registry you use to publish images. Using pulled images as a cache for docker build To use images pulled from a docker registry as a cache for image building you must use `--cache-from` with a list of docker images. This orb provides a `build` command to automatically add the images listed in the 'images' field to 'docker build'. If you use a script or a tool like docker-compose to build images you will need to pass the list of images yourself. With docker-compose make sure to set `build.cache_from` in your docker-compose.yml. Image tags The images listed in 'with-save-restore-images.images' are effectively the keys used for the cache. For best results you may want to use two tags, one which is gauranteed to exist (for example a tag that is pushed when CI runs on the 'master' branch) and one which is more likely to match the current Dockerfile (for example a tag which includes the current branch name or a hash of the Dockerfile). If you are building multiple docker images, you can include one or more tags for each image.
Created: October 25, 2019Version Published: April 23, 2020Releases: 15
Org Usage:
< 25

Orb Quick Start Guide

Use CircleCI version 2.1 at the top of your .circleci/config.yml file.

1 version: 2.1

Add the orbs stanza below your version, invoking the orb:

1 2 orbs: docker-registry-image-cache: cci-x/docker-registry-image-cache@0.2.12

Use docker-registry-image-cache elements in your existing workflows and jobs.

Opt-in to use of uncertified orbs on your organization’s Security settings page.

Usage Examples

cache_using_branch_name_key

This example demonstrates how to use this orb to speed up a 'docker build'. It uses the git branch name as a cache key, and the master branch as a fallback when the branch is new. The first step is to log into a GCR registry. The values for these REGISTRY_ variables may be set by project environment variables or from a CircleCI Context. Once login is complete the 'with-save-restore-images' command pulls images from a registry. This example uses two images. An image tagged with 'master' which acts as a base for new branches, and an image tagged with the name of the git branch. This example removes any slashes from the git branch name using bash string substitution ('${var/\//-}'). The images are listed using the YAML '>-' symbol, which replaces newlines with spaces and removes trailing newlines. This allows you to list images one per line. The nested step uses the 'build' command to build a new docker image. The orb command takes care of adding the '--cache-from' arguments to 'docker build' so that the images that were pulled are used as a cache. You may use as many nested steps as necessary. Only steps which build docker images need to be nested under 'with-save-restore-images'. Once all the nested steps have completed the 'with-save-restore-images' command pushes all the 'images' to the same registry where it pulled from originally. Finally, the rest of the job will continue and may use any of the images pulled or built in previous steps. In this example tests are run using a `test-all` script.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 version: 2.1 orbs: docker-cache: cci-x/docker-registry-image-cache@0.2.0 workflows: ci: jobs: - build-docker-image jobs: build-docker-image: machine: image: ubuntu-1604:201903-01 steps: - run: name: docker login command: | echo "$REGISTRY_PASSWORD" | \ docker login --username "$REGISTRY_USERNAME" --password-stdin https://us.gcr.io - docker-cache/with-save-restore-images: repository: us.gcr.io/myrepo images: myapp:${CIRCLE_BRANCH/\//-} myapp:master steps: - docker-cache/build: command: docker build -t myapp:${CIRCLE_BRANCH/\//-} . - run: name: Test command: ./test-all

cache_using_branch_name_key_with_multistage

This example builds on cache_using_branch_name_key to demonstrate how to use this orb with Dockerfiles that use multi-stage builds. The key thing to note is that you must build, cache, and restore each named stage of the build separately. A Dockerfile with the intermediary stages base1 and base2 is assumed: FROM ubuntu:latest AS base1 ... FROM base1 AS base2 ... FROM base2 ...

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 version: 2.1 orbs: docker-cache: cci-x/docker-registry-image-cache@0.2.0 workflows: ci: jobs: - build-docker-image jobs: build-docker-image: machine: image: ubuntu-1604:201903-01 steps: - run: name: docker login command: | echo "$REGISTRY_PASSWORD" | \ docker login --username "$REGISTRY_USERNAME" --password-stdin https://us.gcr.io - docker-cache/with-save-restore-images: repository: us.gcr.io/myrepo parallelism: 4 images: >- cache-orb:master cache-orb:base1-master cache-orb:base2-master cache-orb:${CIRCLE_BRANCH/\//-} cache-orb:base1-${CIRCLE_BRANCH/\//-} cache-orb:base2-${CIRCLE_BRANCH/\//-} steps: - docker-cache/build: command: >- docker build -t cache-orb:base1-${CIRCLE_BRANCH/\//-} --target base1 . - docker-cache/build: command: >- docker build -t cache-orb:base2-${CIRCLE_BRANCH/\//-} --target base2 . - docker-cache/build: command: docker build -t cache-orb:${CIRCLE_BRANCH/\//-} . - run: name: Test command: ./test-all

cache_using_hash_of_dockerfile_key

This example demonstrates how to use this orb to speed up a 'docker build'. It uses the md5sum of the Dockerfile as a cache key. It follows all the same steps as the example above, except that it uses a different key for the cache. It assumes that an image with a 'latest' tag is pushed from the master CI job and can be used as a fallback when the Dockerfile changes.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 version: 2.1 orbs: docker-cache: cci-x/docker-registry-image-cache@0.2.0 workflows: ci: jobs: - build-docker-image jobs: build-docker-image: machine: image: ubuntu-1604:201903-01 steps: - run: name: docker login command: | echo "$REGISTRY_PASSWORD" | \ docker login --username "$REGISTRY_USERNAME" --password-stdin - run: name: Generate cache key command: | md5sum Dockerfile | awk '{print $1}' > /tmp/cache-key - docker-cache/with-save-restore-images: repository: myhubrepo images: myapp:$(< /tmp/cache-key) myapp:latest steps: - docker-cache/build: command: docker build -t myapp:$(< /tmp/cache-key) . - run: name: Test command: ./test-all

Commands

with-save-restore-images

Restore a cache, run steps, then save a cache of the new image. The docker pull and docker push are optional, any failures will be ignored so that missing images do not cause the job to fail.

Show command Source
PARAMETER
DESCRIPTION
REQUIRED
DEFAULT
TYPE
images
A list of docker image names, separated by spaces, to pull from a registry and push to the registry when the nested steps finish. If repository is set the images will be prefixed with the value of that field before pulling and pushing.
No
''
string
repository
A prefix added to each image name before pulling and pushing from the registry. A repository may be just the hostname of the registry, or it may include the organization prefix as well.
No
''
string
steps
Steps to run after the cache has been pulled from a registry, and before pushing the new images to the registry. Any steps which build docker images must be included in this list.
Yes
-
steps
save-when
Condition used as the when attribute of the save cache operation. See https://circleci.com/docs/2.0/configuration-reference/#the-when-attribute for supported values.
No
on_success
string
image-filename
Path to a file used to store the list of images. You most likely don't need to change this.
No
/tmp/docker-cache/images
string
parallelism
Maximum number of processess to use when pushing and pulling images.
No
1
integer

build

Build a cache image adding a --cache-from parameter for every image in the list provided to with-save-restore-images.

Show command Source
PARAMETER
DESCRIPTION
REQUIRED
DEFAULT
TYPE
command
A `docker build` command to run. Should include a `--tag` parameter for the image.
Yes
-
string
step-name
Name for the step.
No
docker build
string
image-filename
Path to a file used to store the list of images. Must match the path provided to 'with-save-restore-images'. You most likely don't need to change this.
No
/tmp/docker-cache/images
string

tag

Tag a group of Docker images. To make the cache work when you are not using the orb's `build` command, your build must produce Docker images with the tags listed in the `images` parameter. If you do not wish to change your existing tagging, or you are using docker-compose or scripts where the image tags cannot easily be controlled, then this command will help.

Show command Source
PARAMETER
DESCRIPTION
REQUIRED
DEFAULT
TYPE
tags
A list of image tag pairs, separated by spaces, which will be passed to docker tag. The first item in the pair is the source tag, and the second the destination.
Yes
-
string
step-name
Name for the step.
No
Tag docker images
string

Orb Source

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 # This code is licensed from CircleCI to the user under the MIT license. # See here for details: https://circleci.com/developer/ja/orbs/licensing version: 2.1 description: | Save and restore a docker image cache using a registry. This orb provides commands which may be used as an alternative to the CircleCI 'docker_layer_caching' feature. It is intended to be used with 'machine' executors or jobs which use 'setup_remote_docker'. Using this orb This orb requires a docker registry which will be used to push and pull images. Popular docker registries include Docker Hub, Google Container Registry, AWS Elastic Container Registry, and Quay.io. If you already use one of these registries in your job you are all set. The registry you choose for the cache may be different from the registry you use to publish images. Using pulled images as a cache for docker build To use images pulled from a docker registry as a cache for image building you must use `--cache-from` with a list of docker images. This orb provides a `build` command to automatically add the images listed in the 'images' field to 'docker build'. If you use a script or a tool like docker-compose to build images you will need to pass the list of images yourself. With docker-compose make sure to set `build.cache_from` in your docker-compose.yml. Image tags The images listed in 'with-save-restore-images.images' are effectively the keys used for the cache. For best results you may want to use two tags, one which is gauranteed to exist (for example a tag that is pushed when CI runs on the 'master' branch) and one which is more likely to match the current Dockerfile (for example a tag which includes the current branch name or a hash of the Dockerfile). If you are building multiple docker images, you can include one or more tags for each image. commands: with-save-restore-images: description: | Restore a cache, run steps, then save a cache of the new image. The docker pull and docker push are optional, any failures will be ignored so that missing images do not cause the job to fail. parameters: images: description: | A list of docker image names, separated by spaces, to pull from a registry and push to the registry when the nested steps finish. If repository is set the images will be prefixed with the value of that field before pulling and pushing. default: "" type: string repository: description: | A prefix added to each image name before pulling and pushing from the registry. A repository may be just the hostname of the registry, or it may include the organization prefix as well. type: string default: "" steps: description: | Steps to run after the cache has been pulled from a registry, and before pushing the new images to the registry. Any steps which build docker images must be included in this list. type: steps save-when: description: | Condition used as the when attribute of the save cache operation. See https://circleci.com/docs/2.0/configuration-reference/#the-when-attribute for supported values. type: string default: on_success image-filename: description: | Path to a file used to store the list of images. You most likely don't need to change this. type: string default: "/tmp/docker-cache/images" parallelism: description: | Maximum number of processess to use when pushing and pulling images. type: integer default: 1 steps: - run: name: "DRIC: Setup cache" command: | if [[ -n "$CCI_DRIC_USERNAME" ]]; then echo "$CCI_DRIC_PASSWORD" | \ docker login --username "$CCI_DRIC_USERNAME" --password-stdin "$CCI_DRIC_REGISTRY" fi filename="<<parameters.image-filename>>" filename_with_repo="${filename}-with-repo" # clear any previous runs rm -f "${filename}" "${filename_with_repo}" # save images list to a file images="<<parameters.images>>" mkdir -p "$(dirname $filename)" echo $images | \ xargs -n 1 | \ uniq | \ xargs -n 1 echo >> "$filename" filename_with_repo="${filename}-with-repo" cp "$filename" "$filename_with_repo" repo="<<parameters.repository>>" if [[ -n "$repo" ]]; then sed -i -e "s|^|${repo%/}/|" "$filename_with_repo" fi - run: name: "DRIC: Restore cache from registry <<parameters.repository>>" command: | filename="<<parameters.image-filename>>" filename_with_repo="${filename}-with-repo" cat "$filename_with_repo" | xargs -r -P <<parameters.parallelism>> -n 1 docker pull || true docker image ls -a - steps: << parameters.steps >> - run: name: "DRIC: Retag images" command: | repo="<<parameters.repository>>" if [[ -z "$repo" ]]; then exit 0 fi filename="<<parameters.image-filename>>" filename_with_repo="${filename}-with-repo" paste "$filename" "$filename_with_repo" | xargs -n 2 docker tag || true docker image ls -a - run: name: "DRIC: Save cache to registry <<parameters.repository>>" when: <<parameters.save-when>> command: | filename="<<parameters.image-filename>>" filename_with_repo="${filename}-with-repo" cat "${filename_with_repo}" | xargs -r -P <<parameters.parallelism>> -n 1 docker push || true rm "${filename}" "${filename_with_repo}" build: description: | Build a cache image adding a --cache-from parameter for every image in the list provided to with-save-restore-images. parameters: command: description: | A `docker build` command to run. Should include a `--tag` parameter for the image. type: string step-name: description: | Name for the step. type: string default: "docker build" image-filename: description: | Path to a file used to store the list of images. Must match the path provided to 'with-save-restore-images'. You most likely don't need to change this. type: string default: "/tmp/docker-cache/images" steps: - run: name: <<parameters.step-name>> command: | set -x docker() { cache_from=$(sed -e 's|^|--cache-from=|' "<<parameters.image-filename>>-with-repo" | xargs) case "$1" in build) shift 1; command docker build $cache_from "$@";; *) command docker "$@";; esac } <<parameters.command>> tag: description: | Tag a group of Docker images. To make the cache work when you are not using the orb's `build` command, your build must produce Docker images with the tags listed in the `images` parameter. If you do not wish to change your existing tagging, or you are using docker-compose or scripts where the image tags cannot easily be controlled, then this command will help. parameters: tags: description: | A list of image tag pairs, separated by spaces, which will be passed to docker tag. The first item in the pair is the source tag, and the second the destination. type: string step-name: description: | Name for the step. type: string default: "Tag docker images" steps: - run: name: <<parameters.step-name>> command: | set -x cat \<<EOF | xargs -n 2 docker tag <<parameters.tags>> EOF examples: cache_using_branch_name_key: description: | This example demonstrates how to use this orb to speed up a 'docker build'. It uses the git branch name as a cache key, and the master branch as a fallback when the branch is new. The first step is to log into a GCR registry. The values for these REGISTRY_ variables may be set by project environment variables or from a CircleCI Context. Once login is complete the 'with-save-restore-images' command pulls images from a registry. This example uses two images. An image tagged with 'master' which acts as a base for new branches, and an image tagged with the name of the git branch. This example removes any slashes from the git branch name using bash string substitution ('${var/\//-}'). The images are listed using the YAML '>-' symbol, which replaces newlines with spaces and removes trailing newlines. This allows you to list images one per line. The nested step uses the 'build' command to build a new docker image. The orb command takes care of adding the '--cache-from' arguments to 'docker build' so that the images that were pulled are used as a cache. You may use as many nested steps as necessary. Only steps which build docker images need to be nested under 'with-save-restore-images'. Once all the nested steps have completed the 'with-save-restore-images' command pushes all the 'images' to the same registry where it pulled from originally. Finally, the rest of the job will continue and may use any of the images pulled or built in previous steps. In this example tests are run using a `test-all` script. usage: version: 2.1 orbs: docker-cache: cci-x/docker-registry-image-cache@0.2.0 workflows: ci: jobs: - build-docker-image jobs: build-docker-image: machine: {image: 'ubuntu-1604:201903-01'} steps: - run: name: docker login command: | echo "$REGISTRY_PASSWORD" | \ docker login --username "$REGISTRY_USERNAME" --password-stdin https://us.gcr.io - docker-cache/with-save-restore-images: repository: us.gcr.io/myrepo images: >- myapp:${CIRCLE_BRANCH/\//-} myapp:master steps: - docker-cache/build: command: docker build -t myapp:${CIRCLE_BRANCH/\//-} . - run: name: Test command: ./test-all cache_using_branch_name_key_with_multistage: description: | This example builds on cache_using_branch_name_key to demonstrate how to use this orb with Dockerfiles that use multi-stage builds. The key thing to note is that you must build, cache, and restore each named stage of the build separately. A Dockerfile with the intermediary stages base1 and base2 is assumed: FROM ubuntu:latest AS base1 ... FROM base1 AS base2 ... FROM base2 ... usage: version: 2.1 orbs: docker-cache: cci-x/docker-registry-image-cache@0.2.0 workflows: ci: jobs: - build-docker-image jobs: build-docker-image: machine: {image: 'ubuntu-1604:201903-01'} steps: - run: name: docker login command: | echo "$REGISTRY_PASSWORD" | \ docker login --username "$REGISTRY_USERNAME" --password-stdin https://us.gcr.io - docker-cache/with-save-restore-images: repository: us.gcr.io/myrepo parallelism: 4 images: >- cache-orb:master cache-orb:base1-master cache-orb:base2-master cache-orb:${CIRCLE_BRANCH/\//-} cache-orb:base1-${CIRCLE_BRANCH/\//-} cache-orb:base2-${CIRCLE_BRANCH/\//-} steps: - docker-cache/build: command: docker build -t cache-orb:base1-${CIRCLE_BRANCH/\//-} --target base1 . - docker-cache/build: command: docker build -t cache-orb:base2-${CIRCLE_BRANCH/\//-} --target base2 . - docker-cache/build: command: docker build -t cache-orb:${CIRCLE_BRANCH/\//-} . - run: name: Test command: ./test-all cache_using_hash_of_dockerfile_key: description: | This example demonstrates how to use this orb to speed up a 'docker build'. It uses the md5sum of the Dockerfile as a cache key. It follows all the same steps as the example above, except that it uses a different key for the cache. It assumes that an image with a 'latest' tag is pushed from the master CI job and can be used as a fallback when the Dockerfile changes. usage: version: 2.1 orbs: docker-cache: cci-x/docker-registry-image-cache@0.2.0 workflows: ci: jobs: - build-docker-image jobs: build-docker-image: machine: {image: 'ubuntu-1604:201903-01'} steps: - run: name: docker login command: | echo "$REGISTRY_PASSWORD" | \ docker login --username "$REGISTRY_USERNAME" --password-stdin - run: name: "Generate cache key" command: | md5sum Dockerfile | awk '{print $1}' > /tmp/cache-key - docker-cache/with-save-restore-images: repository: myhubrepo images: >- myapp:$(< /tmp/cache-key) myapp:latest steps: - docker-cache/build: command: docker build -t myapp:$(< /tmp/cache-key) . - run: name: Test command: ./test-all
開発者向け最新情報
ビルド最適化のためのヒントを手に入れましょう。
または、リサーチパネルに参加して フィードバックをお寄せください。
このフォームを送信することで、当社の利用規約およびプライバシーポリシーに同意したものと見なされます。