Use CircleCI version 2.1 at the top of your .circleci/config.yml file.
1
version: 2.1
Add the orbs
stanza below your version, invoking the orb:
1
2
orbs:
docker-registry-image-cache: cci-x/docker-registry-image-cache@0.2.12
Use docker-registry-image-cache
elements in your existing workflows and jobs.
Opt-in to use of uncertified orbs on your organization’s Security settings page.
This example demonstrates how to use this orb to speed up a 'docker build'. It uses the git branch name as a cache key, and the master branch as a fallback when the branch is new. The first step is to log into a GCR registry. The values for these REGISTRY_ variables may be set by project environment variables or from a CircleCI Context. Once login is complete the 'with-save-restore-images' command pulls images from a registry. This example uses two images. An image tagged with 'master' which acts as a base for new branches, and an image tagged with the name of the git branch. This example removes any slashes from the git branch name using bash string substitution ('${var/\//-}'). The images are listed using the YAML '>-' symbol, which replaces newlines with spaces and removes trailing newlines. This allows you to list images one per line. The nested step uses the 'build' command to build a new docker image. The orb command takes care of adding the '--cache-from' arguments to 'docker build' so that the images that were pulled are used as a cache. You may use as many nested steps as necessary. Only steps which build docker images need to be nested under 'with-save-restore-images'. Once all the nested steps have completed the 'with-save-restore-images' command pushes all the 'images' to the same registry where it pulled from originally. Finally, the rest of the job will continue and may use any of the images pulled or built in previous steps. In this example tests are run using a `test-all` script.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
version: 2.1
orbs:
docker-cache: cci-x/docker-registry-image-cache@0.2.0
workflows:
ci:
jobs:
- build-docker-image
jobs:
build-docker-image:
machine:
image: ubuntu-1604:201903-01
steps:
- run:
name: docker login
command: |
echo "$REGISTRY_PASSWORD" | \
docker login --username "$REGISTRY_USERNAME" --password-stdin https://us.gcr.io
- docker-cache/with-save-restore-images:
repository: us.gcr.io/myrepo
images: myapp:${CIRCLE_BRANCH/\//-} myapp:master
steps:
- docker-cache/build:
command: docker build -t myapp:${CIRCLE_BRANCH/\//-} .
- run:
name: Test
command: ./test-all
This example builds on cache_using_branch_name_key to demonstrate how to use this orb with Dockerfiles that use multi-stage builds. The key thing to note is that you must build, cache, and restore each named stage of the build separately. A Dockerfile with the intermediary stages base1 and base2 is assumed: FROM ubuntu:latest AS base1 ... FROM base1 AS base2 ... FROM base2 ...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
version: 2.1
orbs:
docker-cache: cci-x/docker-registry-image-cache@0.2.0
workflows:
ci:
jobs:
- build-docker-image
jobs:
build-docker-image:
machine:
image: ubuntu-1604:201903-01
steps:
- run:
name: docker login
command: |
echo "$REGISTRY_PASSWORD" | \
docker login --username "$REGISTRY_USERNAME" --password-stdin https://us.gcr.io
- docker-cache/with-save-restore-images:
repository: us.gcr.io/myrepo
parallelism: 4
images: >-
cache-orb:master cache-orb:base1-master cache-orb:base2-master
cache-orb:${CIRCLE_BRANCH/\//-}
cache-orb:base1-${CIRCLE_BRANCH/\//-}
cache-orb:base2-${CIRCLE_BRANCH/\//-}
steps:
- docker-cache/build:
command: >-
docker build -t cache-orb:base1-${CIRCLE_BRANCH/\//-} --target
base1 .
- docker-cache/build:
command: >-
docker build -t cache-orb:base2-${CIRCLE_BRANCH/\//-} --target
base2 .
- docker-cache/build:
command: docker build -t cache-orb:${CIRCLE_BRANCH/\//-} .
- run:
name: Test
command: ./test-all
This example demonstrates how to use this orb to speed up a 'docker build'. It uses the md5sum of the Dockerfile as a cache key. It follows all the same steps as the example above, except that it uses a different key for the cache. It assumes that an image with a 'latest' tag is pushed from the master CI job and can be used as a fallback when the Dockerfile changes.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
version: 2.1
orbs:
docker-cache: cci-x/docker-registry-image-cache@0.2.0
workflows:
ci:
jobs:
- build-docker-image
jobs:
build-docker-image:
machine:
image: ubuntu-1604:201903-01
steps:
- run:
name: docker login
command: |
echo "$REGISTRY_PASSWORD" | \
docker login --username "$REGISTRY_USERNAME" --password-stdin
- run:
name: Generate cache key
command: |
md5sum Dockerfile | awk '{print $1}' > /tmp/cache-key
- docker-cache/with-save-restore-images:
repository: myhubrepo
images: myapp:$(< /tmp/cache-key) myapp:latest
steps:
- docker-cache/build:
command: docker build -t myapp:$(< /tmp/cache-key) .
- run:
name: Test
command: ./test-all
Restore a cache, run steps, then save a cache of the new image. The docker pull and docker push are optional, any failures will be ignored so that missing images do not cause the job to fail.
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
images | A list of docker image names, separated by spaces, to pull from a registry
and push to the registry when the nested steps finish. If repository is set the
images will be prefixed with the value of that field before pulling and pushing.
| No | '' | string |
repository | A prefix added to each image name before pulling and pushing from the registry.
A repository may be just the hostname of the registry, or it may include the organization
prefix as well.
| No | '' | string |
steps | Steps to run after the cache has been pulled from a registry, and before pushing the
new images to the registry.
Any steps which build docker images must be included in this list.
| Yes | - | steps |
save-when | Condition used as the when attribute of the save cache operation.
See https://circleci.com/docs/2.0/configuration-reference/#the-when-attribute
for supported values.
| No | on_success | string |
image-filename | Path to a file used to store the list of images.
You most likely don't need to change this.
| No | /tmp/docker-cache/images | string |
parallelism | Maximum number of processess to use when pushing and pulling images.
| No | 1 | integer |
Build a cache image adding a --cache-from parameter for every image in the list provided to with-save-restore-images.
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
command | A `docker build` command to run. Should include a `--tag` parameter for
the image.
| Yes | - | string |
step-name | Name for the step.
| No | docker build | string |
image-filename | Path to a file used to store the list of images. Must match the path provided to
'with-save-restore-images'.
You most likely don't need to change this.
| No | /tmp/docker-cache/images | string |
Tag a group of Docker images. To make the cache work when you are not using the orb's `build` command, your build must produce Docker images with the tags listed in the `images` parameter. If you do not wish to change your existing tagging, or you are using docker-compose or scripts where the image tags cannot easily be controlled, then this command will help.
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
tags | A list of image tag pairs, separated by spaces, which will be passed to docker tag.
The first item in the pair is the source tag, and the second the destination.
| Yes | - | string |
step-name | Name for the step.
| No | Tag docker images | string |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
# This code is licensed from CircleCI to the user under the MIT license.
# See here for details: https://circleci.com/developer/ja/orbs/licensing
version: 2.1
description: |
Save and restore a docker image cache using a registry.
This orb provides commands which may be used as an alternative to the CircleCI 'docker_layer_caching' feature. It is intended to be used with 'machine' executors or jobs which use 'setup_remote_docker'.
Using this orb
This orb requires a docker registry which will be used to push and pull images. Popular docker registries include Docker Hub, Google Container Registry, AWS Elastic Container Registry, and Quay.io. If you already use one of these registries in your job you are all set. The registry you choose for the cache may be different from the registry you use to publish images.
Using pulled images as a cache for docker build
To use images pulled from a docker registry as a cache for image building you must use `--cache-from` with a list of docker images. This orb provides a `build` command to automatically add the images listed in the 'images' field to 'docker build'. If you use a script or a tool like docker-compose to build images you will need to pass the list of images yourself. With docker-compose make sure to set `build.cache_from` in your docker-compose.yml.
Image tags
The images listed in 'with-save-restore-images.images' are effectively the keys used for the cache. For best results you may want to use two tags, one which is gauranteed to exist (for example a tag that is pushed when CI runs on the 'master' branch) and one which is more likely to match the current Dockerfile (for example a tag which includes the current branch name or a hash of the Dockerfile).
If you are building multiple docker images, you can include one or more tags for each image.
commands:
with-save-restore-images:
description: |
Restore a cache, run steps, then save a cache of the new image.
The docker pull and docker push are optional, any failures will be ignored so that missing images do not cause the job to fail.
parameters:
images:
description: |
A list of docker image names, separated by spaces, to pull from a registry
and push to the registry when the nested steps finish. If repository is set the
images will be prefixed with the value of that field before pulling and pushing.
default: ""
type: string
repository:
description: |
A prefix added to each image name before pulling and pushing from the registry.
A repository may be just the hostname of the registry, or it may include the organization
prefix as well.
type: string
default: ""
steps:
description: |
Steps to run after the cache has been pulled from a registry, and before pushing the
new images to the registry.
Any steps which build docker images must be included in this list.
type: steps
save-when:
description: |
Condition used as the when attribute of the save cache operation.
See https://circleci.com/docs/2.0/configuration-reference/#the-when-attribute
for supported values.
type: string
default: on_success
image-filename:
description: |
Path to a file used to store the list of images.
You most likely don't need to change this.
type: string
default: "/tmp/docker-cache/images"
parallelism:
description: |
Maximum number of processess to use when pushing and pulling images.
type: integer
default: 1
steps:
- run:
name: "DRIC: Setup cache"
command: |
if [[ -n "$CCI_DRIC_USERNAME" ]]; then
echo "$CCI_DRIC_PASSWORD" | \
docker login --username "$CCI_DRIC_USERNAME" --password-stdin "$CCI_DRIC_REGISTRY"
fi
filename="<<parameters.image-filename>>"
filename_with_repo="${filename}-with-repo"
# clear any previous runs
rm -f "${filename}" "${filename_with_repo}"
# save images list to a file
images="<<parameters.images>>"
mkdir -p "$(dirname $filename)"
echo $images | \
xargs -n 1 | \
uniq | \
xargs -n 1 echo >> "$filename"
filename_with_repo="${filename}-with-repo"
cp "$filename" "$filename_with_repo"
repo="<<parameters.repository>>"
if [[ -n "$repo" ]]; then
sed -i -e "s|^|${repo%/}/|" "$filename_with_repo"
fi
- run:
name: "DRIC: Restore cache from registry <<parameters.repository>>"
command: |
filename="<<parameters.image-filename>>"
filename_with_repo="${filename}-with-repo"
cat "$filename_with_repo" | xargs -r -P <<parameters.parallelism>> -n 1 docker pull || true
docker image ls -a
- steps: << parameters.steps >>
- run:
name: "DRIC: Retag images"
command: |
repo="<<parameters.repository>>"
if [[ -z "$repo" ]]; then
exit 0
fi
filename="<<parameters.image-filename>>"
filename_with_repo="${filename}-with-repo"
paste "$filename" "$filename_with_repo" | xargs -n 2 docker tag || true
docker image ls -a
- run:
name: "DRIC: Save cache to registry <<parameters.repository>>"
when: <<parameters.save-when>>
command: |
filename="<<parameters.image-filename>>"
filename_with_repo="${filename}-with-repo"
cat "${filename_with_repo}" | xargs -r -P <<parameters.parallelism>> -n 1 docker push || true
rm "${filename}" "${filename_with_repo}"
build:
description: |
Build a cache image adding a --cache-from parameter for every image in the list
provided to with-save-restore-images.
parameters:
command:
description: |
A `docker build` command to run. Should include a `--tag` parameter for
the image.
type: string
step-name:
description: |
Name for the step.
type: string
default: "docker build"
image-filename:
description: |
Path to a file used to store the list of images. Must match the path provided to
'with-save-restore-images'.
You most likely don't need to change this.
type: string
default: "/tmp/docker-cache/images"
steps:
- run:
name: <<parameters.step-name>>
command: |
set -x
docker() {
cache_from=$(sed -e 's|^|--cache-from=|' "<<parameters.image-filename>>-with-repo" | xargs)
case "$1" in
build) shift 1; command docker build $cache_from "$@";;
*) command docker "$@";;
esac
}
<<parameters.command>>
tag:
description: |
Tag a group of Docker images. To make the cache work when you are not using the orb's `build` command,
your build must produce Docker images with the tags listed in the `images` parameter. If you do not wish
to change your existing tagging, or you are using docker-compose or scripts where the image tags
cannot easily be controlled, then this command will help.
parameters:
tags:
description: |
A list of image tag pairs, separated by spaces, which will be passed to docker tag.
The first item in the pair is the source tag, and the second the destination.
type: string
step-name:
description: |
Name for the step.
type: string
default: "Tag docker images"
steps:
- run:
name: <<parameters.step-name>>
command: |
set -x
cat \<<EOF | xargs -n 2 docker tag
<<parameters.tags>>
EOF
examples:
cache_using_branch_name_key:
description: |
This example demonstrates how to use this orb to speed up a 'docker build'. It uses the git branch name as a cache key, and the master branch as a fallback when the branch is new.
The first step is to log into a GCR registry. The values for these REGISTRY_ variables may be set by project environment variables or from a CircleCI Context.
Once login is complete the 'with-save-restore-images' command pulls images from a registry. This example uses two images. An image tagged with 'master' which acts as a base for new branches, and an image tagged with the name of the git branch. This example removes any slashes from the git branch name using bash string substitution ('${var/\//-}'). The images are listed using the YAML '>-' symbol, which replaces newlines with spaces and removes trailing newlines. This allows you to list images one per line.
The nested step uses the 'build' command to build a new docker image. The orb command takes care of adding the '--cache-from' arguments to 'docker build' so that the images that were pulled are used as a cache. You may use as many nested steps as necessary. Only steps which build docker images need to be nested under 'with-save-restore-images'.
Once all the nested steps have completed the 'with-save-restore-images' command pushes all the 'images' to the same registry where it pulled from originally.
Finally, the rest of the job will continue and may use any of the images pulled or built in previous steps. In this example tests are run using a `test-all` script.
usage:
version: 2.1
orbs:
docker-cache: cci-x/docker-registry-image-cache@0.2.0
workflows:
ci:
jobs:
- build-docker-image
jobs:
build-docker-image:
machine: {image: 'ubuntu-1604:201903-01'}
steps:
- run:
name: docker login
command: |
echo "$REGISTRY_PASSWORD" | \
docker login --username "$REGISTRY_USERNAME" --password-stdin https://us.gcr.io
- docker-cache/with-save-restore-images:
repository: us.gcr.io/myrepo
images: >-
myapp:${CIRCLE_BRANCH/\//-}
myapp:master
steps:
- docker-cache/build:
command: docker build -t myapp:${CIRCLE_BRANCH/\//-} .
- run:
name: Test
command: ./test-all
cache_using_branch_name_key_with_multistage:
description: |
This example builds on cache_using_branch_name_key to demonstrate how to use this orb with Dockerfiles that use multi-stage builds.
The key thing to note is that you must build, cache, and restore each named stage of the build separately.
A Dockerfile with the intermediary stages base1 and base2 is assumed:
FROM ubuntu:latest AS base1
...
FROM base1 AS base2
...
FROM base2
...
usage:
version: 2.1
orbs:
docker-cache: cci-x/docker-registry-image-cache@0.2.0
workflows:
ci:
jobs:
- build-docker-image
jobs:
build-docker-image:
machine: {image: 'ubuntu-1604:201903-01'}
steps:
- run:
name: docker login
command: |
echo "$REGISTRY_PASSWORD" | \
docker login --username "$REGISTRY_USERNAME" --password-stdin https://us.gcr.io
- docker-cache/with-save-restore-images:
repository: us.gcr.io/myrepo
parallelism: 4
images: >-
cache-orb:master
cache-orb:base1-master
cache-orb:base2-master
cache-orb:${CIRCLE_BRANCH/\//-}
cache-orb:base1-${CIRCLE_BRANCH/\//-}
cache-orb:base2-${CIRCLE_BRANCH/\//-}
steps:
- docker-cache/build:
command: docker build -t cache-orb:base1-${CIRCLE_BRANCH/\//-} --target base1 .
- docker-cache/build:
command: docker build -t cache-orb:base2-${CIRCLE_BRANCH/\//-} --target base2 .
- docker-cache/build:
command: docker build -t cache-orb:${CIRCLE_BRANCH/\//-} .
- run:
name: Test
command: ./test-all
cache_using_hash_of_dockerfile_key:
description: |
This example demonstrates how to use this orb to speed up a 'docker build'. It uses the md5sum of the Dockerfile as a cache key. It follows all the same steps as the example above, except that it uses a different key for the cache. It assumes that an image with a 'latest' tag is pushed from the master CI job and can be used as a fallback when the Dockerfile changes.
usage:
version: 2.1
orbs:
docker-cache: cci-x/docker-registry-image-cache@0.2.0
workflows:
ci:
jobs:
- build-docker-image
jobs:
build-docker-image:
machine: {image: 'ubuntu-1604:201903-01'}
steps:
- run:
name: docker login
command: |
echo "$REGISTRY_PASSWORD" | \
docker login --username "$REGISTRY_USERNAME" --password-stdin
- run:
name: "Generate cache key"
command: |
md5sum Dockerfile | awk '{print $1}' > /tmp/cache-key
- docker-cache/with-save-restore-images:
repository: myhubrepo
images: >-
myapp:$(< /tmp/cache-key)
myapp:latest
steps:
- docker-cache/build:
command: docker build -t myapp:$(< /tmp/cache-key) .
- run:
name: Test
command: ./test-all