Use CircleCI version 2.1 at the top of your .circleci/config.yml file.
1
version: 2.1
Add the orbs
stanza below your version, invoking the orb:
1
2
orbs:
queue: eddiewebb/queue@3.2.1
Use queue
elements in your existing workflows and jobs.
Opt-in to use of uncertified orbs on your organization’s Security settings page.
Use regexp-jobname when you have multiple jobs to block order of.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: '2.1'
orbs:
queue: eddiewebb/queue@volatile
jobs:
DeployStep1:
docker:
- image: circleci/node:10
steps:
- queue/until_front_of_line:
job-regex: ^DeployStep[0-9]$
limit-branch-name: main
max-wait-time: '10'
my-pipeline: <<pipeline.number>>
- run: >-
echo "This job will not overlap with itself or next similar nameds
job"
DeployStep2:
docker:
- image: circleci/node:10
steps:
- run: echo "This job will block step1 on any further workflows"
build:
docker:
- image: circleci/node:10
steps:
- run: echo "This job can overlap"
workflows:
build_deploy:
jobs:
- build
- DeployStep1:
requires:
- build
- DeployStep2:
requires:
- DeployStep1
Used typically as first job and will queue until no previous workflows are running
1
2
3
4
5
6
7
8
9
10
11
12
13
version: '2.1'
orbs:
queue: eddiewebb/queue@volatile
workflows:
build_deploy:
jobs:
- queue/block_workflow:
limit-branch-name: main
max-wait-time: '10'
my-pipeline: <<pipeline.number>>
- some_other_job:
requires:
- queue/block_workflow
Used to ensure that a only single job (deploy) is not run concurrently. By default will only queue if the same job from previous worfklows is running on the same branch. This allows safe jobs like build/test to overlap, minimizing overall queue times.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
version: '2.1'
orbs:
queue: eddiewebb/queue@volatile
jobs:
build:
docker:
- image: circleci/node:10
steps:
- run: echo "This job can overlap"
deploy:
docker:
- image: circleci/node:10
steps:
- queue/until_front_of_line:
limit-branch-name: main
max-wait-time: '10'
my-pipeline: <<pipeline.number>>
- run: echo "This job will not overlap"
workflows:
build_deploy:
jobs:
- build
- deploy:
requires:
- build
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
block-workflow | If true, this job will block until no other workflows with an earlier timestamp are running. Typically used as first job. | No | true | boolean |
circleci-api-key | In case you use a different Environment Variable Name than CIRCLECI_API_KEY, supply it here. | No | CIRCLECI_API_KEY | env_var_name |
circleci-hostname | For server user to specifiy custom hostname for their server | No | circleci.com | string |
confidence | Due to scarce API, we need to requery the recent jobs list to ensure we're not just in a pending state for previous jobs. This number indicates the threhold for API returning no previous pending jobs. Default is a single confirmation. | No | '1' | string |
dont-quit | Quitting is for losers. Force job through once time expires instead of failing. | No | false | boolean |
fail-instead-of-cancel | Fail this command instead of canceling. | No | false | boolean |
force-cancel-previous | No Mercy. Issue cancel commands for any previous competitors (only applies when dont-quit also true) | No | false | boolean |
include-debug | - | No | false | boolean |
job-regex | Used to selectively block individual jobs in a workflow. ex '^deploy*' | No | '' | string |
limit-branch-name | Only apply queue logic on specified branch. | No | '*' | string |
limit-workflow-name | Only queue on a specified workflow. Consider combining this with `this-branch-only`:`false`. | No | '*' | string |
max-wait-time | How many minutes to wait before giving up. | No | '10' | string |
my-pipeline | - | Yes | - | integer |
tag-pattern | Set to queue jobs using a regex pattern f.ex '^v[0-9]+\.[0-9]+\.[0-9]+$' to filter CIRCLECI_TAG | No | '' | string |
this-branch-only | Should we only consider jobs running on the same branch? | No | true | boolean |
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
block-workflow | If true, this job will block until no other workflows with ANY JOBS with an earlier timestamp are running. Typically used as first job. | No | false | boolean |
circleci-api-key | In case you use a different Environment Variable Name than CIRCLECI_API_KEY, supply it here. | No | CIRCLECI_API_KEY | env_var_name |
circleci-hostname | For server user to specifiy custom hostname for their server | No | circleci.com | string |
confidence | Due to scarce API, we need to requery the recent jobs list to ensure we're not just in a pending state for previous jobs. This number indicates the threhold for API returning no previous pending jobs. Default is a single confirmation. | No | '1' | string |
dont-quit | Quitting is for losers. Force job through once time expires instead of failing. | No | false | boolean |
fail-instead-of-cancel | Fail this command instead of canceling. | No | false | boolean |
force-cancel-previous | No Mercy. Issue cancel commands for any previous competitors (only applies when dont-quit also true) | No | false | boolean |
include-debug | - | No | false | boolean |
job-regex | Used to selectively block individual jobs in a workflow. ex '^deploy*' | No | '' | string |
limit-branch-name | Only apply queue logic on specified branch. | No | '*' | string |
limit-workflow-name | Only queue on a specified workflow. Consider combining this with `this-branch-only`:`false`. | No | '*' | string |
max-wait-time | How many minutes to wait before giving up. | No | '10' | string |
my-pipeline | - | Yes | - | integer |
tag-pattern | Set to queue jobs using a regex pattern f.ex '^v[0-9]+\.[0-9]+\.[0-9]+$' to filter CIRCLECI_TAG | No | '' | string |
this-branch-only | Should we only consider jobs running on the same branch? | No | true | boolean |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
# This code is licensed from CircleCI to the user under the MIT license.
# See here for details: https://circleci.com/developer/orbs/licensing
version: 2.1
description: "Allows jobs or entire workflows to be queued to ensure they run in serial. \nThis is ideal for deployments or other activities that must not run concurrently.\nMay optionaly consider branch-level isolation if unique branches should run concurrently. \nThis orb requires the project to have an **Personal** API key in order to query build states.\nIt requires a single environment variable CIRCLECI_API_KEY which can be created in account settings - https://circleci.com/account/api.\n\n3.2.1: Docs improvement, PR #136 Thanks @RainOfTerra\n3.2.0: Adds back-off support to handle 429 throttling from new API. PR #134 Thanks @rainofterra\n3.1.6: Fix #128 again, this time on JQ side. PR #133 Thanks @RainOfTerra\n3.1.5: Fix #128: allow spaces in job names and other bash bugs in PR #131, thanks @RainOfTerra\n3.1.4: Fix missing branch error when using tags. Issue #126\n3.1.2: Fix github VCS switch (API calls failing) thanks @risinga PR #122\n3.1.1: Boolean fix finally isolated, thanks @@charlescqian\n3.1.0: Force fail, not cancel, when blocked. PR #113, thanks @pguinard-public-com\n3.0.0: [BREAKING CHANGE] Use pipeline.number as authoritative comparison. Deep thanks to @PChambino\n2.2.2: Docs clarity on token needs (@davidjb)\n2.2.1: fixes release version bug\n2.2.0: Adds 'filter-by-workflow' (@soniqua)\n2.0.0: Breaking change fixes dyanamic config, but may break Bitbucket users. \n1.12.0: Adds Server Support (@nanophate)\n1.9.0: Doc update\n1.8.4: Adds urlencode for branch names. (@andrew-barnett)\n1.8.1: Adds content-type header to API calls (@kevinr-electric) and prints message on error (@AlexMeuer)\n1.8.0: minor fix same as version 1.8.0 (missing docs)\n1.7.1: patch fix same as version 1.8.1 to catch folsk who dont update\n1.7.0: adds regexp for job names, collab with @jonesie100\n1.6.5: docs contribution by @pwilczynskiclearcode \n1.6.4: support slashes in Tags, thanks @dunial\n1.6.3: addresses API changes that broke branch-job queueing, adds more API checks\n1.6.1: fixes issue in tag matching , thanks @calvin-summer\n1.6.0: Support Tags, thanks @nikolaik, @dunial\n1.5.0: API variables name as parameter , thanks @philnielson\n1.4.4: Docs improvements, thanks @jordan-brough\n1.4.3: more confident confidence thanks @GreshamDanielStephens\n1.4.2: Doc improvements, thanks @olleolleolle\n1.4.1: fixes bug in block-workflow as job. thanks @mu-bro\n1.4.0: Adds confidence checks to avoid race condition \n1.3.0: use small resource class in job\n"
display:
home_url: https://eddiewebb.github.io/circleci-queue/
source_url: https://github.com/eddiewebb/circleci-queue
commands:
until_front_of_line:
parameters:
block-workflow:
default: false
description: If true, this job will block until no other workflows with ANY JOBS with an earlier timestamp are running. Typically used as first job.
type: boolean
circleci-api-key:
default: CIRCLECI_API_KEY
description: In case you use a different Environment Variable Name than CIRCLECI_API_KEY, supply it here.
type: env_var_name
circleci-hostname:
default: circleci.com
description: For server user to specifiy custom hostname for their server
type: string
confidence:
default: "1"
description: Due to scarce API, we need to requery the recent jobs list to ensure we're not just in a pending state for previous jobs. This number indicates the threhold for API returning no previous pending jobs. Default is a single confirmation.
type: string
dont-quit:
default: false
description: Quitting is for losers. Force job through once time expires instead of failing.
type: boolean
fail-instead-of-cancel:
default: false
description: Fail this command instead of canceling.
type: boolean
force-cancel-previous:
default: false
description: No Mercy. Issue cancel commands for any previous competitors (only applies when dont-quit also true)
type: boolean
include-debug:
default: false
type: boolean
job-regex:
default: ""
description: Used to selectively block individual jobs in a workflow. ex '^deploy*'
type: string
limit-branch-name:
default: '*'
description: 'Only apply queue logic on specified branch. '
type: string
limit-workflow-name:
default: '*'
description: Only queue on a specified workflow. Consider combining this with `this-branch-only`:`false`.
type: string
max-wait-time:
default: "10"
description: How many minutes to wait before giving up.
type: string
my-pipeline:
type: integer
tag-pattern:
default: ""
description: Set to queue jobs using a regex pattern f.ex '^v[0-9]+\.[0-9]+\.[0-9]+$' to filter CIRCLECI_TAG
type: string
this-branch-only:
default: true
description: Should we only consider jobs running on the same branch?
type: boolean
steps:
- run:
command: |
echo "export BLOCK_WORKFLOW=<<parameters.block-workflow>>" >> $BASH_ENV
echo "export CCI_API_KEY_NAME=<< parameters.circleci-api-key >>" >> $BASH_ENV
echo "export CIRCLECI_BASE_URL=https://<<parameters.circleci-hostname>>" >> $BASH_ENV
echo "export CONFIDENCE_THRESHOLD=<<parameters.confidence>>" >> $BASH_ENV
echo "export DEBUG=<<parameters.include-debug>>" >> $BASH_ENV
echo "export DONT_QUIT=<<parameters.dont-quit>>" >> $BASH_ENV
echo "export FAIL_INSTEAD_OF_CANCEL=<< parameters.fail-instead-of-cancel >>" >> $BASH_ENV
echo "export FILTER_BRANCH=<< parameters.this-branch-only >>" >> $BASH_ENV
echo "export FORCE_CANCEL_PREVIOUS=<<parameters.force-cancel-previous>>" >> $BASH_ENV
echo "export JOB_REGEXP=\"<<parameters.job-regex>>\"" >> $BASH_ENV
echo "export MAX_TIME='<<parameters.max-wait-time>>'" >> $BASH_ENV
echo "export MY_PIPELINE_NUMBER=<<parameters.my-pipeline>>" >> $BASH_ENV
echo "export ONLY_ON_BRANCH=<<parameters.limit-branch-name>>" >> $BASH_ENV
echo "export ONLY_ON_WORKFLOW=<<parameters.limit-workflow-name>>" >> $BASH_ENV
echo "export TAG_PATTERN=\"<<parameters.tag-pattern>>\"" >> $BASH_ENV
name: Queue - Import Parameters
- run:
command: |+
#!/bin/bash
#
# This script uses many environment variables, some set from pipeline parameters. See orb yaml for source.
#
load_variables(){
TMP_DIR=$(mktemp -d)
SHALLOW_JOBSTATUS_PATH="$TMP_DIR/jobstatus.json"
AUGMENTED_JOBSTATUS_PATH="$TMP_DIR/augmented_jobstatus.json"
echo "Block: $BLOCK_WORKFLOW"
: "${MAX_TIME:?"Required Env Variable not found!"}"
wait_time=0
loop_time=11
max_time_seconds=$(( 60 * $MAX_TIME ))
# just confirm our required variables are present
: "${CIRCLE_BUILD_NUM:?"Required Env Variable not found!"}"
: "${CIRCLE_PROJECT_USERNAME:?"Required Env Variable not found!"}"
: "${CIRCLE_PROJECT_REPONAME:?"Required Env Variable not found!"}"
: "${CIRCLE_REPOSITORY_URL:?"Required Env Variable not found!"}"
: "${CIRCLE_JOB:?"Required Env Variable not found!"}"
VCS_TYPE="github"
if [[ "$CIRCLE_REPOSITORY_URL" =~ .*bitbucket\.org.* ]]; then
VCS_TYPE="bitbucket"
echo "VCS_TYPE set to bitbucket"
fi
: "${VCS_TYPE:?"Required VCS TYPE not found! This is likely a bug in orb, please report."}"
: "${MY_PIPELINE_NUMBER:?"Required MY_PIPELINE_NUMBER not found! This is likely a bug in orb, please report."}"
# If a pattern is wrapped with slashes, remove them.
if [[ "$TAG_PATTERN" == /*/ ]]; then
TAG_PATTERN="${TAG_PATTERN:1:-1}"
fi
echo "Expecting CCI Personal Access TOKEN Named: $CCI_API_KEY_NAME"
CCI_TOKEN="${!CCI_API_KEY_NAME}"
# Only needed for private projects
if [ -z "$CCI_TOKEN" ]; then
echo "CCI_TOKEN not set. Private projects and force cancel will not function."
else
fetch "${CIRCLECI_BASE_URL}/api/v2/me" "$TMP_DIR/me.cci"
me=$(jq -e '.id' "$TMP_DIR/me.cci")
echo "Using API key for user: $me on host $CIRCLECI_BASE_URL"
fi
if [[ $DEBUG != "false" ]]; then
echo "Using Temp Dir: $TMP_DIR"
#set
fi
}
do_we_run(){
if [ -n "$CIRCLE_TAG" ] && [ -z "$TAG_PATTERN" ]; then
echo "TAG_PATTERN defined, but not on tagged run, skip queueing!"
exit 0
fi
if [ -n "$CIRCLE_PR_REPONAME" ]; then
echo "Queueing on forks is not supported. Skipping queue..."
# It's important that we not fail here because it could cause issues on the main repo's branch
exit 0
fi
if [[ "$ONLY_ON_BRANCH" == "*" ]] || [[ "$ONLY_ON_BRANCH" == "$CIRCLE_BRANCH" ]]; then
echo "$CIRCLE_BRANCH matches queueable branch names"
else
echo "Queueing only enforced on branch '$ONLY_ON_BRANCH', skipping queue"
exit 0
fi
}
update_active_run_data(){
fetch_filtered_active_builds
augment_jobs_with_pipeline_data
JOB_NAME="$CIRCLE_JOB"
if [ -n "$JOB_REGEXP" ]; then
JOB_NAME="$JOB_REGEXP"
use_regex=true
else
use_regex=false
fi
# falsey parameters are empty strings, so always compare against 'true'
if [ "$BLOCK_WORKFLOW" != "false" ]; then
echo "Orb parameter block-workflow is true. Any previous (matching) pipelines with running workflows will block this entire workflow."
if [ "$ONLY_ON_WORKFLOW" = "*" ]; then
echo "No workflow name filter. This job will block until no previous workflows with *any* name are running, regardless of job name."
oldest_running_build_num=$(jq 'sort_by(.workflows.pipeline_number)| .[0].build_num' "$AUGMENTED_JOBSTATUS_PATH")
front_of_queue_pipeline_number=$(jq -r 'sort_by(.workflows.pipeline_number)| .[0].workflows.pipeline_number // empty' "$AUGMENTED_JOBSTATUS_PATH")
else
echo "Orb parameter limit-workflow-name is provided."
echo "This job will block until no previous occurrences of workflow $ONLY_ON_WORKFLOW are running, regardless of job name"
oldest_running_build_num=$(jq --arg ONLY_ON_WORKFLOW "$ONLY_ON_WORKFLOW" '. | map(select(.workflows.workflow_name == $ONLY_ON_WORKFLOW)) | sort_by(.workflows.pipeline_number) | .[0].build_num' "$AUGMENTED_JOBSTATUS_PATH")
front_of_queue_pipeline_number=$(jq -r --arg ONLY_ON_WORKFLOW "$ONLY_ON_WORKFLOW" '. | map(select(.workflows.workflow_name == $ONLY_ON_WORKFLOW)) | sort_by(.workflows.pipeline_number) | .[0].workflows.pipeline_number // empty' "$AUGMENTED_JOBSTATUS_PATH")
fi
else
echo "Orb parameter block-workflow is false. Use Job level queueing."
echo "Only blocking execution if running previous jobs matching this job: $JOB_NAME"
if [ "$use_regex" = true ]; then
oldest_running_build_num=$(jq --arg JOB_NAME "$JOB_NAME" '. | map(select(.workflows.job_name | test($JOB_NAME; "sx"))) | sort_by(.workflows.pipeline_number) | .[0].build_num' "$AUGMENTED_JOBSTATUS_PATH")
front_of_queue_pipeline_number=$(jq -r --arg JOB_NAME "$JOB_NAME" '. | map(select(.workflows.job_name | test($JOB_NAME; "sx"))) | sort_by(.workflows.pipeline_number) | .[0].workflows.pipeline_number // empty' "$AUGMENTED_JOBSTATUS_PATH")
else
oldest_running_build_num=$(jq --arg JOB_NAME "$JOB_NAME" '. | map(select(.workflows.job_name == $JOB_NAME)) | sort_by(.workflows.pipeline_number) | .[0].build_num' "$AUGMENTED_JOBSTATUS_PATH")
front_of_queue_pipeline_number=$(jq -r --arg JOB_NAME "$JOB_NAME" '. | map(select(.workflows.job_name == $JOB_NAME)) | sort_by(.workflows.pipeline_number) | .[0].workflows.pipeline_number // empty' "$AUGMENTED_JOBSTATUS_PATH")
fi
if [[ "$DEBUG" != "false" ]]; then
echo "DEBUG: me: $MY_PIPELINE_NUMBER, front: $front_of_queue_pipeline_number"
fi
fi
if [ -z "$front_of_queue_pipeline_number" ]; then
echo "API Call for existing jobs returned no matches. This means job is alone."
if [[ $DEBUG != "false" ]]; then
echo "All running jobs:"
cat "$SHALLOW_JOBSTATUS_PATH" || exit 0
echo "All running jobs with created_at:"
cat "$AUGMENTED_JOBSTATUS_PATH" || exit 0
echo "All workflow details."
cat /tmp/workflow-*.json || echo "Could not load workflows.."
exit 1
fi
fi
}
fetch_filtered_active_builds(){
JOB_API_SUFFIX="?filter=running&shallow=true"
jobs_api_url_template="${CIRCLECI_BASE_URL}/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}${JOB_API_SUFFIX}"
if [ "$FILTER_BRANCH" == "false" ]; then
echo "Orb parameter 'this-branch-only' is false, will block previous builds on any branch."
else
# branch filter
: "${CIRCLE_BRANCH:?"Required Env Variable not found!"}"
echo "Only blocking execution if running previous jobs on branch: $CIRCLE_BRANCH"
jobs_api_url_template="${CIRCLECI_BASE_URL}/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/tree/$(urlencode "$CIRCLE_BRANCH")${JOB_API_SUFFIX}"
fi
if [ -n "$TESTING_MOCK_RESPONSE" ] && [ -f "$TESTING_MOCK_RESPONSE" ]; then
echo "Using test mock response"
cat "$TESTING_MOCK_RESPONSE" > "$SHALLOW_JOBSTATUS_PATH"
else
fetch "$jobs_api_url_template" "$SHALLOW_JOBSTATUS_PATH"
fi
if [ -n "$CIRCLE_TAG" ] && [ -n "$TAG_PATTERN" ]; then
echo "TAG_PATTERN variable non-empty, will only block pipelines with matching tag"
jq "[ .[] | select((.build_num | . == \"${CIRCLE_BUILD_NUM}\") or (.vcs_tag | (. != null and test(\"${TAG_PATTERN}\"))) ) ]" "$SHALLOW_JOBSTATUS_PATH" > /tmp/jobstatus_tag.json
mv /tmp/jobstatus_tag.json "$SHALLOW_JOBSTATUS_PATH"
fi
}
augment_jobs_with_pipeline_data(){
echo "Getting queue ordering"
cp "$SHALLOW_JOBSTATUS_PATH" "$AUGMENTED_JOBSTATUS_PATH"
for workflow in $(jq -r ".[] | .workflows.workflow_id //empty" "$AUGMENTED_JOBSTATUS_PATH" | uniq); do
# get workflow to get pipeline...
workflow_file="${TMP_DIR}/workflow-${workflow}.json"
if [ -f "$TESTING_MOCK_WORKFLOW_RESPONSES/${workflow}.json" ]; then
echo "Using test mock workflow response"
cat "$TESTING_MOCK_WORKFLOW_RESPONSES/${workflow}.json" > "${workflow_file}"
else
fetch "${CIRCLECI_BASE_URL}/api/v2/workflow/${workflow}" "${workflow_file}"
fi
pipeline_number=$(jq -r '.pipeline_number' "${workflow_file}")
echo "Workflow: ${workflow} is from pipeline #${pipeline_number}"
jq --arg pipeline_number "${pipeline_number}" --arg workflow "${workflow}" '(.[] | select(.workflows.workflow_id == $workflow) | .workflows) |= . + {pipeline_number:$pipeline_number}' "$AUGMENTED_JOBSTATUS_PATH" > "${TMP_DIR}/augmented_jobstatus-${workflow}.json"
mv "${TMP_DIR}/augmented_jobstatus-${workflow}.json" "$AUGMENTED_JOBSTATUS_PATH"
done
}
urlencode(){
LC_WAS="${LC_ALL:-}"
export LC_ALL=C
string="$1"
while [ -n "$string" ]; do
tail="${string#?}"
head="${string%"$tail"}"
case "$head" in
[-_.~A-Za-z0-9]) printf '%c' "$head" ;;
*) printf '%%%02X' "'$head" ;;
esac
string="${tail}"
done
echo
export LC_ALL="${LC_WAS}"
}
fetch() {
local max_retries=5
local retry_count=0
local backoff=1
while : ; do
if [[ $DEBUG != "false" ]]; then
echo "DEBUG: Making API Call to ${1}"
fi
url="$1"
target="$2"
response_headers=$(mktemp)
http_response=$(curl -s -X GET -H "Circle-Token:${CCI_TOKEN}" -H "Content-Type: application/json" -D "$response_headers" -o "${target}" -w "%{http_code}" "${url}")
if [ "$http_response" -eq 200 ]; then
if [[ $DEBUG != "false" ]]; then
echo "DEBUG: API Success"
fi
rm -f "$response_headers"
return 0
elif [ "$http_response" -eq 429 ]; then
retry_after=$(grep -i "Retry-After:" "$response_headers" | awk '{print $2}' | tr -d '\r')
if [[ -n "$retry_after" ]]; then
sleep_duration=$((retry_after))
else
sleep_duration=$((backoff))
backoff=$((backoff * 2))
fi
if (( retry_count >= max_retries )); then
echo "ERROR: Maximum retries reached. Exiting."
rm -f "$response_headers"
cat "${target}"
exit 1
fi
if [[ $DEBUG != "false" ]]; then
echo "DEBUG: Rate limit exceeded. Retrying in $sleep_duration seconds..."
fi
sleep "$sleep_duration"
((retry_count++))
else
echo "ERROR: Server returned error code: $http_response"
rm -f "$response_headers"
cat "${target}"
exit 1
fi
done
}
cancel_build_num(){
BUILD_NUM="$1"
echo "Cancelling build ${BUILD_NUM}"
cancel_api_url_template="${CIRCLECI_BASE_URL}/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/${BUILD_NUM}/cancel?circle-token=${CCI_TOKEN}"
curl -s -X POST "$cancel_api_url_template" > /dev/null
}
#
# MAIN LOGIC STARTS HERE
#
load_variables
do_we_run # exit early if we can
echo "Max Queue Time: ${max_time_seconds} seconds."
#
# Queue Loop
#
confidence=0
while true; do
# get running jobs, filtered to branch or tag, with pipeline ID
update_active_run_data
echo "This Job's Pipeline #: $MY_PIPELINE_NUMBER"
echo "Front of Queue (fifo) Pipeline #: $front_of_queue_pipeline_number"
# This condition checks if the current job should proceed based on confidence level:
# 1. If 'front_of_queue_pipeline_number' is empty, it means there are no other jobs in the queue, so the current job can proceed.
# 2. If 'MY_PIPELINE_NUMBER' is non-empty and equals 'front_of_queue_pipeline_number', it means the current job is at the front of the queue and can proceed.
# Confidence level is incremented if either of these conditions is true.
if [[ -z "$front_of_queue_pipeline_number" ]] || ([[ -n "$MY_PIPELINE_NUMBER" ]] && [[ "$front_of_queue_pipeline_number" == "$MY_PIPELINE_NUMBER" ]]); then
# recent-jobs API does not include pending, so it is possible we queried in between a workflow transition, and we're NOT really front of line.
if [ $confidence -lt "$CONFIDENCE_THRESHOLD" ]; then
# To grow confidence, we check again with a delay.
confidence=$((confidence+1))
echo "API shows no conflicting jobs/workflows. However it is possible a previous workflow has pending jobs not yet visible in API. To avoid a race condition we will verify our place in queue."
echo "Rerunning check ${confidence}/$CONFIDENCE_THRESHOLD"
else
echo "Front of the line, WooHoo!, Build continuing"
break
fi
else
# If we fail, reset confidence
confidence=0
echo "This build (${CIRCLE_BUILD_NUM}), pipeline (${MY_PIPELINE_NUMBER}) is queued, waiting for build(${oldest_running_build_num}) pipeline (${front_of_queue_pipeline_number}) to complete."
echo "Total Queue time: ${wait_time} seconds."
fi
if [ $wait_time -ge $max_time_seconds ]; then
echo "Max wait time exceeded, fail or force cancel..."
if [ "${DONT_QUIT}" != "false" ]; then
echo "Orb parameter dont-quit is set to true, letting this job proceed!"
if [ "${FORCE_CANCEL_PREVIOUS}" != "false" ]; then
echo "FEATURE NOT IMPLEMENTED"
exit 1
fi
exit 0
else
if [ "$FAIL_INSTEAD_OF_CANCEL" != "true" ]; then
cancel_build_num "$CIRCLE_BUILD_NUM"
sleep 5 # wait for API to cancel this job, rather than showing as failure
# but just in case, fail job
fi
exit 1
fi
fi
sleep $loop_time
wait_time=$(( loop_time + wait_time ))
done
name: Queue Until Front of Line
shell: bash
jobs:
block_workflow:
docker:
- image: cimg/base:stable
parameters:
block-workflow:
default: true
description: If true, this job will block until no other workflows with an earlier timestamp are running. Typically used as first job.
type: boolean
circleci-api-key:
default: CIRCLECI_API_KEY
description: In case you use a different Environment Variable Name than CIRCLECI_API_KEY, supply it here.
type: env_var_name
circleci-hostname:
default: circleci.com
description: For server user to specifiy custom hostname for their server
type: string
confidence:
default: "1"
description: Due to scarce API, we need to requery the recent jobs list to ensure we're not just in a pending state for previous jobs. This number indicates the threhold for API returning no previous pending jobs. Default is a single confirmation.
type: string
dont-quit:
default: false
description: Quitting is for losers. Force job through once time expires instead of failing.
type: boolean
fail-instead-of-cancel:
default: false
description: Fail this command instead of canceling.
type: boolean
force-cancel-previous:
default: false
description: No Mercy. Issue cancel commands for any previous competitors (only applies when dont-quit also true)
type: boolean
include-debug:
default: false
type: boolean
job-regex:
default: ""
description: Used to selectively block individual jobs in a workflow. ex '^deploy*'
type: string
limit-branch-name:
default: '*'
description: 'Only apply queue logic on specified branch. '
type: string
limit-workflow-name:
default: '*'
description: Only queue on a specified workflow. Consider combining this with `this-branch-only`:`false`.
type: string
max-wait-time:
default: "10"
description: How many minutes to wait before giving up.
type: string
my-pipeline:
type: integer
tag-pattern:
default: ""
description: Set to queue jobs using a regex pattern f.ex '^v[0-9]+\.[0-9]+\.[0-9]+$' to filter CIRCLECI_TAG
type: string
this-branch-only:
default: true
description: Should we only consider jobs running on the same branch?
type: boolean
resource_class: small
steps:
- until_front_of_line:
block-workflow: <<parameters.block-workflow>>
circleci-api-key: <<parameters.circleci-api-key>>
circleci-hostname: <<parameters.circleci-hostname>>
confidence: <<parameters.confidence>>
dont-quit: <<parameters.dont-quit>>
fail-instead-of-cancel: << parameters.fail-instead-of-cancel >>
include-debug: <<parameters.include-debug>>
job-regex: <<parameters.job-regex>>
limit-branch-name: <<parameters.limit-branch-name>>
limit-workflow-name: <<parameters.limit-workflow-name>>
max-wait-time: <<parameters.max-wait-time>>
my-pipeline: <<parameters.my-pipeline>>
tag-pattern: <<parameters.tag-pattern>>
this-branch-only: <<parameters.this-branch-only>>
examples:
multiple_job_names_regexp:
description: Use regexp-jobname when you have multiple jobs to block order of.
usage:
version: "2.1"
orbs:
queue: eddiewebb/queue@volatile
jobs:
DeployStep1:
docker:
- image: circleci/node:10
steps:
- queue/until_front_of_line:
job-regex: ^DeployStep[0-9]$
limit-branch-name: main
max-wait-time: "10"
my-pipeline: <<pipeline.number>>
- run: echo "This job will not overlap with itself or next similar nameds job"
DeployStep2:
docker:
- image: circleci/node:10
steps:
- run: echo "This job will block step1 on any further workflows"
build:
docker:
- image: circleci/node:10
steps:
- run: echo "This job can overlap"
workflows:
build_deploy:
jobs:
- build
- DeployStep1:
requires:
- build
- DeployStep2:
requires:
- DeployStep1
queue_workflow:
description: Used typically as first job and will queue until no previous workflows are running
usage:
version: "2.1"
orbs:
queue: eddiewebb/queue@volatile
workflows:
build_deploy:
jobs:
- queue/block_workflow:
limit-branch-name: main
max-wait-time: "10"
my-pipeline: <<pipeline.number>>
- some_other_job:
requires:
- queue/block_workflow
single_concurrency_job:
description: "Used to ensure that a only single job (deploy) is not run concurrently. \nBy default will only queue if the same job from previous worfklows is running on the same branch. \nThis allows safe jobs like build/test to overlap, minimizing overall queue times.\n"
usage:
version: "2.1"
orbs:
queue: eddiewebb/queue@volatile
jobs:
build:
docker:
- image: circleci/node:10
steps:
- run: echo "This job can overlap"
deploy:
docker:
- image: circleci/node:10
steps:
- queue/until_front_of_line:
limit-branch-name: main
max-wait-time: "10"
my-pipeline: <<pipeline.number>>
- run: echo "This job will not overlap"
workflows:
build_deploy:
jobs:
- build
- deploy:
requires:
- build