Use CircleCI version 2.1 at the top of your .circleci/config.yml file.
1
version: 2.1
Add the orbs
stanza below your version, invoking the orb:
1
2
orbs:
on-demand-runner: cci-labs/on-demand-runner@1.0.0
Use on-demand-runner
elements in your existing workflows and jobs.
Opt-in to use of uncertified orbs on your organization’s Security settings page.
Deploy an On Demand Runner, Do Testing on the Runner and Destroy the On Demand Runner
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
version: '2.1'
orbs:
on-demand-runner: cci-labs/on-demand-runner@1.0.0
jobs:
run-test-on-runner:
machine: true
resource_class: NAMESPACE/<< pipeline.id >>
steps:
- run: echo "Hello World! I am running on an On Demand Runner!"
workflows:
example-workflow:
jobs:
- on-demand-runner/deploy:
context: aws-creds
resource_class: << pipeline.id >>
- run-test-on-runner:
requires:
- on-demand-runner/deploy
- on-demand-runner/destroy:
context: aws-creds
requires:
- run-test-on-runner
resource_class: << pipeline.id >>
Provision and Configure an On Demand Runner Required Environment Variables: - AWS_AMI_ID - AWS_SECURITY_GROUP_ID - AWS_SSH_KEY - CIRCLECI_NAMESPACE - CIRCLE_TOKEN - AWS_ACCESS_KEY_ID - AWS_SECRET_ACCESS_KEY
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
aws-access-key-id | AWS access key id for IAM role. Set this to the name of
the environment variable you will use to hold this
value, i.e. AWS_ACCESS_KEY_ID.
| No | AWS_ACCESS_KEY_ID | env_var_name |
aws-ami-id | An AWS AMI(Amazon Machine Image) ID for the image you want
spun up as an EC2 instance. Set this to the name of
the environment variable you will use to hold this
value, i.e. AWS_AMI_ID.
| No | AWS_AMI_ID | env_var_name |
aws-instance-type | A valid instance type/size for a EC2 instance. If no instance type is provided
a t2.mirco will be selected. Set this to the name of the environment variable
you will use to hold this value, i.e. AWS_INSTANCE_TYPE.
| No | AWS_INSTANCE_TYPE | env_var_name |
aws-region | Env var of AWS region to operate in. If no region is provided US-EAST-1
will be selected.
| No | AWS_DEFAULT_REGION | env_var_name |
aws-secret-access-key | AWS secret key for IAM role. Set this to the name of
the environment variable you will use to hold this
value, i.e. $AWS_SECRET_ACCESS_KEY.
| No | AWS_SECRET_ACCESS_KEY | env_var_name |
aws-security-group-id | The ID of the AWS security group you want to be applied
to the EC2 instance(s). Set this to the name of
the environment variable you will use to hold this
value, i.e. AWS_SECURITY_GROUP_ID.
| No | AWS_SECURITY_GROUP_ID | env_var_name |
aws-ssh-key | The name of the SSH key you want the EC2 instance(s) to have
injected. Please note you need to already have uploaded the key
to AWS to reference the SSH key name. Set this to the name of
the environment variable you will use to hold this value, i.e. AWS_SSH_KEY.
| No | AWS_SSH_KEY | env_var_name |
aws-vpc-id | The ID of the VPC you want the EC2 instance(s) to be
deployed in. If left blank, the default VPC will be selected.
Set this to the name of the environment variable you will use
to hold this value, i.e. AWS_VPC_ID.
| No | AWS_VPC_ID | env_var_name |
circle-token | A valid Circle token for interacting with the CircleCI API. Please see
https://circleci.com/docs/2.0/managing-api-tokens/ for more details.
| No | CIRCLE_TOKEN | env_var_name |
circleci-namespace | A CircleCI generated namespace for your organization. Please see
https://circleci.com/docs/2.0/orb-concepts/#namespaces for more details.
| No | CIRCLECI_NAMESPACE | env_var_name |
number_of_runners | Number of on demand runners to deploy. If no number is provided only 1
on demand runner will be deployed. PLEASE be careful setting this number.
| No | NUMBER_OF_RUNNERS | env_var_name |
number_of_waits | Number of on waits(10 seconds each) for the EC2 instance(s) to be provisioned before failure.
If no number is provided 18 waits(3 minutes) will be selected.
| No | NUMBER_OF_WAITS | env_var_name |
resource_class | Name of the resource class that will be generated and used for On Demand Runners.
If you do not have a resource class to use, please pass in 'pipeline.id'. Generally, it best
practice to pass in 'pipeline.id'. PLEASE note you need to set resource class to a
name that is not currently being used, as the resource class will be deleted.
| Yes | - | string |
runner_user | The username the runner agent will utilize to run its service. If no username is given 'circleci' will be used.
| No | RUNNER_USER | env_var_name |
tag | The `cimg/python` Docker image version tag. | No | '3.10' | string |
version | Ansible Version To Be Installed | No | 2.9.* | string |
Deprovision and Remove an On Demand Runner Required Environment Variables: - AWS_SSH_KEY - CIRCLECI_NAMESPACE - CIRCLE_TOKEN - AWS_ACCESS_KEY_ID - AWS_SECRET_ACCESS_KEY
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
aws-access-key-id | AWS access key id for IAM role. Set this to the name of
the environment variable you will use to hold this
value, i.e. AWS_ACCESS_KEY_ID.
| No | AWS_ACCESS_KEY_ID | env_var_name |
aws-region | Env var of AWS region to operate in. If no region is provided US-EAST-1
will be selected.
| No | AWS_DEFAULT_REGION | env_var_name |
aws-secret-access-key | AWS secret key for IAM role. Set this to the name of
the environment variable you will use to hold this
value, i.e. $AWS_SECRET_ACCESS_KEY.
| No | AWS_SECRET_ACCESS_KEY | env_var_name |
aws-ssh-key | The name of the SSH key you want the EC2 instance(s) to have
injected. Please note you need to already have uploaded the key
to AWS to reference the SSH key name. Set this to the name of
the environment variable you will use to hold this value, i.e. AWS_SSH_KEY.
| No | AWS_SSH_KEY | env_var_name |
circle-token | A valid Circle token for interacting with the CircleCI API. Please see
https://circleci.com/docs/2.0/managing-api-tokens/ for more details.
| No | CIRCLE_TOKEN | env_var_name |
circleci-namespace | A CircleCI generated namespace for your organization. Please see
https://circleci.com/docs/2.0/orb-concepts/#namespaces for more details.
| No | CIRCLECI_NAMESPACE | env_var_name |
number_of_runners | Number of on demand runners to deploy. If no number is provided only 1
on demand runner will be deployed. PLEASE be careful setting this number.
| No | NUMBER_OF_RUNNERS | env_var_name |
number_of_waits | Number of on waits(10 seconds each) for the EC2 instance(s) to be provisioned before failure.
If no number is provided 18 waits(3 minutes) will be selected.
| No | NUMBER_OF_WAITS | env_var_name |
resource_class | Name of the resource class that will be generated and used for an On Demand Runner.
If you do not have a resource class to use, please pass in 'pipeline.id'. Generally, it best
practice to pass in 'pipeline.id'. PLEASE note you need to set resource class to a
name that is not currently being used, as the resource class will be deleted.
| Yes | - | string |
tag | The `cimg/python` Docker image version tag. | No | '3.10' | string |
version | Ansible Version To Be Installed | No | 2.9.* | string |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
# This code is licensed from CircleCI to the user under the MIT license.
# See here for details: https://circleci.com/developer/orbs/licensing
version: 2.1
description: |
A orb that lets users spin up runners on demand durning a pipeline. Users are able to provision an EC2 instance on the fly, install the runner agent, and allow the runner to accept units of work.
display:
home_url: https://github.com/CircleCI-Public/on-demand-runner-orb
source_url: https://github.com/CircleCI-Public/on-demand-runner-orb
jobs:
deploy:
description: |
Provision and Configure an On Demand Runner Required Environment Variables:
- AWS_AMI_ID
- AWS_SECURITY_GROUP_ID
- AWS_SSH_KEY
- CIRCLECI_NAMESPACE
- CIRCLE_TOKEN
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
docker:
- image: cimg/python:<< parameters.tag >>
parameters:
aws-access-key-id:
default: AWS_ACCESS_KEY_ID
description: |
AWS access key id for IAM role. Set this to the name of
the environment variable you will use to hold this
value, i.e. AWS_ACCESS_KEY_ID.
type: env_var_name
aws-ami-id:
default: AWS_AMI_ID
description: |
An AWS AMI(Amazon Machine Image) ID for the image you want
spun up as an EC2 instance. Set this to the name of
the environment variable you will use to hold this
value, i.e. AWS_AMI_ID.
type: env_var_name
aws-instance-type:
default: AWS_INSTANCE_TYPE
description: |
A valid instance type/size for a EC2 instance. If no instance type is provided
a t2.mirco will be selected. Set this to the name of the environment variable
you will use to hold this value, i.e. AWS_INSTANCE_TYPE.
type: env_var_name
aws-region:
default: AWS_DEFAULT_REGION
description: |
Env var of AWS region to operate in. If no region is provided US-EAST-1
will be selected.
type: env_var_name
aws-secret-access-key:
default: AWS_SECRET_ACCESS_KEY
description: |
AWS secret key for IAM role. Set this to the name of
the environment variable you will use to hold this
value, i.e. $AWS_SECRET_ACCESS_KEY.
type: env_var_name
aws-security-group-id:
default: AWS_SECURITY_GROUP_ID
description: |
The ID of the AWS security group you want to be applied
to the EC2 instance(s). Set this to the name of
the environment variable you will use to hold this
value, i.e. AWS_SECURITY_GROUP_ID.
type: env_var_name
aws-ssh-key:
default: AWS_SSH_KEY
description: |
The name of the SSH key you want the EC2 instance(s) to have
injected. Please note you need to already have uploaded the key
to AWS to reference the SSH key name. Set this to the name of
the environment variable you will use to hold this value, i.e. AWS_SSH_KEY.
type: env_var_name
aws-vpc-id:
default: AWS_VPC_ID
description: |
The ID of the VPC you want the EC2 instance(s) to be
deployed in. If left blank, the default VPC will be selected.
Set this to the name of the environment variable you will use
to hold this value, i.e. AWS_VPC_ID.
type: env_var_name
circle-token:
default: CIRCLE_TOKEN
description: |
A valid Circle token for interacting with the CircleCI API. Please see
https://circleci.com/docs/2.0/managing-api-tokens/ for more details.
type: env_var_name
circleci-namespace:
default: CIRCLECI_NAMESPACE
description: |
A CircleCI generated namespace for your organization. Please see
https://circleci.com/docs/2.0/orb-concepts/#namespaces for more details.
type: env_var_name
number_of_runners:
default: NUMBER_OF_RUNNERS
description: |
Number of on demand runners to deploy. If no number is provided only 1
on demand runner will be deployed. PLEASE be careful setting this number.
type: env_var_name
number_of_waits:
default: NUMBER_OF_WAITS
description: |
Number of on waits(10 seconds each) for the EC2 instance(s) to be provisioned before failure.
If no number is provided 18 waits(3 minutes) will be selected.
type: env_var_name
resource_class:
description: |
Name of the resource class that will be generated and used for On Demand Runners.
If you do not have a resource class to use, please pass in 'pipeline.id'. Generally, it best
practice to pass in 'pipeline.id'. PLEASE note you need to set resource class to a
name that is not currently being used, as the resource class will be deleted.
type: string
runner_user:
default: RUNNER_USER
description: |
The username the runner agent will utilize to run its service. If no username is given 'circleci' will be used.
type: env_var_name
tag:
default: "3.10"
description: The `cimg/python` Docker image version tag.
type: string
version:
default: 2.9.*
description: Ansible Version To Be Installed
type: string
steps:
- run:
command: |-
#!/bin/bash
pip install "ansible==${ORB_PARAM_ANSIBLE_VERSION}"
pip install boto3 botocore --upgrade
ansible-galaxy collection install amazon.aws community.general
environment:
ORB_PARAM_ANSIBLE_VERSION: <<parameters.version>>
name: Installing Dependencies
- run:
command: |
{
echo "export AWS_DEFAULT_REGION=$(echo ${<< parameters.aws-region >>:-us-east-1})"
echo "export AWS_INSTANCE_TYPE=$(echo ${<< parameters.aws-instance-type >>:-t2.micro})"
echo "export RESOURCE_CLASS=$(echo ${RESOURCE_CLASS:-<< parameters.resource_class >>})"
echo "export NUMBER_OF_RUNNERS=$(echo ${<< parameters.number_of_runners >>:-1})"
echo "export NUMBER_OF_WAITS=$(echo ${<< parameters.number_of_waits >>:-18})"
echo "export ANSIBLE_FORCE_COLOR=$(echo ${ANSIBLE_FORCE_COLOR:-true})"
echo "export ANSIBLE_GIT_URL=$(echo ${ANSIBLE_GIT_URL:-https://github.com/CircleCI-Public/on-demand-runner-orb.git})"
echo "export ANSIBLE_GIT_BRANCH=$(echo ${ANSIBLE_GIT_BRANCH:-main})"
echo "export ANSIBLE_HOST_KEY_CHECKING=$(echo ${ANSIBLE_HOST_KEY_CHECKING:-false})"
echo "export RESOURCE_CLASS_CREATION=$(echo ${RESOURCE_CLASS_CREATION:-true})"
echo "export RUNNER_TOKEN_CREATION=$(echo ${RUNNER_TOKEN_CREATION:-true})"
echo "export TEMPLATE_DYNAMIC_INVENTORY=$(echo ${TEMPLATE_DYNAMIC_INVENTORY:-true})"
echo "export CLOUD_PROVIDER=$(echo ${CLOUD_PROVIDER:-aws})"
echo "export CIRCLECI_USER=$(echo ${<< parameters.runner_user >>:-circleci})"
} >> $BASH_ENV
name: Setting Defaults if Variables are not Set
- run:
command: |
git clone ${ANSIBLE_GIT_URL} && cd ./on-demand-runner-orb && git checkout ${ANSIBLE_GIT_BRANCH}
name: Clone On Demand Runner Ansible from Github
- run:
command: |
ansible-playbook provision-on-demand-runner.yml -e "REGION=$AWS_DEFAULT_REGION AMI_ID=$<< parameters.aws-ami-id >> SECURITY_GROUP=$<< parameters.aws-security-group-id >> VPC_ID=$<< parameters.aws-vpc-id >> SSH_KEY_NAME=$<< parameters.aws-ssh-key >> INSTANCE_TYPE=$AWS_INSTANCE_TYPE NAMESPACE=$<< parameters.circleci-namespace >> RESOURCE_CLASS=$RESOURCE_CLASS NUMBER_OF_RUNNERS=$NUMBER_OF_RUNNERS UNTIL_RETRIES_RUNNERS=$NUMBER_OF_WAITS TEMPLATE_DYNAMIC_INVENTORY=$TEMPLATE_DYNAMIC_INVENTORY CLOUD_PROVIDER=$CLOUD_PROVIDER"
name: Provision an On Demand Runner
working_directory: ~/project/on-demand-runner-orb/auto-runner/
- run:
command: |
ansible-playbook -i aws_ec2.yml install-runner.yml -u ansible -e "NAMESPACE=$<< parameters.circleci-namespace >> RESOURCE_CLASS=$RESOURCE_CLASS CIRCLE_TOKEN=$<< parameters.circle-token >> RESOURCE_CLASS_CREATION=$RESOURCE_CLASS_CREATION RUNNER_TOKEN_CREATION=$RUNNER_TOKEN_CREATION target_hosts=all CIRCLECI_USER=$CIRCLECI_USER"
name: Configure the On Demand Runner
working_directory: ~/project/on-demand-runner-orb/auto-runner/
destroy:
description: |
Deprovision and Remove an On Demand Runner Required Environment Variables:
- AWS_SSH_KEY
- CIRCLECI_NAMESPACE
- CIRCLE_TOKEN
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
docker:
- image: cimg/python:<< parameters.tag >>
parameters:
aws-access-key-id:
default: AWS_ACCESS_KEY_ID
description: |
AWS access key id for IAM role. Set this to the name of
the environment variable you will use to hold this
value, i.e. AWS_ACCESS_KEY_ID.
type: env_var_name
aws-region:
default: AWS_DEFAULT_REGION
description: |
Env var of AWS region to operate in. If no region is provided US-EAST-1
will be selected.
type: env_var_name
aws-secret-access-key:
default: AWS_SECRET_ACCESS_KEY
description: |
AWS secret key for IAM role. Set this to the name of
the environment variable you will use to hold this
value, i.e. $AWS_SECRET_ACCESS_KEY.
type: env_var_name
aws-ssh-key:
default: AWS_SSH_KEY
description: |
The name of the SSH key you want the EC2 instance(s) to have
injected. Please note you need to already have uploaded the key
to AWS to reference the SSH key name. Set this to the name of
the environment variable you will use to hold this value, i.e. AWS_SSH_KEY.
type: env_var_name
circle-token:
default: CIRCLE_TOKEN
description: |
A valid Circle token for interacting with the CircleCI API. Please see
https://circleci.com/docs/2.0/managing-api-tokens/ for more details.
type: env_var_name
circleci-namespace:
default: CIRCLECI_NAMESPACE
description: |
A CircleCI generated namespace for your organization. Please see
https://circleci.com/docs/2.0/orb-concepts/#namespaces for more details.
type: env_var_name
number_of_runners:
default: NUMBER_OF_RUNNERS
description: |
Number of on demand runners to deploy. If no number is provided only 1
on demand runner will be deployed. PLEASE be careful setting this number.
type: env_var_name
number_of_waits:
default: NUMBER_OF_WAITS
description: |
Number of on waits(10 seconds each) for the EC2 instance(s) to be provisioned before failure.
If no number is provided 18 waits(3 minutes) will be selected.
type: env_var_name
resource_class:
description: |
Name of the resource class that will be generated and used for an On Demand Runner.
If you do not have a resource class to use, please pass in 'pipeline.id'. Generally, it best
practice to pass in 'pipeline.id'. PLEASE note you need to set resource class to a
name that is not currently being used, as the resource class will be deleted.
type: string
tag:
default: "3.10"
description: The `cimg/python` Docker image version tag.
type: string
version:
default: 2.9.*
description: Ansible Version To Be Installed
type: string
steps:
- run:
command: |-
#!/bin/bash
pip install "ansible==${ORB_PARAM_ANSIBLE_VERSION}"
pip install boto3 botocore --upgrade
ansible-galaxy collection install amazon.aws community.general
environment:
ORB_PARAM_ANSIBLE_VERSION: <<parameters.version>>
name: Installing Dependencies
- run:
command: |
{
echo "export AWS_DEFAULT_REGION=$(echo ${<< parameters.aws-region >>:-us-east-1})"
echo "export RESOURCE_CLASS=$(echo ${RESOURCE_CLASS:-<< parameters.resource_class >>})"
echo "export NUMBER_OF_RUNNERS=$(echo ${<< parameters.number_of_runners >>:-1})"
echo "export NUMBER_OF_WAITS=$(echo ${<< parameters.number_of_waits >>:-18})"
echo "export ANSIBLE_FORCE_COLOR=$(echo ${ANSIBLE_FORCE_COLOR:-true})"
echo "export ANSIBLE_GIT_URL=$(echo ${ANSIBLE_GIT_URL:-https://github.com/CircleCI-Public/on-demand-runner-orb.git})"
echo "export ANSIBLE_GIT_BRANCH=$(echo ${ANSIBLE_GIT_BRANCH:-main})"
echo "export ANSIBLE_HOST_KEY_CHECKING=$(echo ${ANSIBLE_HOST_KEY_CHECKING:-false})"
echo "export RESOURCE_CLASS_CREATION=$(echo ${RESOURCE_CLASS_CREATION:-true})"
echo "export RUNNER_TOKEN_CREATION=$(echo ${RUNNER_TOKEN_CREATION:-true})"
echo "export TEMPLATE_DYNAMIC_INVENTORY=$(echo ${TEMPLATE_DYNAMIC_INVENTORY:-true})"
echo "export CLOUD_PROVIDER=$(echo ${CLOUD_PROVIDER:-aws})"
} >> $BASH_ENV
name: Setting Defaults if Variables are not Set
- run:
command: |
git clone ${ANSIBLE_GIT_URL} && cd ./on-demand-runner-orb && git checkout ${ANSIBLE_GIT_BRANCH}
name: Clone On Demand Runner Ansible from Github
- run:
command: |
ansible-playbook deprovision-on-demand-runner.yml -e "REGION=$AWS_DEFAULT_REGION SSH_KEY_NAME=$<< parameters.aws-ssh-key >> NAMESPACE=$<< parameters.circleci-namespace >> RESOURCE_CLASS=$RESOURCE_CLASS CIRCLE_TOKEN=$<< parameters.circle-token >> NUMBER_OF_RUNNERS=$NUMBER_OF_RUNNERS UNTIL_RETRIES_RUNNERS=$NUMBER_OF_WAITS RESOURCE_CLASS_CREATION=$RESOURCE_CLASS_CREATION RUNNER_TOKEN_CREATION=$RUNNER_TOKEN_CREATION TEMPLATE_DYNAMIC_INVENTORY=$TEMPLATE_DYNAMIC_INVENTORY CLOUD_PROVIDER=$CLOUD_PROVIDER"
name: Deprovision the On Demand Runner
working_directory: ~/project/on-demand-runner-orb/auto-runner/
examples:
run-test-on-runner:
description: |
Deploy an On Demand Runner, Do Testing on the Runner and Destroy the On Demand Runner
usage:
version: "2.1"
orbs:
on-demand-runner: cci-labs/on-demand-runner@1.0.0
jobs:
run-test-on-runner:
machine: true
resource_class: NAMESPACE/<< pipeline.id >>
steps:
- run: echo "Hello World! I am running on an On Demand Runner!"
workflows:
example-workflow:
jobs:
- on-demand-runner/deploy:
context: aws-creds
resource_class: << pipeline.id >>
- run-test-on-runner:
requires:
- on-demand-runner/deploy
- on-demand-runner/destroy:
context: aws-creds
requires:
- run-test-on-runner
resource_class: << pipeline.id >>