Use CircleCI version 2.1 at the top of your .circleci/config.yml file.
1
version: 2.1
Add the orbs
stanza below your version, invoking the orb:
1
2
orbs:
rainforest: rainforest-qa/rainforest@5.1.1
Use rainforest
elements in your existing workflows and jobs.
Opt-in to use of uncertified orbs on your organization’s Security settings page.
Run tests from a specific run group
1
2
3
4
5
6
7
8
9
orbs:
rainforest: rainforest-qa/rainforest@3
version: 2.1
workflows:
build:
jobs:
- rainforest/run:
pipeline_id: << pipeline.id >>
run_group_id: '123'
Start a new Rainforest run
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
automation_max_retries | If set to a value > 0 and a test run using automation fails, it will be retried within the same run, up to that number of times | No | '' | string |
background | Do not wait for a rainforest run to complete before exiting | No | false | boolean |
branch | Use a specific Rainforest branch for this run | No | '' | string |
cache_key | The cache key, used to store/restore a Rainforest Run ID | No | rainforest-run-{{ .Revision }} | string |
conflict | How other currently in-progress runs should be handled. Values are `cancel` to cancel runs in the same environment as your new run and `cancel-all` to cancel all runs | No | '' | string |
custom_url | Use a specific URL for this run | No | '' | string |
description | An arbitrary string to associate with the run | No | >- $CIRCLE_PROJECT_REPONAME - $CIRCLE_BRANCH $CIRCLE_BUILD_NUM $(date -u +'%FT%TZ') | string |
dry_run | Set to true to run parameter validations without starting a new Rainforest run | No | false | boolean |
environment_id | Use a specific environment for this run | No | '' | string |
execution_method | The execution method to use for this run | No | '' | string |
executor | The executor to run this command in | No | default | executor |
junit_path | Folder under ~/results to store the JUnit results.xml file | No | ~/results/rainforest | string |
pipeline_id | The CircleCI Pipeline ID ( << pipeline.id >> ), used to rerun only failed tests when a CircleCI workflow is rerun | Yes | - | string |
release | Manually entered release information about the release the run is associated with | No | $CIRCLE_SHA1 | string |
run_group_id | Only run tests tied to this Run Group | Yes | - | string |
token | The name of the environment variable containing your Rainforest QA API token | No | RAINFOREST_API_TOKEN | env_var_name |
Install the Rainforest QA CLI
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
architecture | 386 (32-bit) or amd64 (64-bit). | No | amd64 | enum |
channel | [DEPRECATED] This parameter no longer has any effect and will be removed in a future version | No | '' | string |
install_path | Where to install the CLI. | No | /usr/local/bin | string |
platform | Which platform to install the CLI for. | No | linux | enum |
version | Which (v-prefixed) version of the CLI to install. Defaults to the latest. | No | '' | string |
Start a new Rainforest run
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
automation_max_retries | If set to a value > 0 and a test run using automation fails, it will be retried within the same run, up to that number of times | No | '' | string |
background | Do not wait for a run to complete before exiting | No | false | boolean |
branch | Use a specific Rainforest branch for this run | No | '' | string |
conflict | How other currently in-progress runs should be handled. Values are `cancel` to cancel runs in the same environment as your new run and `cancel-all` to cancel all runs | No | '' | string |
custom_url | Use a specific URL for this run | No | '' | string |
description | An arbitrary string to associate with the run | No | >- $CIRCLE_PROJECT_REPONAME - $CIRCLE_BRANCH $CIRCLE_BUILD_NUM $(date -u +'%FT%TZ') | string |
dry_run | Set to true to run parameter validations without starting a new Rainforest run | No | false | boolean |
environment_id | Use a specific environment for this run | No | '' | string |
execution_method | The execution method to use for this run | No | '' | string |
junit_path | Folder under ~/results to store the JUnit results.xml file | No | rainforest | string |
pipeline_id | The CircleCI Pipeline ID ( << pipeline.id >> ), used to rerun only failed tests when a CircleCI workflow is rerun | Yes | - | string |
release | Manually entered release information about the release the run is associated with | No | $CIRCLE_SHA1 | string |
run_group_id | Only run tests tied to this Run Group | Yes | - | string |
token | The name of the environment variable containing your Rainforest QA API token | No | RAINFOREST_API_TOKEN | env_var_name |
Save Pipeline and RF Run ID
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
pipeline_id | The CircleCI Pipeline ID ( << pipeline.id >> ), used to rerun only failed tests when a CircleCI workflow is rerun | Yes | - | string |
when | When to run the steps in this command (always, on_success, or on_fail). Default is on_fail | No | on_fail | enum |
Latest Rainforest CLI Docker image
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
tag | A tag pointing to an older version of the Rainforest CLI Docker image.
A list of tags can be found at https://gcr.io/rf-public-images/rainforest-cli.
| No | latest | string |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
# This code is licensed from CircleCI to the user under the MIT license.
# See here for details: https://circleci.com/developer/orbs/licensing
commands:
install:
description: Install the Rainforest QA CLI
parameters:
architecture:
default: amd64
description: 386 (32-bit) or amd64 (64-bit).
enum:
- "386"
- amd64
type: enum
channel:
default: ""
description: '[DEPRECATED] This parameter no longer has any effect and will be removed in a future version'
type: string
install_path:
default: /usr/local/bin
description: Where to install the CLI.
type: string
platform:
default: linux
description: Which platform to install the CLI for.
enum:
- darwin
- linux
type: enum
version:
default: ""
description: Which (v-prefixed) version of the CLI to install. Defaults to the latest.
type: string
steps:
- run:
command: |
if [ -n "<< parameters.channel >>" ]; then
echo "The 'channel' parameter is deprecated, please remove it from your configuration."
echo "If you'd like to use a specific version, use the new 'version' parameter instead."
echo "See https://github.com/rainforestapp/rainforest-orb/releases/tag/v3.0.1 for more info."
exit 1
fi
name: Warn about deprecations
- run:
command: |
release_url=https://api.github.com/repos/rainforestapp/rainforest-cli/releases
if [ -n "<< parameters.version >>" ]; then
release_url="$release_url/tags/<< parameters.version >>"
else
release_url="$release_url/latest"
fi
echo $release_url
asset_url=$(curl $release_url | jq -r '.assets[].browser_download_url' | grep << parameters.platform >>-<< parameters.architecture >>)
wget $asset_url -O rainforest-cli.tgz
tar xvf rainforest-cli.tgz -C << parameters.install_path >>
name: Download and set up executable
run_qa:
description: Start a new Rainforest run
parameters:
automation_max_retries:
default: ""
description: If set to a value > 0 and a test run using automation fails, it will be retried within the same run, up to that number of times
type: string
background:
default: false
description: Do not wait for a run to complete before exiting
type: boolean
branch:
default: ""
description: Use a specific Rainforest branch for this run
type: string
conflict:
default: ""
description: How other currently in-progress runs should be handled. Values are `cancel` to cancel runs in the same environment as your new run and `cancel-all` to cancel all runs
type: string
custom_url:
default: ""
description: Use a specific URL for this run
type: string
description:
default: $CIRCLE_PROJECT_REPONAME - $CIRCLE_BRANCH $CIRCLE_BUILD_NUM $(date -u +'%FT%TZ')
description: An arbitrary string to associate with the run
type: string
dry_run:
default: false
description: Set to true to run parameter validations without starting a new Rainforest run
type: boolean
environment_id:
default: ""
description: Use a specific environment for this run
type: string
execution_method:
default: ""
description: The execution method to use for this run
type: string
junit_path:
default: rainforest
description: Folder under ~/results to store the JUnit results.xml file
type: string
pipeline_id:
description: The CircleCI Pipeline ID ( << pipeline.id >> ), used to rerun only failed tests when a CircleCI workflow is rerun
type: string
release:
default: $CIRCLE_SHA1
description: Manually entered release information about the release the run is associated with
type: string
run_group_id:
description: Only run tests tied to this Run Group
type: string
token:
default: RAINFOREST_API_TOKEN
description: The name of the environment variable containing your Rainforest QA API token
type: env_var_name
steps:
- run:
command: |
# Show Orb Version
echo "Using Rainforest Orb v${ORB_VERSION}"
# Ensure results directory is there
mkdir -p ~/results/<< parameters.junit_path >>
# Validate token
if [ -z "${<< parameters.token >>}" ] ; then
echo "Error: Token not set"
exit 1
fi
# Validate conflict
if [ -n "<< parameters.conflict >>" ] ; then
if ! echo "cancel cancel-all" | grep -Eq "\b"<< parameters.conflict >>"\b" ; then
echo "Error: << parameters.conflict >> not in (cancel cancel-all)"
exit 1
fi
fi
# Check for rerun
if [ -n "<< parameters.pipeline_id >>" ] && [ -s ~/pipeline/<< parameters.pipeline_id >> ] ; then
export RAINFOREST_RUN_ID=$(cat ~/pipeline/<< parameters.pipeline_id >>)
echo "Rerunning Run ${RAINFOREST_RUN_ID}"
if ! << parameters.dry_run >> ; then
# Create the rerun
rainforest rerun \
--skip-update \
--token "${<< parameters.token >>}" \
<<# parameters.conflict >> --conflict "<< parameters.conflict >>" <</ parameters.conflict >> \
--background="<< parameters.background >>" \
--junit-file ~/results/<< parameters.junit_path >>/results.xml \
--save-run-id ~/.rainforest_run_id
fi
exit 0
fi
# Validate run_group_id
if ! echo "<< parameters.run_group_id >>" | grep -Eq '^[0-9]+$' ; then
echo "Error: run_group_id not a positive integer (<< parameters.run_group_id >>)"
exit 1
fi
# Set custom_url, or validate and set environment_id
if [ -n "<< parameters.custom_url >>" ] ; then
if [ -n "<< parameters.environment_id >>" ] ; then
echo "Warning: Environment ID ignored. You've set values for the mutually exclusive custom_url and environment_id parameters. Unset one of these to fix this warning."
fi
elif [ -n "<< parameters.environment_id >>" ] ; then
if ! echo "<< parameters.environment_id >>" | grep -Eq '^[0-9]+$' ; then
echo "Error: environment_id not a positive integer (<< parameters.environment_id >>)"
exit 1
fi
fi
# Validate execution_method
if [ -n "<< parameters.execution_method >>" ] ; then
if ! echo "automation crowd automation_and_crowd on_premise" | grep -Eq "\b"<< parameters.execution_method >>"\b" ; then
echo "Error: << parameters.execution_method >> not in (automation crowd automation_and_crowd on_premise)"
exit 1
fi
fi
# Validate automation_max_retries
if [ -n "<< parameters.automation_max_retries >>" ] ; then
if ! echo "<< parameters.automation_max_retries >>" | grep -Eq '^[0-9]+$' ; then
error "automation_max_retries not a positive integer (<< parameters.automation_max_retries >>)"
fi
fi
if ! << parameters.dry_run >> ; then
# Create the run
rainforest run \
--skip-update \
--token "${<< parameters.token >>}" \
--description "<< parameters.description >>" \
--run-group "<< parameters.run_group_id >>" \
<<# parameters.custom_url >> --custom-url "<< parameters.custom_url >>" <</ parameters.custom_url >> \
<<# parameters.environment_id >> --environment-id "<< parameters.environment_id >>" <</ parameters.environment_id >> \
<<# parameters.conflict >> --conflict "<< parameters.conflict >>" <</ parameters.conflict >> \
<<# parameters.execution_method >> --execution-method "<< parameters.execution_method >>" <</ parameters.execution_method >> \
<<# parameters.automation_max_retries >> --automation-max-retries "<< parameters.automation_max_retries >>" <</ parameters.automation_max_retries >> \
<<# parameters.branch >> --branch "<< parameters.branch >>" <</ parameters.branch >> \
--release "<< parameters.release >>" \
--background="<< parameters.background >>" \
--junit-file ~/results/<< parameters.junit_path >>/results.xml \
--save-run-id ~/.rainforest_run_id
fi
environment:
ORB_VERSION: 5.1.1
name: Run Rainforest
- unless:
condition: << parameters.dry_run >>
steps:
- store_test_results:
path: ~/results
- store_artifacts:
path: ~/results
save_run_id:
description: Save Pipeline and RF Run ID
parameters:
pipeline_id:
description: The CircleCI Pipeline ID ( << pipeline.id >> ), used to rerun only failed tests when a CircleCI workflow is rerun
type: string
when:
default: on_fail
description: When to run the steps in this command (always, on_success, or on_fail). Default is on_fail
enum:
- always
- on_fail
- on_success
type: enum
steps:
- run:
command: |
mkdir -p ~/pipeline
RUN_ID_FILE=~/.rainforest_run_id
PIPELINE_FILE=~/pipeline/<< parameters.pipeline_id >>
cat ${RUN_ID_FILE} > ${PIPELINE_FILE}
name: Save Pipeline and RF Run ID
when: << parameters.when >>
description: |
Rainforest QA's CircleCI Orb
Prerequisites:
- a Rainforest QA account (https://app.rainforestqa.com/trial)
- a Rainforest QA API token (https://app.rainforestqa.com/settings/integrations)
- a RunGroup with some tests (https://app.rainforestqa.com/run_groups)
display:
home_url: https://rainforestqa.com
source_url: https://github.com/rainforestapp/rainforest-orb
examples:
simple:
description: Run tests from a specific run group
usage:
orbs:
rainforest: rainforest-qa/rainforest@3
version: 2.1
workflows:
build:
jobs:
- rainforest/run:
pipeline_id: << pipeline.id >>
run_group_id: "123"
executors:
default:
description: Latest Rainforest CLI Docker image
docker:
- image: gcr.io/rf-public-images/rainforest-cli:<< parameters.tag >>
parameters:
tag:
default: latest
description: |
A tag pointing to an older version of the Rainforest CLI Docker image.
A list of tags can be found at https://gcr.io/rf-public-images/rainforest-cli.
type: string
jobs:
run:
description: Start a new Rainforest run
executor: <<parameters.executor>>
parameters:
automation_max_retries:
default: ""
description: If set to a value > 0 and a test run using automation fails, it will be retried within the same run, up to that number of times
type: string
background:
default: false
description: Do not wait for a rainforest run to complete before exiting
type: boolean
branch:
default: ""
description: Use a specific Rainforest branch for this run
type: string
cache_key:
default: rainforest-run-{{ .Revision }}
description: The cache key, used to store/restore a Rainforest Run ID
type: string
conflict:
default: ""
description: How other currently in-progress runs should be handled. Values are `cancel` to cancel runs in the same environment as your new run and `cancel-all` to cancel all runs
type: string
custom_url:
default: ""
description: Use a specific URL for this run
type: string
description:
default: $CIRCLE_PROJECT_REPONAME - $CIRCLE_BRANCH $CIRCLE_BUILD_NUM $(date -u +'%FT%TZ')
description: An arbitrary string to associate with the run
type: string
dry_run:
default: false
description: Set to true to run parameter validations without starting a new Rainforest run
type: boolean
environment_id:
default: ""
description: Use a specific environment for this run
type: string
execution_method:
default: ""
description: The execution method to use for this run
type: string
executor:
default: default
description: The executor to run this command in
type: executor
junit_path:
default: ~/results/rainforest
description: Folder under ~/results to store the JUnit results.xml file
type: string
pipeline_id:
description: The CircleCI Pipeline ID ( << pipeline.id >> ), used to rerun only failed tests when a CircleCI workflow is rerun
type: string
release:
default: $CIRCLE_SHA1
description: Manually entered release information about the release the run is associated with
type: string
run_group_id:
description: Only run tests tied to this Run Group
type: string
token:
default: RAINFOREST_API_TOKEN
description: The name of the environment variable containing your Rainforest QA API token
type: env_var_name
steps:
- restore_cache:
keys:
- << parameters.cache_key >>
- run_qa:
automation_max_retries: << parameters.automation_max_retries >>
background: << parameters.background >>
branch: << parameters.branch >>
conflict: << parameters.conflict >>
custom_url: << parameters.custom_url >>
description: << parameters.description >>
dry_run: << parameters.dry_run >>
environment_id: << parameters.environment_id >>
execution_method: << parameters.execution_method >>
junit_path: << parameters.junit_path >>
pipeline_id: << parameters.pipeline_id >>
release: << parameters.release >>
run_group_id: << parameters.run_group_id >>
token: << parameters.token >>
- save_run_id:
pipeline_id: << parameters.pipeline_id >>
- save_cache:
key: << parameters.cache_key >>-{{ .BuildNum }}
paths:
- ~/pipeline
when: on_fail
version: 2.1