Use CircleCI version 2.1 at the top of your .circleci/config.yml file.
1
version: 2.1
Add the orbs
stanza below your version, invoking the orb:
1
2
orbs:
trigger-tests: mabl/trigger-tests@1.0.8
Use trigger-tests
elements in your existing workflows and jobs.
Opt-in to use of uncertified orbs on your organization’s Security settings page.
Triggering tests using an application ID and an environment ID.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
version: '2.1'
orbs:
mabl-trigger-tests: mabl/trigger-tests@1.0.6
jobs:
run-mabl-tests:
machine: true
steps:
- mabl-trigger-tests/run-tests:
api-key: MABL_API_KEY
application-id: myApplicationID-a
environment-id: myEnvironmentID-e
workflows:
test-my-app:
jobs:
- run-mabl-tests
Triggering tests that have a particular plan label executing on Chrome only using a mabl test branch.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
version: '2.1'
orbs:
mabl-trigger-tests: mabl/trigger-tests@1.0.6
jobs:
run-mabl-tests:
machine: true
steps:
- mabl-trigger-tests/run-tests:
api-key: MABL_API_KEY
application-id: myApplicationID-a
browsers: chrome
environment-id: myEnvironmentID-e
labels: myPlanLabel
mabl-branch: developmentBranch
workflows:
test-my-app:
jobs:
- run-mabl-tests
Triggers tests to execute in mabl.
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
api-key | The API key of your workspace. You can find the API key on the Settings/API page.
Either an Integration or a CI/CD Deployment Integration key is required.
| No | MABL_API_KEY | env_var_name |
application-id | (Optional) The application ID to run tests on. Note that either an environment ID or
an application ID must be provided.
| No | '' | string |
await-completion | Wait for the deployment tests to finish before proceeding.
| No | true | boolean |
browsers | Comma separated list of browsers to test against (internet_explorer, safari, chrome,
firefox). If not provided, mabl will test the browsers configured on the triggered test.
| No | '' | string |
debug | Set to true to see more output in the logs.
| No | false | boolean |
environment-id | (Optional) The environment ID to run the tests in. Note that either an environment ID or
an application ID must be provided.
| No | '' | string |
labels | (Optional) A comma-separated list of plan labels. Plans with any of the labels will be
executed.
| No | '' | string |
mabl-branch | (Optional) The mabl branch to run tests against.
| No | '' | string |
rebaseline-images | Set true to reset the visual baseline to the current deployment.
| No | false | boolean |
revision | The code revision hash for the application under test.
| No | '' | string |
set-static-baseline | Set to true to use current deployment as an exact static baseline. If set, mabl will not
model dynamic areas and will use the current deployment as the pixel-exact visual baseline.
| No | false | boolean |
url | The base uri to test against. If provided, this will override the default URL associated
with the environment in mabl.
| No | '' | string |
This command parses the execution results and generates a JUnit report.
This command triggers test runs and waits for completion if requested.
PARAMETER | DESCRIPTION | REQUIRED | DEFAULT | TYPE |
---|---|---|---|---|
api-key | The environment variable holding the API key of your workspace. You can find the API key
on the Settings/API page. Either an Integration or a CI/CD Deployment Integration key is
required.
| No | MABL_API_KEY | env_var_name |
application-id | (Optional) The application ID to run tests on. Note that either an environment ID or
an application ID must be provided.
| No | '' | string |
await-completion | Wait for the deployment tests to finish before proceeding.
| No | true | boolean |
browsers | Comma separated list of browsers to test against (internet_explorer, safari, chrome,
firefox). If not provided, mabl will test the browsers configured on the triggered test.
| No | '' | string |
debug | Set to true to see more output in the logs.
| No | false | boolean |
environment-id | (Optional) The environment ID to run the tests in. Note that either an environment ID or
an application ID must be provided.
| No | '' | string |
labels | (Optional) A comma-separated list of plan labels. Plans with any of the labels will be
executed.
| No | '' | string |
mabl-branch | (Optional) The mabl branch to run tests against.
| No | '' | string |
rebaseline-images | Set true to reset the visual baseline to the current deployment.
| No | false | boolean |
revision | The code revision hash for the application under test.
| No | '' | string |
set-static-baseline | Set to true to use current deployment as an exact static baseline. If set, mabl will not
model dynamic areas and will use the current deployment as the pixel-exact visual baseline.
| No | false | boolean |
url | The base uri to test against. If provided, this will override the default URL associated
with the environment in mabl.
| No | '' | string |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
# This code is licensed from CircleCI to the user under the MIT license.
# See here for details: https://circleci.com/developer/orbs/licensing
version: 2.1
description: |
Automatically trigger specific environment or application plans in mabl.
display:
home_url: https://mabl.com/
source_url: https://github.com/mablhq-public/circleci-orb
commands:
parse-results:
description: |
This command parses the execution results and generates a JUnit report.
steps:
- run:
command: |
function parse_exec_results() {
local KEYS PAIR KEY TMPFILE FIRST_CALL KV_FILE DATE_CMD
DATE_CMD="date"
# MacOS built-in date command does not work, try gdate instead
date --help >/dev/null 2>&1 || DATE_CMD="gdate"
export DATE_CMD
${DATE_CMD} >/dev/null 2>&1 || (echo "missing date command."; return 1)
TMPFILE=$(mktemp)
jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" <"$1" >"${TMPFILE}"
KEYS=''
FIRST_CALL="true"
if [ $# -gt 1 ]; then
KEYS="$2"
KV_FILE="$3"
FIRST_CALL="false"
else
KV_FILE=$(mktemp)
fi
while read -r PAIR; do
KEY=''
if [ -z "$PAIR" ]; then
break
fi
PAIR_KEY=$(echo "${PAIR}" | cut -d'=' -f1)
PAIR_VALUE=$(echo "${PAIR}" | cut -d'=' -f2)
if [ -z "$KEYS" ]; then
KEY="$PAIR_KEY"
else
KEY="$KEYS:$PAIR_KEY"
fi
if [[ "${PAIR_VALUE}" == \{* ]] || [[ "${PAIR_VALUE}" == [* ]]; then
PV_FILE=$(mktemp)
echo "${PAIR_VALUE}" > "${PV_FILE}"
parse_exec_results "${PV_FILE}" "${KEY}" "${KV_FILE}"
rm -f "${PV_FILE}"
else
echo "${KEY}" >>"${KV_FILE}"
echo "${PAIR_VALUE}" >>"${KV_FILE}"
fi
done < "${TMPFILE}"
rm -f "${TMPFILE}"
if [ "${FIRST_CALL}" = "true" ]; then
local i j k epoch timestamp suiteTime testcases testName testFailures testTime \
testLink testSuccess testStatus testStatusCause line kv_key
declare -A EXEC_RESULTS
kv_key=""
while read -r line; do
if [ "${kv_key}" = "" ]; then
kv_key="${line}"
else
EXEC_RESULTS[$kv_key]="${line}"
kv_key=""
fi
done < "${KV_FILE}"
rm -f "${KV_FILE}"
echo "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>"
echo "<testsuites xmlns:xlink=\"http://www.w3.org/1999/xlink\">"
i=0
plan="${EXEC_RESULTS[executions:${i}:plan:name]}"
while [ -n "${plan}" ]; do
epoch=$(echo "${EXEC_RESULTS[executions:${i}:start_time]}" | sed 's/\([0-9]*\)\([0-9][0-9][0-9]\)/\1.\2/g')
timestamp=$(${DATE_CMD} -d @"${epoch}" -u +%Y-%m-%dT%H:%M:%S)
suiteTime=$(((${EXEC_RESULTS[executions:${i}:stop_time]}-${EXEC_RESULTS[executions:${i}:start_time]})/1000))
declare -a testcases
j=0
testFailures=0
while true; do
testName=$(get_test_name "${i}" "${EXEC_RESULTS[executions:${i}:journey_executions:${j}:journey_id]}")
testTime=$(((${EXEC_RESULTS[executions:${i}:journey_executions:${j}:stop_time]}-${EXEC_RESULTS[executions:${i}:journey_executions:${j}:start_time]})/1000))
testLink=${EXEC_RESULTS[executions:${i}:journey_executions:${j}:app_href]}
testSuccess=${EXEC_RESULTS[executions:${i}:journey_executions:${j}:success]}
if [ "${testSuccess}" = "true" ]; then
testcases[$j]="<testcase classname=\"${plan}\" name=\"${testName}\" time=\"${testTime}\" xlink:type=\"simple\" xlink:href=\"${testLink}\"/>"
else
testStatus=${EXEC_RESULTS[executions:${i}:journey_executions:${j}:status]}
testStatusCause=${EXEC_RESULTS[executions:${i}:journey_executions:${j}:status_cause]}
if [ -z "${testStatusCause}" ]; then
testStatusCause=${EXEC_RESULTS[executions:${i}:journey_executions:${j}:failure_summary:error]}
fi
testcases[$j]="<testcase classname=\"${plan}\" name=\"${testName}\" time=\"${testTime}\" xlink:type=\"simple\" xlink:href=\"${testLink}\">\
<failure message=\"${testStatus}\">${testStatusCause}</failure></testcase>"
testFailures=$((testFailures+1))
fi
j=$((j+1))
if [ -n "${EXEC_RESULTS[executions:${i}:journey_executions:${j}:journey_id]}" ]; then
testName=$(get_test_name "${i}" "${EXEC_RESULTS[executions:${i}:journey_executions:${j}:journey_id]}")
else
break
fi
done
echo "<testsuite name=\"${plan}\" tests=\"${j}\" errors=\"0\" failures=\"${testFailures}\" timestamp=\"${timestamp}\" time=\"${suiteTime}\">"
k=0
while [ $k -lt ${#testcases[@]} ]; do
echo "${testcases[$k]}"
k=$((k+1))
done
echo "</testsuite>"
i=$((i+1))
plan=${EXEC_RESULTS[executions:${i}:plan:name]}
done
echo "</testsuites>"
fi
}
function get_test_name() {
local exec_id journey_id i id test_name
exec_id="$1"
journey_id="$2"
i=0
id=${EXEC_RESULTS[executions:${exec_id}:journeys:${i}:id]}
test_name=""
while [ -n "${id}" ]; do
if [ "${id}" != "${journey_id}" ]; then
test_name=${EXEC_RESULTS[executions:${exec_id}:journeys:${i}:name]}
break
fi
i=$((i+1))
id=${EXEC_RESULTS[executions:${exec_id}:journeys:${i}:id]}
done
echo "${test_name}"
}
function parse_results() {
if [ ! -f "${PARAM_MABL_RESULTS}" ]; then
echo "No execution result was found."
return 1
fi
if [ -z "${PARAM_MABL_JUNIT}" ]; then
echo "Missing JUnit report file parameter."
return 1
fi
parse_exec_results "${PARAM_MABL_RESULTS}" > "${PARAM_MABL_JUNIT}"
return $?
}
# Will not run if sourced for bats.
# View src/tests for more information.
TEST_ENV="bats-core"
if [ "${0#*$TEST_ENV}" == "$0" ]; then
parse_results
fi
environment:
PARAM_MABL_JUNIT: test-results/mabl/junit.xml
PARAM_MABL_RESULTS: test-results/mabl/execution_result.json
name: Parse results
run-tests:
description: |
This command triggers test runs and waits for completion if requested.
parameters:
api-key:
default: MABL_API_KEY
description: |
The environment variable holding the API key of your workspace. You can find the API key
on the Settings/API page. Either an Integration or a CI/CD Deployment Integration key is
required.
type: env_var_name
application-id:
default: ""
description: |
(Optional) The application ID to run tests on. Note that either an environment ID or
an application ID must be provided.
type: string
await-completion:
default: true
description: |
Wait for the deployment tests to finish before proceeding.
type: boolean
browsers:
default: ""
description: |
Comma separated list of browsers to test against (internet_explorer, safari, chrome,
firefox). If not provided, mabl will test the browsers configured on the triggered test.
type: string
debug:
default: false
description: |
Set to true to see more output in the logs.
type: boolean
environment-id:
default: ""
description: |
(Optional) The environment ID to run the tests in. Note that either an environment ID or
an application ID must be provided.
type: string
labels:
default: ""
description: |
(Optional) A comma-separated list of plan labels. Plans with any of the labels will be
executed.
type: string
mabl-branch:
default: ""
description: |
(Optional) The mabl branch to run tests against.
type: string
rebaseline-images:
default: false
description: |
Set true to reset the visual baseline to the current deployment.
type: boolean
revision:
default: ""
description: |
The code revision hash for the application under test.
type: string
set-static-baseline:
default: false
description: |
Set to true to use current deployment as an exact static baseline. If set, mabl will not
model dynamic areas and will use the current deployment as the pixel-exact visual baseline.
type: boolean
url:
default: ""
description: |
The base uri to test against. If provided, this will override the default URL associated
with the environment in mabl.
type: string
steps:
- run:
command: "function run_tests() {\n USER_AGENT=\"mabl-circleci-orb/1.0.5\"\n \n # Verify parameters\n if [ -z \"${PARAM_API_KEY}\" ]; then\n echo \"api-key mandatory parameter is missing.\"\n return 1\n fi\n\n KEY=${!PARAM_API_KEY}\n if [ -z \"${KEY}\" ]; then\n echo \"missing API key value.\"\n return 1\n fi\n \n if [ -z \"${PARAM_ENVIRONMENT_ID}\" ] && [ -z \"${PARAM_APPLICATION_ID}\" ]; then\n echo \"Either environment-id or application-id must be provided.\"\n return 1\n fi\n \n if [ -n \"${PARAM_ENVIRONMENT_ID}\" ] && [[ ! \"${PARAM_ENVIRONMENT_ID}\" == *-e ]]; then\n echo \"environment-id parameter (${PARAM_ENVIRONMENT_ID}) must end with a '-e'.\"\n return 1\n fi\n \n if [ -n \"${PARAM_APPLICATION_ID}\" ] && [[ ! \"${PARAM_APPLICATION_ID}\" == *-a ]]; then\n echo \"application-id parameter (${PARAM_APPLICATION_ID}) must end with a '-a'.\"\n return 1\n fi\n \n BROWSER_TYPES=\"\"\n if [ -n \"${PARAM_BROWSERS}\" ]; then\n for BROWSER in $(echo \"${PARAM_BROWSERS}\" | sed 's/,/ /g'); do\n case ${BROWSER} in\n chrome|firefox|internet_explorer|safari)\n ;;\n *)\n echo \"Invalid browser value provided: ${PARAM_BROWSERS}\"\n echo \"Permitted values: chrome, firefox, internet_explorer, safari\"\n exit 1\n ;;\n esac\n done\n BROWSER_TYPES=$(echo \"${PARAM_BROWSERS}\" | sed 's/^/\\\"/' | sed 's/$/\\\"/' | sed 's/,/\\\",\\\"/g')\n fi\n \n shopt -s extglob\n if [ -n \"${PARAM_URL}\" ]; then\n # Check if this is an environment variable\n if [[ \"${PARAM_URL}\" == \\$* ]]; then\n PARAM_URL=${PARAM_URL:1}\n PARAM_URL=${!PARAM_URL}\n fi\n\n if [ -n \"${PARAM_URL}\" ] && [[ \"${PARAM_URL}\" != http*(s)://* ]]; then\n echo \"Invalid URL parameter provided: ${PARAM_URL}\"\n return 1\n fi\n fi\n \n # Create JSON payload for deployment event\n declare -A PARAMS\n declare -A PLAN_OVERRIDES\n \n if [ -n \"${PARAM_ENVIRONMENT_ID}\" ]; then\n PARAMS[environment_id]=\"${PARAM_ENVIRONMENT_ID}\"\n fi\n \n if [ -n \"${PARAM_APPLICATION_ID}\" ]; then\n PARAMS[application_id]=\"${PARAM_APPLICATION_ID}\"\n fi\n \n if [ -n \"${PARAM_LABELS}\" ]; then\n LABELS=$(echo \"${PARAM_LABELS}\" | sed 's/^/\\\"/' | sed 's/$/\\\"/' | sed 's/,/\\\",\\\"/g')\n PARAMS[plan_labels]=\"[${LABELS}]\"\n fi\n \n if [ -n \"${PARAM_MABL_BRANCH}\" ]; then\n PARAMS[source_control_tag]=\"${PARAM_MABL_BRANCH}\"\n fi\n \n if [ -n \"${BROWSER_TYPES}\" ]; then\n PLAN_OVERRIDES[browser_types]=\"[${BROWSER_TYPES}]\"\n fi\n \n if [ -n \"${PARAM_URL}\" ]; then\n PLAN_OVERRIDES[uri]=\"${PARAM_URL}\"\n fi\n \n if [ -n \"${PARAM_REVISION}\" ]; then\n PLAN_OVERRIDES[revision]=\"${PARAM_REVISION}\"\n fi\n \n PLAN_OVERRIDES_JSON=\"\"\n if [ ${#PLAN_OVERRIDES[@]} -gt 0 ]; then\n PLAN_OVERRIDES_JSON=\"\\\"plan_overrides\\\": { \"\n for key in \"${!PLAN_OVERRIDES[@]}\"; do\n VALUE=${PLAN_OVERRIDES[$key]}\n if [[ \"${VALUE}\" != \\[* ]]; then\n VALUE=\"\\\"${VALUE}\\\"\"\n fi\n PLAN_OVERRIDES_JSON=\"${PLAN_OVERRIDES_JSON} \\\"${key}\\\":${VALUE},\"\n done\n PLAN_OVERRIDES_JSON=$(echo \"${PLAN_OVERRIDES_JSON}\" | sed 's/,$/ },/g')\n fi\n \n PARAMS_JSON=\"{ \"\n for key in \"${!PARAMS[@]}\"; do\n VALUE=${PARAMS[$key]}\n if [[ \"${VALUE}\" != \\[* ]]; then\n VALUE=\"\\\"${VALUE}\\\"\"\n fi\n PARAMS_JSON=\"${PARAMS_JSON} \\\"${key}\\\":${VALUE},\"\n done\n \n if [ -n \"${PLAN_OVERRIDES_JSON}\" ]; then\n PARAMS_JSON=\"${PARAMS_JSON} ${PLAN_OVERRIDES_JSON}\"\n fi\n \n REBASELINE=\"false\"\n if [ \"${PARAM_REBASELINE_IMAGES}\" = \"1\" ] || [ \"${PARAM_REBASELINE_IMAGES}\" = \"true\" ]; then\n REBASELINE=\"true\"\n fi\n SET_STATIC_BASELINE=\"false\"\n if [ \"${PARAM_SET_STATIC_BASELINE}\" = \"1\" ] || [ \"${PARAM_SET_STATIC_BASELINE}\" = \"true\" ]; then\n SET_STATIC_BASELINE=\"true\"\n fi\n ACTIONS_JSON=\"\\\"actions\\\": { \\\"rebaseline_images\\\":\\\"${REBASELINE}\\\",\"\n ACTIONS_JSON=\"${ACTIONS_JSON}\\\"set_static_baseline\\\":\\\"${SET_STATIC_BASELINE}\\\" }\"\n \n PARAMS_JSON=\"${PARAMS_JSON}${ACTIONS_JSON} }\"\n debug \"Parameters: ${PARAMS_JSON}\"\n \n # Create output directory\n mkdir -p test-results/mabl\n \n # Submit deployment event\n deployment_event=$(curl -s \"https://api.mabl.com/events/deployment\" -u \"key:${KEY}\" -A \"${USER_AGENT}\" -H 'Content-Type:application/json' -d \"${PARAMS_JSON}\")\n debug \"Received event: $deployment_event\"\n event_id=$(echo \"${deployment_event}\" | jq -r '.id')\n if [ -n \"${event_id}\" ] && [[ \"${event_id}\" != *-v ]]; then\n echo \"Failed to submit deployment event.\"\n return 1\n fi\n workspace_id=$(echo \"${deployment_event}\" | jq -r '.workspace_id')\n echo \"Successfully triggered deployment at https://app.mabl.com/workspaces/${workspace_id}/events/${event_id}\"\n echo \"${deployment_event}\" > test-results/mabl/deployment_event.json\n \n # Poll execution result if configured to do so\n succeeded=false\n failed_plans=0\n if [ \"${PARAM_AWAIT_COMPLETION}\" = \"1\" ]; then\n complete=false\n while [ ${complete} == false ]; do\n echo \"Waiting for executions to complete...\"\n sleep 10\n results=$(curl -s \"https://api.mabl.com/execution/result/event/${event_id}\" -A \"${USER_AGENT}\" -u \"key:${KEY}\")\n debug \"execution event result: $results\"\n plan_metrics=$(echo \"${results}\" | jq '.plan_execution_metrics')\n if [ \"${plan_metrics}\" == \"null\" ]; then\n continue\n fi\n failed_plans=$(echo \"${plan_metrics}\" | jq -r '.failed')\n event_status=$(echo \"${results}\" | jq -r .event_status.succeeded)\n case ${event_status} in\n null)\n continue\n ;;\n true)\n succeeded=true\n complete=true\n ;;\n false)\n succeeded=false\n complete=true\n ;;\n *)\n debug \"Unexpected event status received: ${event_status}\"\n succeeded=false\n complete=false\n ;;\n esac\n \n done\n echo\n echo \"${results}\" >test-results/mabl/execution_result.json\n if [ \"${PARAM_DEBUG}\" = \"1\" ]; then\n echo \"Full Results:\"\n cat test-results/mabl/execution_result.json\n echo\n fi\n if [ ${succeeded} == true ]; then\n echo \"All plans passed.\"\n return 0\n else\n echo \"Some plans have failed. Total number of failed plans: ${failed_plans}\"\n return 1\n fi\n fi\n}\n\nfunction debug() {\n if [ \"${PARAM_DEBUG}\" = \"1\" ]; then\n echo \"$1\"\n fi\n}\n\n# Will not run if sourced for bats.\n# View src/tests for more information.\nTEST_ENV=\"bats-core\"\nif [ \"${0#*$TEST_ENV}\" == \"$0\" ]; then\n run_tests\nfi\n"
environment:
PARAM_API_KEY: <<parameters.api-key>>
PARAM_APPLICATION_ID: <<parameters.application-id>>
PARAM_AWAIT_COMPLETION: <<parameters.await-completion>>
PARAM_DEBUG: <<parameters.debug>>
PARAM_ENVIRONMENT_ID: <<parameters.environment-id>>
PARAM_LABELS: <<parameters.labels>>
PARAM_MABL_BRANCH: <<parameters.mabl-branch>>
PARAM_REBASELINE_IMAGES: <<parameters.rebaseline-images>>
PARAM_REVISION: <<parameters.revision>>
PARAM_SET_STATIC_BASELINE: <<parameters.set-static-baseline>>
PARAM_URL: <<parameters.url>>
name: Run tests
jobs:
test:
description: |
Triggers tests to execute in mabl.
docker:
- image: cimg/base:stable
parameters:
api-key:
default: MABL_API_KEY
description: |
The API key of your workspace. You can find the API key on the Settings/API page.
Either an Integration or a CI/CD Deployment Integration key is required.
type: env_var_name
application-id:
default: ""
description: |
(Optional) The application ID to run tests on. Note that either an environment ID or
an application ID must be provided.
type: string
await-completion:
default: true
description: |
Wait for the deployment tests to finish before proceeding.
type: boolean
browsers:
default: ""
description: |
Comma separated list of browsers to test against (internet_explorer, safari, chrome,
firefox). If not provided, mabl will test the browsers configured on the triggered test.
type: string
debug:
default: false
description: |
Set to true to see more output in the logs.
type: boolean
environment-id:
default: ""
description: |
(Optional) The environment ID to run the tests in. Note that either an environment ID or
an application ID must be provided.
type: string
labels:
default: ""
description: |
(Optional) A comma-separated list of plan labels. Plans with any of the labels will be
executed.
type: string
mabl-branch:
default: ""
description: |
(Optional) The mabl branch to run tests against.
type: string
rebaseline-images:
default: false
description: |
Set true to reset the visual baseline to the current deployment.
type: boolean
revision:
default: ""
description: |
The code revision hash for the application under test.
type: string
set-static-baseline:
default: false
description: |
Set to true to use current deployment as an exact static baseline. If set, mabl will not
model dynamic areas and will use the current deployment as the pixel-exact visual baseline.
type: boolean
url:
default: ""
description: |
The base uri to test against. If provided, this will override the default URL associated
with the environment in mabl.
type: string
steps:
- run-tests:
api-key: << parameters.api-key >>
application-id: << parameters.application-id >>
await-completion: << parameters.await-completion >>
browsers: << parameters.browsers >>
debug: << parameters.debug >>
environment-id: << parameters.environment-id >>
labels: << parameters.labels >>
mabl-branch: << parameters.mabl-branch >>
rebaseline-images: << parameters.rebaseline-images >>
revision: << parameters.revision >>
set-static-baseline: << parameters.set-static-baseline >>
url: << parameters.url >>
- when:
condition: << parameters.await-completion >>
steps:
- parse-results
- store_test_results:
path: test-results
- store_artifacts:
path: test-results/mabl
examples:
trigger_tests_app_and_env_only:
description: |
Triggering tests using an application ID and an environment ID.
usage:
version: "2.1"
orbs:
mabl-trigger-tests: mabl/trigger-tests@1.0.6
jobs:
run-mabl-tests:
machine: true
steps:
- mabl-trigger-tests/run-tests:
api-key: MABL_API_KEY
application-id: myApplicationID-a
environment-id: myEnvironmentID-e
workflows:
test-my-app:
jobs:
- run-mabl-tests
trigger_tests_chrome_with_labels_and_branch:
description: |
Triggering tests that have a particular plan label executing on Chrome only using a mabl test branch.
usage:
version: "2.1"
orbs:
mabl-trigger-tests: mabl/trigger-tests@1.0.6
jobs:
run-mabl-tests:
machine: true
steps:
- mabl-trigger-tests/run-tests:
api-key: MABL_API_KEY
application-id: myApplicationID-a
browsers: chrome
environment-id: myEnvironmentID-e
labels: myPlanLabel
mabl-branch: developmentBranch
workflows:
test-my-app:
jobs:
- run-mabl-tests