Testing is a critical piece of any CI/CD pipeline. Most teams do pretty well with application level testing, and there are plenty of frameworks (JUnit, RSpec, etc.) to support it. But server-level testing–the validation of server configuration and services–is, too often, omitted. In this blog post, we’ll explore an approach to execute tests against our custom Docker images as part of a CircleCI pipeline using Goss.
Firstly, what is Goss?
Goss is a YAML based serverspec alternative tool for validating a server’s configuration.
We’re only going to cover a few types of tests in this post, so it’s worth exploring the Goss manual to learn all the operations available.
The project also includes
dgoss which is a Docker-focused wrapper for
goss. It includes two basic operations:
In order to run tests locally, you’ll need to install dgoss using the appropriate steps depending on your OS.
You’ll also need a Docker image under development. Through the rest of this post I’ll be referencing my sample project, which you should fork and follow along!
As suggested by Goss manual, the easiest way to get started is through the use of
goss add <TYPE> <ARGUMENTS> run inside the server/container you want to test. Because we’re running side the container, we’ll need to build it first.
docker build . -t my-image:test
Then using the tag name from above, we can run the container with Goss installed for us via dgoss.
dgoss edit supports any Docker arguments you need to start the image. In my sample project this includes a modified entrypoint.
dgoss edit --entrypoint=/test/gossEntrypoint.sh my-image:test
You can run
test/editTests.sh to as a shortcut to repeat these commands.
Once inside the running Docker image, you can explore different tests, which the Goss command will automatically append to a goss.yaml file. This file will be copied into your local workstation after your exit.
Verifying a file exists
Our first test is pretty primitive: make sure our entrypoint.sh file made it onto the filesystem.
/goss # goss add file /entrypoint.sh
You can see the output specifies the location, type, owner, and other key attributes. You can now
cat goss.yml to see the initial test structure, but a healthy image includes more than the mere existence of files, so let’s add a few more tests.
Verifying command output
Our Docker image is pretty simple, and the entrypoint just runs some simple logic based on the presence of configuration files mounted in our production environment. We want to make sure it handles this configuration file as expected.
For our first test, we want to make sure the script exits with a read warning if no file is found. Since this is the default state, we can run the following:
goss add command "/entrypoint.sh /config.txt /schedule.txt"
It is important to quote arguments, otherwise Goss will treat them as separate commands.
You can see that Goss is expecting an exit status of 1, which a message printed to stderr.
We have a few more tests that are a bit more complicated, as they modify files to mimic certain conditions that could occur in production. For these tests, I found it more readable and maintainable to encapsulate the logic into their own scripts with clear intentions indicated in the filename. One example can be seen in the
testNonEmptyScheduleModifiesScalerConfig.sh. That test will make sure that a non-empty schedule modifies our config.
Each test in that directory sets some expected state, then calls
/entrypoint.sh /config.txt /schedule.txt just as we did above.
Sometimes output will have dynamic content, and fortunately Goss supports some basic pattern matching. The test above, for instance, uses the current day and hour in the test, and will be printed in the output. Since this will change the output depending on when we execute the tests, we use a regexp to handle this.
/test/testNonEmptyScheduleModifiesScalerConfig.sh: exit-status: 0 stdout: - /Matching rule - Day:\s[1-7], Hour:\s[1-2]?[0-9], Type :\sdocker, Count:\s5/ - Updated docker preallocation count to 5 - /Matching rule - Day:\s[1-7], Hour:\s[1-2]?[0-9], Type :\smachine, Count:\s5/ - Updated machine preallocation count to 5 - schedule updated stderr:  timeout: 10000
Note: the use of
\s is required so that yaml doesn’t parse the colons as a yaml key: value.
The final spec (goss.yaml)
When you type
exit after running all your
goss add steps, you will see that Goss copies the generated goss.yaml back to your local machine before stopping the instance.
/goss # exit INFO: Copied '/goss/goss.yaml' from container to 'test' INFO: Deleting container
The file is an aggregation of the individual outputs we’ve already seen, grouped by their type.
file: /entrypoint.sh: exists: true mode: "0755" size: 1530 owner: root group: root filetype: file contains:  /schedule.sh: exists: false contains:  command: /test/testEmptyScheduleIgnored.sh: exit-status: 1 stdout:  stderr:  timeout: 10000 /test/testNonEmptyScheduleModifiesScalerConfig.sh: exit-status: 0 stdout: - /Matching rule - Day:\s[1-7], Hour:\s[1-2]?[0-9], Type :\sdocker, Count:\s5/ - Updated docker preallocation count to 5 - /Matching rule - Day:\s[1-7], Hour:\s[1-2]?[0-9], Type :\smachine, Count:\s5/ - Updated machine preallocation count to 5 - schedule updated stderr:  timeout: 10000 /test/testNonExistentScheduleIgnored.sh: exit-status: 1 stdout:  stderr: - 'cat: can''t open ''/schedule.txt'': No such file or directory' timeout: 10000
Running our tests
Now that we should have a working Docker image (we hope) and some tests defined in our goss.yaml, we want to execute our tests against a fresh image. dgoss expects a file named goss.yaml in the current directory. Since I placed ours in the
test folder, we need to include the
docker build . -t my-image:test GOSS_FILES_PATH=test dgoss run --entrypoint=/test/gossEntrypoint.sh my-image:test # OR provided wrapper for this tutorial test/runTests.sh
The commands above will build a new Docker image, mount Goss and goss.yaml, and execute our tests.
That output is not very exciting, but that’s the way passing tests should be. So, let’s break something!
#!/bin/bash # # Any changes are written to the SCALING_FILE # set -euo pipefail #this won't work.. CMD=`./nonexistentScript.sh` SCALING_FILE=$1 SCHEDULE_FILE=$2
What happens when you execute
test/runTest.sh? Goss will print the expected instead of the actual output, and include an error summary. It also exits with a non-zero status which is important for our next step: integrating with our continuous integration pipeline!
Running our tests on every commit
It’s great that we’ve got tests running locally, but the idea here is to integrate Docker-level testing into our CI/CD pipeline so that we don’t ship bad images.
CircleCI does not provide an image with Goss preloaded, but installing it only takes a second. You can view the sample config.yml for the full setup. I’m including just the relevant snippets here.
jobs: test: docker: - image: circleci/python:2-jessie steps: - checkout - setup_remote_docker: # (2) docker_layer_caching: true # (3) - run: name: Install goss command: | # rather than give internet scripts SU rights, we install to local user bin and add to path mkdir ~/bin export GOSS_DST=~/bin export PATH=$PATH:~/bin curl -fsSL https://goss.rocks/install | sh goss -version - run: name: Test command: | # Don't forget path! export PATH=$PATH:~/bin # Important, change from mount to work on remote docker, see https://github.com/aelsabbahy/goss/pull/271 # If using machine image you do not need this. export GOSS_FILES_STRATEGY=cp test/runTests.sh junit - store_test_results: path: goss
Note: for CI execution we pass the argument
junit to our test runner. This converts the output format and pipes it to ~/goss/report.xml to be included in CircleCI’s test summary:
Well that’s it! With this basic structure running, you can add more mature tests, and include Goss-based image testing as a core step in your CI/CD pipeline to keep your team’s codebase tested from service to server.
Be sure to add that happy green status badge to your repo to let the world know!