Testing Docker Images with CircleCI and Goss

Testing is a critical piece of any CI/CD pipeline. Most teams do pretty well with application level testing, and there are plenty of frameworks (JUnit, RSpec, etc.) to support it. But server-level testing–the validation of server configuration and services–is, too often, omitted. In this blog post, we’ll explore an approach to execute tests against our custom Docker images as part of a CircleCI pipeline using Goss.

Firstly, what is Goss?

Goss is a YAML based serverspec alternative tool for validating a server’s configuration.

We’re only going to cover a few types of tests in this post, so it’s worth exploring the Goss manual to learn all the operations available. The project also includes dgoss which is a Docker-focused wrapper for goss. It includes two basic operations: edit and run.

Pre-requisites

In order to run tests locally, you’ll need to install dgoss using the appropriate steps depending on your OS.

You’ll also need a Docker image under development. Through the rest of this post I’ll be referencing my sample project, which you should fork and follow along!

Creating tests

As suggested by Goss manual, the easiest way to get started is through the use of goss add <TYPE> <ARGUMENTS> run inside the server/container you want to test. Because we’re running side the container, we’ll need to build it first.

docker build . -t my-image:test

Then using the tag name from above, we can run the container with Goss installed for us via dgoss. dgoss edit supports any Docker arguments you need to start the image. In my sample project this includes a modified entrypoint.

dgoss edit --entrypoint=/test/gossEntrypoint.sh my-image:test

You can run test/editTests.sh to as a shortcut to repeat these commands.

Once inside the running Docker image, you can explore different tests, which the Goss command will automatically append to a goss.yaml file. This file will be copied into your local workstation after your exit.

Verifying a file exists

Our first test is pretty primitive: make sure our entrypoint.sh file made it onto the filesystem.

/goss # goss add file /entrypoint.sh

You can see the output specifies the location, type, owner, and other key attributes. You can now cat goss.yml to see the initial test structure, but a healthy image includes more than the mere existence of files, so let’s add a few more tests.

Verifying command output

Our Docker image is pretty simple, and the entrypoint just runs some simple logic based on the presence of configuration files mounted in our production environment. We want to make sure it handles this configuration file as expected.

For our first test, we want to make sure the script exits with a read warning if no file is found. Since this is the default state, we can run the following:

goss add command "/entrypoint.sh /config.txt /schedule.txt"

It is important to quote arguments, otherwise Goss will treat them as separate commands.

You can see that Goss is expecting an exit status of 1, which a message printed to stderr.

We have a few more tests that are a bit more complicated, as they modify files to mimic certain conditions that could occur in production. For these tests, I found it more readable and maintainable to encapsulate the logic into their own scripts with clear intentions indicated in the filename. One example can be seen in the testNonEmptyScheduleModifiesScalerConfig.sh. That test will make sure that a non-empty schedule modifies our config.

Each test in that directory sets some expected state, then calls /entrypoint.sh /config.txt /schedule.txt just as we did above.

Pattern matching

Sometimes output will have dynamic content, and fortunately Goss supports some basic pattern matching. The test above, for instance, uses the current day and hour in the test, and will be printed in the output. Since this will change the output depending on when we execute the tests, we use a regexp to handle this.

/test/testNonEmptyScheduleModifiesScalerConfig.sh:
  exit-status: 0
  stdout:
  - /Matching rule - Day:\s[1-7], Hour:\s[1-2]?[0-9], Type :\sdocker, Count:\s5/
  - Updated docker preallocation count to 5
  - /Matching rule - Day:\s[1-7], Hour:\s[1-2]?[0-9], Type :\smachine, Count:\s5/
  - Updated machine preallocation count to 5
  - schedule updated
  stderr: []
  timeout: 10000

Note: the use of \s is required so that yaml doesn’t parse the colons as a yaml key: value.

The final spec (goss.yaml)

When you type exit after running all your goss add steps, you will see that Goss copies the generated goss.yaml back to your local machine before stopping the instance.

/goss # exit
INFO: Copied '/goss/goss.yaml' from container to 'test'
INFO: Deleting container

The file is an aggregation of the individual outputs we’ve already seen, grouped by their type.

file:
  /entrypoint.sh:
    exists: true
    mode: "0755"
    size: 1530
    owner: root
    group: root
    filetype: file
    contains: []
  /schedule.sh:
    exists: false
    contains: []
command:
  /test/testEmptyScheduleIgnored.sh:
    exit-status: 1
    stdout: []
    stderr: []
    timeout: 10000
  /test/testNonEmptyScheduleModifiesScalerConfig.sh:
    exit-status: 0
    stdout:
    - /Matching rule - Day:\s[1-7], Hour:\s[1-2]?[0-9], Type :\sdocker, Count:\s5/
    - Updated docker preallocation count to 5
    - /Matching rule - Day:\s[1-7], Hour:\s[1-2]?[0-9], Type :\smachine, Count:\s5/
    - Updated machine preallocation count to 5
    - schedule updated
    stderr: []
    timeout: 10000
  /test/testNonExistentScheduleIgnored.sh:
    exit-status: 1
    stdout: []
    stderr:
    - 'cat: can''t open ''/schedule.txt'': No such file or directory'
    timeout: 10000

Running our tests

Now that we should have a working Docker image (we hope) and some tests defined in our goss.yaml, we want to execute our tests against a fresh image. dgoss expects a file named goss.yaml in the current directory. Since I placed ours in the test folder, we need to include the GOSS_FILES_PATH argument.

docker build . -t my-image:test
GOSS_FILES_PATH=test dgoss run --entrypoint=/test/gossEntrypoint.sh my-image:test
# OR provided wrapper for this tutorial
test/runTests.sh

The commands above will build a new Docker image, mount Goss and goss.yaml, and execute our tests.

That output is not very exciting, but that’s the way passing tests should be. So, let’s break something!

#!/bin/bash
#
# Any changes are written to the SCALING_FILE
#
set -euo pipefail

#this won't work..
CMD=`./nonexistentScript.sh`

SCALING_FILE=$1
SCHEDULE_FILE=$2

What happens when you execute test/runTest.sh? Goss will print the expected instead of the actual output, and include an error summary. It also exits with a non-zero status which is important for our next step: integrating with our continuous integration pipeline!

Running our tests on every commit

It’s great that we’ve got tests running locally, but the idea here is to integrate Docker-level testing into our CI/CD pipeline so that we don’t ship bad images.

CircleCI does not provide an image with Goss preloaded, but installing it only takes a second. You can view the sample config.yml for the full setup. I’m including just the relevant snippets here.

jobs:
  test:
    docker:
      - image: circleci/python:2-jessie
    steps:
      - checkout

      - setup_remote_docker:   # (2)
          docker_layer_caching: true # (3)
      - run:
          name: Install goss
          command: |
            # rather than give internet scripts SU rights, we install to local user bin and add to path
            mkdir ~/bin
            export GOSS_DST=~/bin
            export PATH=$PATH:~/bin
            curl -fsSL https://goss.rocks/install | sh
            goss -version
      - run:
          name: Test
          command: |
            # Don't forget path!
            export PATH=$PATH:~/bin
            # Important, change from mount to work on remote docker, see https://github.com/aelsabbahy/goss/pull/271
            # If using machine image you do not need this.
            export GOSS_FILES_STRATEGY=cp
            test/runTests.sh junit
      - store_test_results:
          path: goss

Test results

Note: for CI execution we pass the argument junit to our test runner. This converts the output format and pipes it to ~/goss/report.xml to be included in CircleCI’s test summary:

Summary

Well that’s it! With this basic structure running, you can add more mature tests, and include Goss-based image testing as a core step in your CI/CD pipeline to keep your team’s codebase tested from service to server.

Be sure to add that happy green status badge to your repo to let the world know!

Happy shipping!