Adding test coverage to your continuous integration pipeline

This tutorial covers:

1. Creating a sample Ruby codebase with test coverage
2. Setting up a CI pipeline for the codebase
3. Sending test coverage results to Coveralls

One of the key indicators of a healthy codebase is good test coverage. Once you’ve bought into the value of CI/CD, it makes sense to use a test coverage service to track changes to your project over time. Not only will it ensure tests increase at the same rate as code, it can also help you control your development workflow with pass/fail checks and PR comments showing where coverage is lacking and how to improve it.

In this tutorial, you will put a simple Ruby codebase with test coverage into a CI pipeline. Then you will configure CircleCI to send your project’s test coverage results to Coveralls, a popular test coverage service that is free for open source projects. Using CircleCI’s orb technology, makes it fast and easy to integrate with third-party tools like Coveralls.

Prerequisites

To follow along with this post, you’ll need the following:

• Enough familiarity with Ruby to read some basic code and tests
• CircleCI account
• GitHub account
• Free or enterprise Coveralls account

Our tutorials are platform-agnostic, but use CircleCI as an example. If you don’t have a CircleCI account, sign up for a free one here.

How test coverage works

If you’re new to test coverage, here’s how it works:

For a project made up of code and tests, a test coverage library can be added to assess how well the project’s code is being covered by its tests. On each run of the project’s test suite, the test coverage library generates a test coverage report.

In this case, you will use a test coverage library called Simplecov, a code coverage analysis tool for Ruby. You will use this library to determine the percentage of code that has been tested in your Ruby project.

How test coverage works in CI/CD

1. You push changes to your code at your source code management (SCM) tool such as GitHub.
2. Your CI service builds your project, runs your tests, and generates your test coverage report.
3. Your CI posts the test coverage report to Coveralls.
4. Coveralls publishes your coverage changes to a shared workspace.
5. Coveralls can send comments and pass/fail checks to your PRs to control your development workflow.

Cloning the test coverage sample app

Clone the Ruby project into a ruby-coveralls-project folder from GitHub using this command:

git clone https://github.com/CIRCLECI-GWP/ruby-coveralls-project.git

All of the code in this project is located in lib/class_one.rb:

class ClassOne

  def self.covered
    "covered"
  end

  def self.uncovered
    "uncovered"
  end

end

And these are the tests in spec/class_one_spec.rb:

require 'spec_helper'
require 'class_one'

describe ClassOne do

  describe "covered" do
    it "returns 'covered'" do
      expect(ClassOne.covered).to eql("covered")
    end
  end

  # Uncomment below to achieve 100% coverage
  # describe "uncovered" do
  #   it "returns 'uncovered'" do
  #     expect(ClassOne.uncovered).to eql("uncovered")
  #   end
  # end
end

Note: Right now, only one of the two methods in ClassOne is being tested.

At this point you have installed:

• Simplecov: A test coverage library
• [RSpec JUnit Formatter](https://www.rubydoc.info/gems/rspec_junit_formatter)

as gems in your Gemfile.

# frozen_string_literal: true

source "https://rubygems.org"

# gem "rails"

gem 'rspec'
gem 'simplecov'
gem 'rspec_junit_formatter'

You have passed a configuration setting to Simplecov in your spec/spec_helper.rb telling it to ignore files in the test directory:

require 'simplecov'

SimpleCov.start do
  add_filter "/spec/"
end

Running tests

Run the test suite for the first time and review the results:

bundle exec rspec

Results:

.

Finished in 0.0029 seconds (files took 0.88058 seconds to load)
1 example, 0 failures

Coverage report generated for RSpec to /Users/yemiwebby/ruby-demo-coverall/coverage. 4 / 5 LOC (80.0%) covered.

Note: In addition to the providing the test results, Simplecov generated a test coverage report in a new /coverage directory.

Conveniently, it generated those results in HTML format, which you can open like this:

open coverage/index.html

The first coverage report looks like this:

Coverage stands at 80% for the entire project.

Click lib/class_one.rb to review results for the file.

You’ll notice covered lines in green, and uncovered lines in red. In this case, 4/5 lines are covered, translating to 80% coverage.

Adding tests to complete coverage

To add tests, un-comment the test of the second method in ClassOne:

require 'spec_helper'
require 'class_one'

describe ClassOne do

  describe "covered" do
    it "returns 'covered'" do
      expect(ClassOne.covered).to eql("covered")
    end
  end

  # Uncomment below to achieve 100% coverage
  describe "uncovered" do
    it "returns 'uncovered'" do
      expect(ClassOne.uncovered).to eql("uncovered")
    end
  end
end

Now run the test suite again:

bundle exec rspec

Open the new results at coverage/index.html.

Coverage has increased from 80% to 100% (and turned green).

Click lib/class_one.rb.

Five out of five relevant lines are now covered, resulting in 100% coverage for the file. That means 100% total coverage for your one-file project.

Adding a configuration file to the project

Now that you understand how test coverage works in this project, you can verify the same results through Coveralls.

First you need to set up the continuous integration pipeline on CircleCI.

At the base directory of your project, create a new folder named .circleci and create a file called config.yml in it.

Open the newly created file and use the following configuration settings for it:

version: 2.1

orbs:
  ruby: circleci/ruby@2.0.0

jobs:
  build:
    docker:
      - image: cimg/ruby:3.1.2-node
    steps:
      - checkout
      - ruby/install-deps
      - ruby/rspec-test

workflows:
  build_and_test:
    jobs:
      - build

What do those config settings mean?

This tutoria uses v2.1 of CircleCI’s configuration spec for pipelines, indicated at the top of the file:

version: 2.1

Two of the core concepts of the v2.1 config spec are orbs and workflows.

Orbs are reusable packages of configuration that can be used across projects for convenience and standardization. For this tutorial, we are using the Ruby orb, which makes quick work of setting up a new Ruby project.

orbs:
  ruby: circleci/ruby@2.0.0

Workflows are a means of collecting and orchestrating jobs. Your next step is to define a simple workflow called build_and_test.

workflows:
  build_and_test:
    jobs:
      - build

This code invokes a job, called build, that is defined to check out your code, install your dependencies, and run your tests in the CI environment. That environment is a Docker image running Ruby 3.1.2 and Node:

jobs:
  build:
    docker:
      - image: cimg/ruby:3.1.2-node
    steps:
      - checkout
      - ruby/install-deps
      - ruby/rspec-test

Jobs are the main building blocks of your pipeline, which comprise steps and the commands that do the work of your pipeline.

The final step of this job uses a built-in command for running RSpec tests that comes with CircleCI’s Ruby orb, called rspec-test:

steps:
   [...]
   - ruby/rspec-test

Not only does this provide a one-liner for running your RSpec tests, it also provides some freebies, including automated parallelization and a default test results directory.

Why use automated parallelization?

Parallelization allows you to run tests from the test suite in parallel. It improves speed, which is particularly handy when running multiple tests. For more hands-on practice, review this tutorial on test splitting in CircleCI.

Adding the project to CircleCI

Now that you have created and have an understanding of the configuration file, you need to push the project to your repository on GitHub. Review pushing a project to GitHub for intstructions.

Once you are done, log in to your CircleCI account to review all repositories linked with your GitHub account. Select ruby-coveralls-project from the list and click Set Up Project.

You will be prompted to enter the name of the branch your configuration file is on. Enter main for this tutorial and click Set Up Project to start the workflow.

This will immediately trigger the pipeline and build successfully.

The coverage report generated for RSpec? is similar to those you got locally. Just like in your local environment, Simplecov generates a coverage report and stores it in the /coverage directory.

Coverage report generated for RSpec to /home/circleci/project/coverage. 5 / 5 LOC (100.0%) covered.

You now have test coverage in CI.

Configuring the project to use Coveralls

Now you can tell CircleCI to start sending those test coverage results to Coveralls.

You’re in luck here, because Coveralls has published a Coveralls orb following the CircleCI orb standard, which makes this plug-and-play.

Before you can set this up, you’ll need to create a new account at Coveralls, which is free for individual developers with public (open source) repos.

Adding the project to Coveralls

Go to Coveralls and sign in with GitHub.

Upon first sign-in, you won’t have any active repos. Go to ADD SOME REPOS to find a list of your public repos.

To add your repo, click the toggle next to your repo name, switching it to ON.

Great! Coveralls is now tracking your repo.

Finishing the setup

Before the release of the Coveralls orb, the default approach to setting up a Ruby project to use Coveralls would be to install the Coveralls RubyGem, which leverages Simplecov as its main dependency and takes care of uploading Simplecov’s results to Coveralls.

To stick with the v2.1 config at CircleCI, and to leverage the benefits of CircleCI’s Ruby orb, set up the Coveralls orb to work with the Ruby orb.

Preparing to use the Coveralls orb

Now the first consideration, which is a little counterintuitive, is that the Coveralls orb is written in JavaScript, rather than Ruby, with a dependency of Node. This is not a problem though, because you configure the Ruby orb to install a Docker image containing both Ruby and Node:

jobs:
  build:
    docker:
      - image: cimg/ruby:3.1.2-node
    steps: [...]

Another requirement of the Coveralls orb is that it expects test coverage reports in LCOV format. To meet that requirement, just make a few more changes to your project.

First, add the simplecov-lcov gem to your Gemfile:

# Gemfile
[...]
gem 'simplecov-lcov'

Then change some Simplecov-related configuration in your spec_helper:

# spec_helper.rb
...
require 'simplecov-lcov'

SimpleCov::Formatter::LcovFormatter.config.report_with_single_file = true
SimpleCov.formatter = SimpleCov::Formatter::LcovFormatter

SimpleCov.start do
  add_filter "/spec/"
end

This specifies that simplecov-lcov is required. You need to tell Simplecov to do two things:

First, combine multiple report files into a single file:

SimpleCov::Formatter::LcovFormatter.config.report_with_single_file = true

Second, export results in LCOV format:

SimpleCov.formatter = SimpleCov::Formatter::LcovFormatter

Be sure to run bundle install to update the Gemfile.lock.

Updating .circleci/config.yml

Next, add the Coveralls orb to the orbs section of .circelci/config.yml. Update the build section with a new step:

version: 2.1

orbs:
  ruby: circleci/ruby@2.0.0
  coveralls: coveralls/coveralls@1.0.6
jobs:
  build:
    docker:
      - image: cimg/ruby:3.1.2-node
    steps:
      - checkout
      - ruby/install-deps
      - ruby/rspec-test
      - coveralls/upload:
          path_to_lcov: ./coverage/lcov/project.lcov

workflows:
  build_and_test:
    jobs:
      - build

That command, coveralls/upload, calls the Coveralls orb’s upload command. Below it, the path_to_lcov parameter is passed, telling the orb where to find the coverage report it should upload to the Coveralls API.

Note: The complete source code for the coveralls configuration can be found here on config-coveralls branch.

Adding a COVERALLS_REPO_TOKEN

If you’re using a private CI service like CircleCI, the Coveralls API requires an access token to securely identify your repo. This is called your COVERALLS_REPO_TOKEN.

You’ll encounter this if you visit the start page for your Coveralls project by clicking DETAILS before you have any builds.

You can also grab it at any time from your project’s Settings page.

To let CircleCI POST securely to the Coveralls API on behalf of your repo, just add your COVERALLS_REPO_TOKEN as an environment variable in the CircleCI web interface. Click Project Settings > Environment Variables.

Now you’re ready to send coverage results to Coveralls from CircleCI.

Push the changes you just made:

git add .
git commit -m "Finish coveralls setup."
git push

Verifying test coverage via Coveralls

Since you understand how test coverage works in this project, let’s verify those same results through the Coveralls service.

Given that you configured your project to use CircleCI and Coveralls, and pushed those changes to your repo, that last push triggered a new build at CircleCI.

Which in turn uploaded test results to the Coveralls API per the build log above.

This triggers a new build at Coveralls.

Which shows coverage at 100%. Which is as expected.

To validate that Coveralls is tracking changes in test coverage for your project, re-add that test that lifts coverage to 100%.

Now you have automated test coverage updates from Coveralls.

Next steps

Now that your project is set up to automatically to track test coverage, you can:

1. Get badged adds a nifty “coverage” badge to your repo’s README.
2. Configure PR comments informs team members of changes to test coverage before merging.
3. Set up pass/fail checks blocks merging unless coverage thresholds are met.
4. Explore more complex scenarios leverages parallelism for larger projects.

Start with the Coveralls docs here.

Conclusion

A healthy codebase is a well-tested codebase, and a healthy project is one where test coverage stays front and center throughout development.

A test coverage service like Coveralls lets you track changes to your project’s test coverage over time, surface those changes for your whole team to see, and even stop merges that degrade the project’s quality.

Using CircleCI’s orb spec, this tutorial showed how easy it can be to connect your project with a test coverage service by making it part of your CI/CD pipeline, especially when the service leverages the configuration standard of your CI platform.