Test, test again

Here at CircleCI, we care a LOT about testing. And after running more than 65 million builds, we have learned a thing or two. Empowering teams to test often and better is a driving force in what we do. Running tests makes your code more reliable.

But are you getting the most out of your tests? Beyond simply implementing them, are you adequately testing your tests? After all, the tests themselves are only as good as they are reliable.

If you can’t trust tests when they fail, why should you trust them when they pass?

A flaky test (one that passes sometimes and fails other times when the code hasn’t changed) is hardly better than no test at all. In the Insights tab, you can quickly see which of your tests might be flaky, as well as which are your most-failed and slowest running tests. While we can’t re-write your tests for you, we can help you see exactly which tests are slow, flaky and failing, so that your team can have laser focus when working on shoring up your tests.

You can only manage what you can measure

But don’t take our word for it! Our own tests haven’t always been great. They used to be slow, and it was frustrating. But we didn’t know why, or specifically which ones were slow. Engineer Marc O’Morain knew he couldn’t speed up everyone’s tests, but he could do the next best thing: shame encourage them into doing it themselves.

He made a small change to the build page, so that it would display the time taken by the slowest test by default. His hope was to guilt people into speeding up their tests. And it worked. After shipping, we could all see that our slowest test was taking 2 minutes 20 seconds, and it didn’t take long for someone to sort that out.

Now that our team was aware of the weak spots in our testing operation, we were able to improve the worst offending tests and speed up our testing.

We want to help you get the most out of your tests, so here are our favorite tools, now available to all CircleCI users:

Failed tests: Which of your tests are failing the most? Failed Tests

Slowest Failed Tests: Of your failed tests, which are the slowest? Start with these when trying to speed up your tests. Slowest Failed Tests

Success Rate: How many builds on a branch are successful? On master, where tests should be passing at 100%, any lesser percentage could indicate tests in your arsenal that might be flaky. Success Rate1.png