Note from the publisher: You have managed to find some of our old content and it may be outdated and/or incorrect. Try searching in our docs or on the blog for current information.
TL;DR: CircleCI now tells you exactly which tests failed in our web UI, email notifications, and chat integrations.
CircleCI was founded on the promise of making testing less of a pain for developers, and today we’re announcing a big step forward in delivering on that promise. Most of us are used to learning of a failure on CI being the beginning of a little hunting session. We get an email or a chat message that says a build failed, and then the spelunking begins to track down the root cause. CircleCI now mitigates or removes this nuisance by displaying detailed information about which exact test cases failed in our web UI, build emails, and chat integrations.
Of course there will still be cases where it is tricky to track down the exact cause of a failure on CI, but we’ve found that just seeing the specific tests that fails results in a sort of “Oh yea, duh…” moment, where it is obvious why the last push caused some integration test to break, completely eliminating the hassle of digging through STDOUT for failure details.
Thank you, JUnit!
Many of us programming Clojure, JavaScript, or Ruby day-to-day probably don’t often think of the leading Java unit test runner, but JUnit’s xml output format has come to be the de facto standard for machine-readable test output, supported by test runners on many platforms, from Ruby to Node.js. Because of this widespread support, CircleCI can now understand the results of any test runner that can produce JUnit-formatted XML. (We actually also handle Cucumber’s JSON output, but JUnit XML is much more of a standard.)
Out-of-the-box support
CircleCI’s inferred test commands will automatically collect and interpret test failure details for the following test runners:
-
RSpec
-
Minitest
-
Cucumber
-
nose
-
Xcode
If you run your tests with CircleCI’s automatic test commands for one of these runners, everything should just work. (Note though that in the case of RSpec or Minitest you will need to include a special JUnit-formatting gem in your project. See the docs for more info.)
Other test runners
If you use a different test runner, or if you override CircleCI’s default test commands, don’t despair! You can still enjoy all the benefits of detailed test failure info. All you need to do is make sure that you save any JUnit-formatted XML to the directory ` $CIRCLE_TEST_REPORTS/<my-runner>/<my-tests>.xml, where <my-runner> and <my-tests>` are names of your choosing. We at CircleCI still use this method to save our Karma test output. See the docs for all the details.
Much more to come!
This is not nearly the end of CircleCI’s evolution into a smarter, stronger, better understander of your tests! We have a lot of related features slated to come out soon, so keep your eye out for more. Also, if you have any ideas of your own about how we could be smarter about running your tests, or if you have any other feedback at all, please give us a shout at sayhi@circleci.com!
