At CircleCI, we’ve got a lot of tests. We build a product that’s fundamental to the workflow of developers across the world, so we try hard to validate that what goes into production is well tested. Historically, we’ve tested our frontend entirely through Webdriver tests, but with our transition to using ClojureScript for our UI, we took some time to reevaluate how to best test our new code.

As great as Webdriver is when it comes to automating browser interactions, it’s often hard to get tests working quickly and reliably for a myriad of reasons:

  • AJAX calls– one of the more common issues that we see customers run into is nondeterministic tests failing due to not waiting long enough for asynchronous activities to finish up. We have this pain too.

  • Webdriver is way slower than Karma. In contrast to running a suite of standalone unit tests with Karma, the webdriver flow goes something like this:

    • The client library that runs the tests tells webdriver what to do.

    • Webdriver tells the browser what to do.

    • Browser starts doing it. Webdriver waits.

    • Browser finishes. Webdriver responds to the client.

    • Rinse and repeat.

  • When step three involves loading a full web page (loading HTML, CSS, JavaScript, images at the very least), rendering the page, and the overhead of network requests needed for the different webdriver pieces to communicate, it’s only natural that webdriver tests are going to take longer than a standalone suite of tests that run independently of the site itself.

  • Compatibility between browsers, Webdriver clients, and Selenium server often breaks between releases. This tends to manifest as mysterious test timeouts or errors indicating that the client was unable to connect to the Selenium server.

  • You need to actually have a server running to serve content. For unit testing purposes, You shouldn’t need all of the underlying dependencies that the backend introduces to verify that individual portions of the site behave properly.

Consequently, we decided that converting more of our frontend tests to clojurescript.test would help reduce test flakiness and enable us to better test a wider variety of application states.

If you’re already using Clojure on the backend, then there’s a decent likelihood that you’re using Clojure’s core.test already for running your tests. In any case, clojure.test is what we use, so writing tests in the same fashion regardless of whether we’re developing on the frontend or backend is convenient– we don’t have to deal with the confusion of slightly different syntaxes depending on what we’re developing.

With our test library settled on, we still needed a way to actually run the frontend tests. Many test runners for frontend code (e.g. SlimerJS et al) only run in the console, so when a test fails, you can’t readily reach out to the lovely developer tools that modern browsers ship with. Since we already have the mental overhead of having to correlate JavaScript errors and stack traces of compiled JavaScript back to ClojureScript, being able to use a debugger against tests is an absolute necessity.

Enter Karma.

Karma is a test runner for JavaScript that makes it easy to quickly run tests. It was initially developed by the Angular team to make it easier to test the Angular codebase against a wide array of modern browsers. Karma also comes with built-in support for watching file changes, so it’s a snap to re-run your frontend unit tests on each save.

If your browser supports websockets and iframes, Karma can run tests for your browser when you navigate to the page http://localhost:9876 by default). Karma comes with built-in support for launching Chrome, Firefox, and PhantomJS, but there are a number of other community supported browser integrations as well.

Additionally, with our ongoing efforts to improve test failure reporting, Karma’s ability to output colorized test results in development and JUnit XML during our builds lets us quickly see which tests have failed via the CircleCI UI.

Like all tools, Karma has tasks that it accomplishes well, and tasks that it doesn’t. When should you use Karma instead of Webdriver? In short, Karma probably isn’t the right tool for the job if you’re trying to write integration tests. Since Karma effectively loads your tests into a sandbox (an iframe) in the browser, you typically can’t (and shouldn’t) attempt to interact with the outside world. Navigating to a different page will cause tests to break, and unless you set up CORS appropriately, you can’t make AJAX calls either.

A word on frontend testing best practices

In most cases, you probably shouldn’t actually be attaching elements to the DOM. There are a few good reasons to skip attaching elements if you can:

  • Appending to the DOM is a somewhat costly operation. The browser has to determine whether to trigger a reflow, apply CSS rules, determine whether to notify MutationObservers, etc. While these are all necessary actions for presenting a page to a user, they often don’t actually need to occur for unit tests to work. Anecdotally, I’ve seen tests that unnecessarily attach elements to the DOM take up to 100x longer to run.

  • Performance aside, consistently cleaning up the DOM between tests is hard. Uncaught exceptions or squirrelly third party libraries that dump elements into the DOM will-nilly (cough jQuery UI cough) can leave elements in the DOM that interfere with future tests.

Moral of the story? If you don’t really need the DOM, don’t use it!

How it all works

The big hiccup that we encountered as we started moving towards clojurescript.test + Karma is that there weren’t any existing plugins for Karma that provided clojurescript.test integration that actually worked.

OK, easy enough to build, right? However, the documentation around writing one’s own karma plugins is pretty scarce, so we had to poke around in existing plugin source code to figure out how to wire it up.

In a nutshell, Karma expects a test framework adapter to have a file called index.js that tells Karma how to load up the entry-point for your tests. Typically this means providing a small adapter file that gets loaded into the browser.


1   var createPattern = function(path) {
2     return {
3       pattern: path,
4       included: true,
5       served: true,
6       watched: false
7     };
8   };
10  var initClojurescriptTest = function (files) {
11    // Add the adapter.js file from the clojurescript.test
12    // plugin to the list of files to load for running tests.
13    files.unshift(createPattern(__dirname + '/adapter.js'));
14  };
1   // Setting the $inject property tells karma what arguments
2   // to pass in when the initClojurescriptTest function is
3   // invoked by the test runner. In our case, we just care
4   // about adding a setup file to the user-specified file list.
5   initClojurescriptTest.$inject = ['config.files'];
6   module.exports = {
7     // The 'framework:' prefix is magical.
8     // In the karma configuration file
9     // for a project, you just add 'cljsTest' to
10    // the frameworks array.
11    'framework:cljsTest': ['factory', initClojurescriptTest]
12  };


Here’s where we run into the one unresolved issue involved in our setup: to actually report test results and information printed during the execution of ClojureScript tests, we have to hook into cljs.test itself. While this is probably technically possible to do in JavaScript, it’s a bit of a pain. Additionally, advanced compilation of ClojureScript causes variables not explicitly exported to be renamed, and potentially even optimized away. Consequently, the adapter.js file that we load into the browser is only responsible for ensuring that all of the test files are loaded and calling circle.karma.run_tests_for_karma, which is located in our frontend test code:

1   // loaded into the browser at test run time.
2   (function(karma, window) {
3     var createClojureScriptTest = function (tc, runnerPassedIn) {
4       return function () {
5         // a hack to load all of the separate files when compiling
6         // with :optimizations :none
7         if ('undefined' !== typeof goog && goog.dependencies_
8           && goog.dependencies_.nameToPath) {
9           for(var namespace in goog.dependencies_.nameToPath)
10            goog.require(namespace);
11        };
12        circle.karma.run_tests_for_karma(tc);
13      };
14    };
16    karma.start = createClojureScriptTest(karma);
17  })(window.__karma__, window);

Thankfully, our front-end source code is open source, so the code is open for inspection, so feel free to use our code for your own testing needs:


1   (ns circle.karma
2     (:require [clojure.string :as string]
3               [cemerick.cljs.test :as test])
4     (:require-macros [cemerick.cljs.test :as test]))
6   (defn get-total-test-count []
7     (reduce + (map count (vals @test/registered-tests))))
9   ;; A report function to override the default cljs-test one.
10  (defmulti report :type)
12  (defmethod report :begin-test-var [{:keys [test-env]}]
13    ;; Set the current timer.
14    (swap! test-env assoc ::test-start (.getTime (js/Date.)))
15    ;; Collect the output.
16    (swap! test-env assoc ::test-output [])
17    (swap! test-env assoc ::old-print-fn *print-fn*)
18    (test/set-print-fn!
19      (fn [output]
20        (swap! test-env update-in [::test-output] #(conj %1 output)))))
22  (defmethod report :end-test-var [{:keys [test-env] :as report}]
23    (let [;; Get the start time from the test env, and the end time is now.
24          start (-> @test-env ::test-start )
25          end (.getTime (js/Date.))
26          ;; Collect the output from the test env.
27          output (->> @test-env ::test-output (string/join "\n"))]
28      (test/set-print-fn! (-> @test-env ::old-print-fn))
29      ;; Clean up the test env.
30      (swap! test-env dissoc
31          ::test-start
32          ::test-output
33          ::old-print-fn)
34      ;; Report results to karma.
35      (.result js/__karma__
36               (clj->js { "id" ""
37                         "description" (-> report :test-name str) 
38                          "suite" [(-> report :test-name
39                                              namespace
40                                              str)]
41                          "success" (and (zero? (:error @test-env))
42                                         (zero? (:fail @test-env)))
43                          "skipped" nil
44                          "time" (- end start)
45                          "log" [output]}))))
47  ;; Make beginning a namespace's tests less noisy.
48  (defmethod report :begin-test-ns [_])
50  ;; Fall back to the default report function so that
51  ;; e.g. errors are logged correctly.
52  (let [cljs-test-report test/report]
53    (defmethod report :default [data]
54      (cljs-test-report data)))
56  (defn ^:export run-tests-for-karma []
57    (.info js/__karma__ (clj->js {:total (get-total-test-count)}))
58    (doseq [[ns ns-tests] @test/registered-tests]
59      (with-redefs [test/report report]
60        (test/test-ns ns)))
61    (.complete js/__karma__ (clj->js {}))) 

Putting it All Together

With all of that infrastructure in place, you can now write tests for your ClojureScript frontend with confidence! Here’s the output of one of our very own green builds:


A big advantage of using Karma is that we can leverage existing plugins like JUnit XML test reporter to generate test reports. On a good day, it’ll look something like this:


Sometimes we make mistakes though, and Karma makes it easy to see where things went wrong!

6 failing Karma tests

We hope this brief tour of testing ClojureScript with Karma will inspire you to write your own blazingly fast frontend tests!

Discuss on Hacker News