Test > Browser Testing
This document describes common methods for running and debugging browser testing in your CircleCI config in the following sections:
- Sauce Labs
- BrowserStack and Appium
- Debugging Browser Tests
- X11 forwarding over SSH
Refer to the Pre-Built CircleCI Docker Images and add
-browsers: to the image name for a variant that includes Java 8, PhantomJS, Firefox, and Chrome.
Every time you commit and push code, CircleCI automatically runs all of your tests against the browsers you choose. You can configure your browser-based tests to run whenever a change is made, before every deployment, or on a certain branch.
Many automation tools used for browser tests use Selenium WebDriver, a widely-adopted browser driving standard.
Selenium WebDriver provides a common API for programatically driving browsers implemented in several popular languages, including Java, Python, and Ruby. Because Selenium WebDriver provides a unified interface for these browsers, you only need to write your browser tests once. These tests will work across all browsers and platforms. See the Selenium documentation for details on set up. Refer to the Xvfb man page for virtual framebuffer X server documentation.
WebDriver can operate in two modes: local or remote. When run locally, your tests use the Selenium WebDriver library to communicate directly with a browser on the same machine. When run remotely, your tests interact with a Selenium Server, and it is up to the server to drive the browsers.
If Selenium is not included in your primary docker image, install and run Selenium as shown below::
version: 2 jobs: build: docker: - image: circleci/node-browsers steps: - checkout - run: mkdir test-reports - run: name: Download Selenium command: curl -O http://selenium-release.storage.googleapis.com/3.5/selenium-server-standalone-3.5.3.jar - run: name: Start Selenium command: java -jar selenium-server-standalone-3.5.3.jar -log test-reports/selenium.log background: true
Refer to the Install and Run Selenium to Automate Browser Testing section of the 2.0 Project Tutorial for a sample application. Refer to the Knapsack Pro documentation for an example of Capybara/Selenium/Chrome headless CircleCI 2.0 configuration for Ruby on Rails.
For more information about working with Headless Chrome, see the CircleCI blog post Headless Chrome for More Reliable, Efficient Browser Testing and the related discuss thread.
As an alternative to configuring your environment for Selenium, Sauce Labs provides a Selenium Server as a service, with a large number of browsers and system combinations available to test. Sauce Labs also has some extra goodies like videos of all test runs.
Sauce Labs operates browsers on a network that is separate from CircleCI build containers. To enable the browsers with a way to access the web application you want to test, you can run Selenium WebDriver tests with Sauce Labs on CircleCI using Sauce Labs’ secure tunnel, Sauce Connect.
Sauce Connect allows you to run a test server within the CircleCI build container
and expose it (using a URL like
localhost:8080) to Sauce Labs’ browsers. If you
run your browser tests after deploying to a publicly accessible staging environment,
then you can use Sauce Labs in the usual way without worrying about Sauce Connect.
config.yml file demonstrates how to run browser tests through Sauce Labs
against a test server running within a CircleCI build container.
version: 2 jobs: build: docker: - image: circleci/python:jessie-node-browsers steps: - checkout - run: name: Install Sauce command: npm install saucelabs - run: name: sauce testing command: npm run-script sauce environment: SAUCE_USERNAME: # Refer to circleci.com/docs/2.0/env-vars documentation for info SAUCE_ACCESS_KEY: # about setting up environment variables for auth secrets - run: # Wait for app to be ready command: wget --retry-connrefused --no-check-certificate -T 30 http://localhost:5000 - run: # Run selenium tests command: nosetests - run: # wait for Sauce Connect to close the tunnel command: killall --wait sc
BrowserStack and Appium
As in the Sauce Labs example above, you could replace the installation of Sauce Labs with an installation of another cross-browser testing platform such as BrowserStack. Then, set the USERNAME and ACCESS_KEY environment variables to those associated with your BrowserStack account.
For mobile applications, it is possible to use Appium or an equivalent platform that also uses the WebDriver protocol by installing Appium in your job and using CircleCI environment variables for the USERNAME and ACCESS_KEY.
Debugging Browser Tests
Integration tests can be hard to debug, especially when they’re running on a remote machine. This section provides some examples of how to debug browser tests on CircleCI.
Using Screenshots and Artifacts
CircleCI may be configured to collect build artifacts
and make them available from your build. For example, artifacts enable you to save screenshots as part of your job,
and view them when the job finishes. You must explicitly collect those files with the
store_artifacts step and specify the
destination. See the store_artifacts section of the Configuring CircleCI document for an example.
Saving screenshots is straightforward: it’s a built-in feature in WebKit and Selenium, and is supported by most test suites:
- Manually, using Selenium directly
- Automatically on failure, using Cucumber
- Automatically on failure, using Behat and Mink
Using a Local Browser to Access HTTP server on CircleCI
If you are running a test that runs an HTTP server on CircleCI, it is sometimes helpful to use a browser running on your local machine to debug a failing test. Setting this up is easy with an SSH-enabled run.
- Run an SSH build using the Rerun Job with SSH button on the Job page of the CircleCI app. The command to log into
the container over SSH apears, as follows:
ssh -p 64625 firstname.lastname@example.org
- To add port-forwarding to the command, use the
-Lflag. The following example forwards requests to
8080on the CircleCI container. This would be useful, for example, if your job runs a debug Ruby on Rails app, which listens on port 8080.
ssh -p 64625 email@example.com -L 3000:localhost:8080
- Then, open your browser on your local machine and navigate to
http://localhost:8080to send requests directly to the server running on port
3000on the CircleCI container. You can also manually start the test server on the CircleCI container (if it is not already running), and you should be able to access the running test server from the browser on your development machine.
This is a very easy way to debug things when setting up Selenium tests, for example.
Interacting With the Browser Over VNC
VNC allows you to view and interact with the browser that is running your tests. This only works if you are using a driver that runs a real browser. You can interact with a browser that Selenium controls, but PhantomJS is headless, so there is nothing to interact with.
- Open a Terminal window,
start an SSH run to a CircleCI container
and forward the remote port 5901 to the local port 5902.
ssh -p PORT ubuntu@IP_ADDRESS -L 5902:localhost:5901
- Install the
metacitypackages. You can use
metacityto move the browser around and return to your Terminal window.
sudo apt install vnc4server metacity
- After connecting to the CircleCI container, start the VNC server.
ubuntu@box159:~$ vnc4server -geometry 1280x1024 -depth 24
Since your connection is secured with SSH, there is no need for a strong password. However, you still need a password, so enter
passwordat the prompt.
Start your VNC viewer and connect to
localhost:5902. Enter your
passwordat the prompt.
You should see a display containing a terminal window. Since your connection is secured through the SSH tunnel, ignore any warnings about an insecure or unencrypted connection.
- To allow windows to open in the VNC server,
DISPLAYvariable. Without this command, windows would open in the default (headless) X server.
ubuntu@box159:~$ export DISPLAY=:1.0
metacityin the background.
ubuntu@box159:~$ metacity &
firefoxin the background.
ubuntu@box159:~$ firefox &
Now, you can run integration tests from the command line and watch the browser for unexpected behavior. You can even interact with the browser as if the tests were running on your local machine.
Sharing CircleCI’s X Server
If you find yourself setting up a VNC server often, then you might want to automate the process. You can use
x11vnc to attach a VNC server to X.
x11vncand start it before your tests:
steps: - run: name: Download and start X command: | sudo apt-get install -y x11vnc x11vnc -forever -nopw: background: true
- Now when you start an SSH build, you’ll be able to connect to the VNC server while your default test steps run. You can either use a VNC viewer that is capable of SSH tunneling, or set up a tunnel on your own:
$ ssh -p PORT ubuntu@IP_ADDRESS -L 5900:localhost:5900
X11 forwarding over SSH
CircleCI also supports X11 forwarding over SSH. X11 forwarding is similar to VNC — you can interact with the browser running on CircleCI from your local machine.
Install an X Window System on your computer. If you’re using macOS, consider XQuartz.
- With X set up on your system, start an SSH build to a CircleCI VM, using the
-Xflag to set up forwarding:
daniel@mymac$ ssh -X -p PORT ubuntu@IP_ADDRESS
This will start an SSH session with X11 forwarding enabled.
- To connect your VM’s display to your machine, set the display environment variable to
ubuntu@box10$ export DISPLAY=localhost:10.0
- Check that everything is working by starting xclock.
You can kill xclock with
Ctrl+cafter it appears on your desktop.
Now you can run your integration tests from the command line and watch the browser for unexpected behavior. You can even interact with the browser as if the tests were running on your local machine.