Docs
Start Building for Free
CircleCI.comAcademyBlogCommunitySupport

Browser testing

1 week ago6 min read
Cloud
Server v4+
On This Page

This page describes common methods for running and debugging browser testing in your CircleCI config.

Overview

Every time you commit and push code, CircleCI automatically runs your tests against the browsers you choose. You can configure your browser-based tests to run:

  • On every change

  • Before every deployment

  • On a specified branch

Prerequisites

Refer to the Pre-built CircleCI convenience images and add -browsers: to the image name. This will give you a variant that includes Java 8, geckodriver, Firefox, and Chrome. Add -browsers-legacy to the image name for a variant which includes PhantomJS.

Selenium

Automation tools used for browser tests often use Selenium WebDriver, a widely-adopted browser driving standard.

Selenium WebDriver provides a common API for programmatically driving browsers implemented in several popular languages, including Java, Python, and Ruby. Because Selenium WebDriver provides a unified interface for these browsers, you only need to write your browser tests once. These tests will work across all browsers and platforms. See the Selenium documentation for details on setup. Refer to the Xvfb man page for virtual framebuffer X Server documentation.

WebDriver can operate in two modes, local, or remote:

  • When run locally, your tests use the Selenium WebDriver library to communicate directly with a browser on the same machine.

  • When run remotely, your tests interact with a Selenium server, and the server drives the browsers.

If Selenium is not included in your primary Docker image, install, and run Selenium as shown below:

version: 2.1
jobs:
  build:
    docker:
      - image: cimg/node:16.13.1-browsers
    steps:
      - checkout
      - run: mkdir test-reports
      - run:
          name: Download Selenium
          command: curl -O http://selenium-release.storage.googleapis.com/3.5/selenium-server-standalone-3.5.3.jar
      - run:
          name: Start Selenium
          command: java -jar selenium-server-standalone-3.5.3.jar -log test-reports/selenium.log
          background: true

As an alternative, move to cloud-based platforms such as LambdaTest, Sauce Labs, or BrowserStack. These cross browser testing clouds provide you with a ready-made infrastructure so you do not have to spend time configuring a Selenium environment.

LambdaTest

LambdaTest integrates with CircleCI to boost your go-to-market delivery. Perform automated cross browser testing with LambdaTest. Ensure your development code renders seamlessly through an online Selenium grid providing 2000+ real browsers running through machines, on the cloud. Perform automation testing in parallel with LambdaTest’s Selenium grid to drastically trim down your test cycles.

LambdaTest provides an SSH (Secure Shell) tunnel connection, Lambda Tunnel, to help you perform cross browser testing of your locally stored web pages. With Lambda Tunnel, you can see how your website will look to your audience before making it live. Lambda Tunnel executes a test server inside your CircleCI build container to perform automated cross-browser testing on the range of browsers offered by Selenium Grid on LambdaTest.

LambdaTest has developed a CircleCI orb for browser compatibility testing that enables you to open a Lambda Tunnel before performing any browser testing, easing the process of integrating LambdaTest with CircleCI. Use the orb to set up a Lambda tunnel and the define your test steps.

version: 2.1

orbs:
  lambda-tunnel: lambdatest/lambda-tunnel@0.0.1

jobs:
  lambdatest/with_tunnel:
    tunnel_name: <your-tunnel-name>
    steps:
      - <your-test-steps>

Sauce Labs

Sauce Labs has an extensive network of operating system and browser combinations you can test your web application against. Sauce Labs supports automated web app testing using Selenium WebDriver scripts as well as through saucectl, their test orchestrator CLI. Use saucectl to execute tests directly from a variety of JavaScript frameworks.

saucectl

If you are using JavaScript to test your web application, you can still take advantage of the Sauce Labs platform by using saucectl with the JS framework of your choice, and then integrating the saucectl-run orb in your CircleCI workflow.

  1. Add your SAUCE_USERNAME and SAUCE_ACCESS_KEY as environment variables in your Circle CI project.

  2. Edit your CircleCI project config.yml to include the saucectl-run orb and then call the orb as a job in your workflow.

version: 2.1

orbs:
  saucectl: saucelabs/saucectl-run@2.0.0

jobs:
  test-cypress:
    docker:
      - image: cimg/node:lts
    steps:
      - checkout
      - setup_remote_docker:
          version: 20.10.2
      - saucectl/saucectl-run

workflows:
  default_workflow:
    jobs:
      - test-cypress

BrowserStack and Appium

Following the example above, you can replace the installation of Sauce Labs with another cross-browser testing platform, such as BrowserStack. Then, set the USERNAME and ACCESS_KEY environment variables to those associated with your BrowserStack account.

For mobile applications, you can use Appium or an equivalent platform that uses the WebDriver protocol. To do this, install Appium in your job, and use CircleCI environment variables for the USERNAME and ACCESS_KEY.

Cypress

Another browser testing solution you can use in your JavaScript end-to-end testing is Cypress. Unlike a Selenium-architected browser testing solution, when using Cypress, you can run tests in the same run-loop as your application.

To simplify this process, use the Cypress orb. You can run all Cypress tests without posting the results to your Cypress dashboard. The example below shows configuration for this scenario:

version: 2.1

orbs:
  cypress: cypress-io/cypress@1

workflows:
  build:
    jobs:
      - cypress/run:
          no-workspace: true

For more examples using the Cypress orb, refer to the Cypress orb page in the CircleCI orbs registry.

Debugging browser tests

This section provides some examples of how to debug browser tests on CircleCI.

Using screenshots and artifacts

Configure CircleCI to collect build artifacts and make them available from your build. For example, artifacts enable you to save screenshots as part of your job, and view them when the job finishes. You must explicitly collect those files with the store_artifacts step and specify the path and destination. See the store_artifacts section of the configuration reference for an example.

Saving screenshots is supported by most test suites, and is a built-in feature in WebKit and Selenium:

Using a local browser to access HTTP server on CircleCI

If your test runs an HTTP server on CircleCI, use a browser running on your local machine to debug a failing test. You can do this with an SSH-enabled run.

  1. Run an SSH build using the Rerun Job with SSH button on the Job page of the CircleCI app. The command to log into the container over SSH is as follows:

    ssh -p 64625 ubuntu@54.221.135.43

  2. To add port-forwarding to the command, use the -L flag. The following example forwards requests to http://localhost:3000 on your local browser to port 8080 on the CircleCI container. An example use case for this would be if your job runs a debug Ruby on Rails app, which listens on port 8080. After you run this, if you go to your local browser and request http://localhost:3000, you should see whatever is served on port 8080 of the container.

    ssh -p 64625 ubuntu@54.221.135.43 -L 3000:localhost:8080

  3. Then, open your browser on your local machine and navigate to http://localhost:3000 to send requests directly to the server running on port 8080 on the CircleCI container. You can also manually start the test server on the CircleCI container (if it is not already running), and you should be able to access the running test server from the browser on your development machine.

The steps described here are a good way to debug when setting up Selenium tests.

Interacting with the browser over VNC

VNC allows you to view and interact with the browser that is running your tests. This only works if you are using a driver that runs a real browser. You can interact with a browser that Selenium controls, but PhantomJS is headless, so there is nothing to interact with.

  1. Install a VNC viewer. If you are using macOS, consider Chicken of the VNC. RealVNC is also available on most platforms.

  2. Open a Terminal window, start an SSH run to a CircleCI container and forward the remote port 5901 to the local port 5902.

    ssh -p PORT ubuntu@IP_ADDRESS -L 5902:localhost:5901

  3. Install the vnc4server and metacity packages. You can use metacity to move the browser around and return to your Terminal window.

    sudo apt install vnc4server metacity

  4. After connecting to the CircleCI container, start the VNC server.

    ubuntu@box159:~$ vncserver -geometry 1280x1024 -depth 24

  5. Since your connection is secured with SSH, there is no need for a strong password. You do still need a password, so enter password at the prompt.

  6. Start your VNC viewer and connect to localhost:5902. Enter your password at the prompt.

  7. You should see a display containing a terminal window. Since your connection is secured through the SSH tunnel, ignore any warnings about an insecure or unencrypted connection.

  8. To allow Windows to open in the VNC server, set the DISPLAY variable. Without this command, Windows would open in the default (headless) X Server.

    ubuntu@box159:~$ export DISPLAY=:1.0

  9. Start metacity in the background.

    ubuntu@box159:~$ metacity &

  10. Start firefox in the background.

    ubuntu@box159:~$ firefox &

Now, you can run integration tests from the command line and watch the browser for unexpected behavior. You can even interact with the browser as if the tests were running on your local machine.

Sharing CircleCI’s X Server

If you find yourself setting up a VNC server often, then you might want to automate the process. You can use x11vnc to attach a VNC server to X.

  1. Download x11vnc and start it before your tests:

    jobs:
  build:
    docker:
      - image: cimg/node:17.2.0
    steps:
      - run:
          name: Download and start X
          command: |
            sudo apt-get install -y x11vnc
            x11vnc -forever -nopw
          background: true

  2. Now when you start an SSH build, you’ll be able to connect to the VNC server while your default test steps run. You can either use a VNC viewer that is capable of SSH tunneling, or set up a tunnel on your own:

    ssh -p PORT ubuntu@IP_ADDRESS -L 5900:localhost:5900

X11 forwarding over SSH

CircleCI also supports X11 forwarding over SSH. X11 forwarding is similar to VNC — you can interact with the browser running on CircleCI from your local machine.

  1. Install an X Window System on your computer. If you are using macOS, consider XQuartz.

  2. With X set up on your system, start an SSH build to a CircleCI VM, using the -X flag to set up forwarding:

    daniel@mymac$ ssh -X -p PORT ubuntu@IP_ADDRESS

    This will start an SSH session with X11 forwarding enabled.

  3. To connect your VMs display to your machine, set the display environment variable to localhost:10.0

    ubuntu@box10$ export DISPLAY=localhost:10.0

  4. Check that everything is working by starting xclock.

    ubuntu@box10$ xclock

    You can kill xclock with Ctrl+c after it appears on your desktop.

Now you can run your integration tests from the command line and watch the browser for unexpected behavior. You can even interact with the browser as if the tests were running on your local machine.

See also

Suggest an edit to this page

Make a contribution
Learn how to contribute

Still need help?

Ask the CircleCI community
Join the research community
Visit our Support site