Troubleshoot test splitting

Cloud

Server v4.x

Server v3.x

Using test splitting with Python Django tests

To use test splitting with CircleCI, you must pass in a list of tests to run. However, with how you execute tests with Django, you are unable to simply glob the tests and pass them in.

Sometimes, users run into specific issues when performing test splitting for their own unique use cases. One example of a user resolving an issue when test splitting in Python Django did not perform correctly can be found in the following Discuss post.

Using this example, here is a quick example of how you can accomplish test splitting:

- run:
    command: |
      # get test files while ignoring __init__ files
      TESTFILES=$(circleci tests glob "catalog/tests/*.py" | sed 's/\S\+__init__.py//g')
      echo $TESTFILES | tr ' ' '\n' | sort | uniq > circleci_test_files.txt
      cat circleci_test_files.txt
      TESTFILES=$(circleci tests split --split-by=timings circleci_test_files.txt)
      # massage filepaths into format manage.py test accepts
      TESTFILES=$(echo $TESTFILES | tr "/" "." | sed 's/\.py$//g')
      echo $TESTFILES
      pipenv run python manage.py test --verbosity=2 $TESTFILES

Using test splitting with `pytest`

If you try to split your tests across containers with pytest, you may encounter any of the following errors:

No timing found for "tests/commands/__init__.py"
No timing found for "tests/commands/test_1.py"
No timing found for "tests/commands/test_2.py"

If any of these errors are returned, you may need to make a few adjustments, which are listed below.

Are you setting a custom `working_directory`?

If so, you may need to adjust the file paths that are saving to your test metadata XML file. Alternatively, if you are able to, try working out of the standard working directory we set for a container to see if that helps (you can do this by removing any instances of working_directory in your test run job).

Where does your `pytest.ini` live?

To ensure test splitting performs correctly, make sure you are running your tests in the root directory. If your tests are not being run in the root directory, you may need to run the following command before you test the run command:

cp -f .circleci/resources/pytest_build_config.ini pytest.ini

The .circleci/resources/pytest_build_config.ini path may need to be replaced to point to where it’s located in your project.

Are you setting the `junit_family` in your `pytest.ini`?

Check to see if you have something like junit_family=legacy set in your pytest.ini file. For more information on how to set junit_family, refer to the pytest docs. Search for "families" to see the relevant information.

A breaking change was introduced in pytest 6.1 the junit_family XML format changed to xunit2, which does not include filenames. This means that --split-by=timings will not work unless you specify xunit1. For more information see the pytest changelog. If this is your issue, you may see the following error:

Error autodetecting timing type, falling back to weighting by name. Autodetect no matching filename or classname.  If file names are used, double check paths for absolute vs relative.

Example project that correctly splits by timings

The example below is from the sample-python-cfd project and shows an implementation of test splitting:

version: 2.1
orbs:
  python: circleci/python@1.2
jobs:
  build-and-test:
    parallelism: 2
    docker:
      - image: cimg/python:3.8
    steps:
      - checkout
      - python/install-packages:
          pkg-manager: pip
      - run:
          name: Run tests
          command: |
            set -e
            TEST_FILES=$(circleci tests glob "openapi_server/**/test_*.py" | circleci tests split --split-by=timings)
            mkdir -p test-results
            pytest --verbose --junitxml=test-results/junit.xml $TEST_FILES
      - store_test_results:
          path: test-results
      - store_artifacts:
          path: test-results
workflows:
  sample:
    jobs:
      - build-and-test

Video: troubleshooting globbing

To follow along with the commands in the video below you will need to be SSH-ed into a job.

Other ways to split tests

Some third party applications and libraries might help you to split your test suite. These applications are not developed or supported by CircleCI. Please check with the owner if you have issues using it with CircleCI. If you’re unable to resolve the issue you can search and ask on our forum, Discuss.

Knapsack Pro - Enables allocating tests dynamically across parallel CI nodes, allowing your test suite execution to run faster. See CI build time graph examples.
phpunit-finder - This is a helper CLI tool that queries phpunit.xml files to get a list of test filenames and print them. This is useful if you want to split tests to run them in parallel based on timings on CI tools.
go list - Use the built-in Go command go list ./… to glob Golang packages. This allows splitting package tests across multiple containers.

go test -v $(go list ./... | circleci tests split)

Next steps

Suggest an edit to this page

Make a contribution

Learn how to contribute

Still need help?

Ask the CircleCI community

Join the research community

Visit our Support site