Troubleshoot test splitting
On This Page
Using test splitting with Python Django tests
To use test splitting with CircleCI, you must pass in a list of tests to run. However, with how you execute tests with Django, you are unable to simply glob the tests and pass them in.
Sometimes, users run into specific issues when performing test splitting for their own unique use cases. One example of a user resolving an issue when test splitting in Python Django did not perform correctly can be found in the following Discuss post.
Using this example, here is a quick example of how you can accomplish test splitting:
- run: command: | # get test files while ignoring __init__ files TESTFILES=$(circleci tests glob "catalog/tests/*.py" | sed 's/\S\+__init__.py//g') echo $TESTFILES | tr ' ' '\n' | sort | uniq > circleci_test_files.txt cat circleci_test_files.txt TESTFILES=$(circleci tests split --split-by=timings circleci_test_files.txt) # massage filepaths into format manage.py test accepts TESTFILES=$(echo $TESTFILES | tr "/" "." | sed 's/\.py$//g') echo $TESTFILES pipenv run python manage.py test --verbosity=2 $TESTFILES
Using test splitting with
If you try to split your tests across containers with
pytest, you may encounter any of the following errors:
No timing found for "tests/commands/__init__.py" No timing found for "tests/commands/test_1.py" No timing found for "tests/commands/test_2.py"
If any of these errors are returned, you may need to make a few adjustments, which are listed below.
Are you setting a custom
If so, you may need to adjust the file paths that are saving to your test metadata XML file. Alternatively, if you are able to, try working out of the standard working directory we set for a container to see if that helps (you can do this by removing any instances of
working_directory in your test run job).
Where does your
To ensure test splitting performs correctly, make sure you are running your tests in the root directory. If your tests are not being run in the root directory, you may need to run the following command before you test the
cp -f .circleci/resources/pytest_build_config.ini pytest.ini
.circleci/resources/pytest_build_config.ini path may need to be replaced to point to where it’s located in your project.
Are you setting the
junit_family in your
Check to see if you have something like
junit_family=legacy set in your
pytest.ini file. For more information on how to set
junit_family, refer to the
pytest docs. Search for "families" to see the relevant information.
A breaking change was introduced in
pytest 6.1 the
junit_family XML format changed
to xunit2, which does not include filenames. This means that
--split-by=timings will not work unless you specify
xunit1. For more information see the pytest changelog. If this is your issue, you may see the following error:
Error autodetecting timing type, falling back to weighting by name. Autodetect no matching filename or classname. If file names are used, double check paths for absolute vs relative.
Example project that correctly splits by timings
The example below is from the
sample-python-cfd project and shows an implementation of test splitting:
version: 2.1 orbs: python: email@example.com jobs: build-and-test: parallelism: 2 docker: - image: cimg/python:3.8 steps: - checkout - python/install-packages: pkg-manager: pip - run: name: Run tests command: | set -e TEST_FILES=$(circleci tests glob "openapi_server/**/test_*.py" | circleci tests split --split-by=timings) mkdir -p test-results pytest --verbose --junitxml=test-results/junit.xml $TEST_FILES - store_test_results: path: test-results - store_artifacts: path: test-results workflows: sample: jobs: - build-and-test
Video: troubleshooting globbing
Other ways to split tests
Some third party applications and libraries might help you to split your test suite. These applications are not developed or supported by CircleCI. Please check with the owner if you have issues using it with CircleCI. If you’re unable to resolve the issue you can search and ask on our forum, Discuss.
phpunit-finder - This is a helper CLI tool that queries
phpunit.xmlfiles to get a list of test filenames and print them. This is useful if you want to split tests to run them in parallel based on timings on CI tools.
go list - Use the built-in Go command
go list ./…to glob Golang packages. This allows splitting package tests across multiple containers.
go test -v $(go list ./... | circleci tests split)
Help make this document better
This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.
- Suggest an edit to this page (please read the contributing guide first).
- To report a problem in the documentation, or to submit feedback and comments, please open an issue on GitHub.
- CircleCI is always seeking ways to improve your experience with our platform. If you would like to share feedback, please join our research community.
Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.
You can also visit our support site to find support articles, community forums, and training resources.
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.