Rerun failed tests

Use the cypress-circleci-reporter package (note this is a 3rd party tool that is not maintained by CircleCI). You can install in your .circleci/config.yml or add to your package.json. Example for adding to .circleci/config.yml:

  #add required reporters (or add to package.json)
  -run:
    name: Install coverage reporter
    command: |
      npm install --save-dev cypress-circleci-reporter

Use the cypress-circleci-reporter, circleci tests run, and upload test results to CircleCI:

     -run:
        name: run tests
        command: |
          mkdir test_results
          cd ./cypress
          npm ci
          npm run start &
          circleci tests glob "cypress/**/*.cy.js" | circleci tests run --command="xargs npx cypress run --reporter cypress-circleci-reporter --spec" --verbose --split-by=timings #--split-by=timings is optional, only use if you are using CircleCI's test splitting

     - store_test_results:
        path: test_results

Ensure you are using xargs in your circleci tests run command to pass the list of test files/classnames via stdin to --command.

Update the glob command to match your specific use case. If your current job is using CircleCI’s intelligent test splitting, you must change the circleci tests split command to circleci tests run with the --split-by=timings parameter. If you are not using test splitting, --split-by=timings can be omitted.

Modify your test command to use circleci tests run:

- run:
    name: Run tests
    command: |
      circleci tests glob 'lib/**/*_test.exs'
      | circleci tests run --command='xargs -n1 echo > test_file_paths.txt'

      mix ecto.setup --quiet
      cat test_file_paths.txt | xargs mix test

- store_test_results:
    path: _build/test/my_app/test-junit-report.xml
    when: always

Ensure you are using xargs in your circleci tests run command to pass the list of test files/classnames via stdin to --command.

Update the glob command to match your use case.

Modify your test command to use circleci tests run:

- run:
    command: go list ./... | circleci tests run --command "xargs gotestsum --junitfile junit.xml --format testname --" --split-by=timings --timings-type=name

- store_test_results:
    path: junit.xml

Ensure you are using xargs in your circleci tests run command to pass the list of test files/classnames via stdin to --command.

If your current job is using CircleCI’s intelligent test splitting, you must change the circleci tests split command to circleci tests run with the --split-by=timings parameter.. If you are not using test splitting, --split-by=timings can be omitted.

Install the jest-junit dependency. You can add this step in your .circleci/config.yml:

  - run:
      name: Install JUnit coverage reporter
      command: yarn add --dev jest-junit

Modify your test command to look something similar to:

- run:
    command: |
      npx jest --listTests | circleci tests run --command="JEST_JUNIT_ADD_FILE_ATTRIBUTE=true xargs npx jest --config jest.config.js --runInBand --" --verbose --split-by=timings
    environment:
      JEST_JUNIT_OUTPUT_DIR: ./reports/
- store_test_results:
    path: ./reports/

Ensure you are using xargs in your circleci tests run command to pass the list of test files/classnames via stdin to --command.

Update the npx jest --listTests command to match your use case. See the Jest section in the Collect Test Data document for details on how to output test results in an acceptable format for jest. If your current job is using CircleCI’s intelligent test splitting, you must change the circleci tests split command to circleci tests run with the --split-by=timings parameter. If you are not using test splitting, --split-by=timings can be omitted.

Modify your test command to use circleci tests run:

-run:
  command: |
    cd src/test/java

    # Get list of classnames of tests that should run on this node.
    circleci tests glob "**/*.java" | cut -c 1- | sed 's@/@.@g' | sed 's/.\{5\}$//' | circleci tests run --command=">classnames.txt xargs echo" --verbose --split-by=timings --timings-type=classname

    #if this is a re-run and it is a parallel run that does not have tests to run, halt execution of this parallel run
    [ -s classnames.txt ] || circleci-agent step halt

-run:
  command: |

    # Format the arguments to "./gradlew test"

    GRADLE_ARGS=$(cat src/test/java/classnames.txt | awk '{for (i=1; i<=NF; i++) print "--tests",$i}')
    echo "Prepared arguments for Gradle: $GRADLE_ARGS"

    ./gradlew test $GRADLE_ARGS

- store_test_results:
    path: build/test-results/test

Update the glob command to match your use case. If your current job is using CircleCI’s intelligent test splitting, you must change the circleci tests split command to circleci tests run with the --split-by=timings parameter.. If you are not using test splitting, --split-by=timings can be omitted.

Edit your test command to use circleci tests run:

# Use phpunit-finder to output list of tests to stdout for a test suite named functional
# Pass those tests as stdin to circleci tests run

- run:
    name: Run functional tests
    command: |
      TESTS_TO_RUN=$(/data/vendor/bin/phpunit-finder -- functional)
      echo "$TESTS_TO_RUN" | circleci tests run --command="xargs -I{} -d\" \" /data/vendor/bin/phpunit {} --log-junit /data/artifacts/phpunit/phpunit-functional-$(basename {}).xml" --verbose --split-by=timings

- store_test_results:
    path: artifacts/phpunit
    when: always

Ensure you are using xargs in your circleci tests run command to pass the list of test files/classnames via stdin to --command.

Note that this example uses a utility named phpunit-finder which is a third party tool that is not supported by CircleCI, use at your own risk. If your current job is using CircleCI’s intelligent test splitting, you must change the circleci tests split command to circleci tests run with the --split-by=timings parameter.**. If you are not using test splitting, --split-by=timings can be omitted.

Modify your test command to use circleci tests run:

 - run:
    command: |
      mkdir test-results #can also be switched out for passing PLAYWRIGHT_JUNIT_OUTPUT_NAME directly to Playwright
      pnpm run serve &
      TESTFILES=$(circleci tests glob "specs/e2e/**/*.spec.ts")
      echo "$TESTFILES" | circleci tests run --command="xargs pnpm playwright test --config=playwright.config.ci.ts --reporter=junit" --verbose --split-by=timings

 - store_test_results:
    path: results.xml

Ensure you are using xargs in your circleci tests run command to pass the list of test files/classnames via stdin to --command.

Update the glob command to match your use case. If your current job is using CircleCI’s intelligent test splitting, you must change the circleci tests split command to circleci tests run with the --split-by=timings parameter.. If you are not using test splitting, --split-by=timings can be omitted. Note: you may also use Playwright’s built-in flag (PLAYWRIGHT_JUNIT_OUTPUT_NAME) to specify the JUnit XML output directory.

Add the following gem to your Gemfile:

gem 'rspec_junit_formatter'

Modify your test command to use circleci tests run:

 - run: mkdir ~/rspec
 - run:
    command: |
      circleci tests glob "spec/**/*_spec.rb" | circleci tests run --command="xargs bundle exec rspec --format progress --format RspecJunitFormatter -o ~/rspec/rspec.xml" --verbose --split-by=timings

 - store_test_results:
    path: ~/rspec

--format RspecJunitFormater must come after any other --format RSpec argument

Ensure you are using xargs in your circleci tests run command to pass the list of test files/classnames via stdin to --command.

Update the glob command to match your use case. See the RSpec section in the Collect Test Data document for details on how to output test results in an acceptable format for rspec. If your current job is using CircleCI’s intelligent test splitting, you must change the circleci tests split command to circleci tests run with the --split-by=timings parameter. If you are not using test splitting, --split-by=timings can be omitted.

Install Knapsack Pro in your project.

Add the following gem to your Gemfile:

gem 'rspec_junit_formatter'

Modify your test command to use circleci tests:

 - run:
    name: RSpec with Knapsack Pro
    command: |
      mkdir -p /tmp/test-results

      export KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true

      export KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE=/tmp/tests_to_run.txt
      # Retrieve the tests to run (all or just the failed ones), and let Knapsack Pro split them optimally.
      circleci tests glob "spec/**/*_spec.rb" | circleci tests run --index 0 --total 1 --command ">$KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE xargs -n1 echo" --verbose
      bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out /tmp/test-results/rspec.xml]"

 - store_test_results:
     path: /tmp/test-results

 - store_artifacts:
     path: /tmp/test-results
     destination: test-results

You may also enable Split by Test Examples to parallelize tests across CI nodes by individual it/specify. This is useful when you have slow test files but do not want to manually split test examples into smaller test files.

Modify your test command to look something similar to:

- run: mkdir -p ~/cucumber
- run:
    command: |
    circleci tests glob "features/**/*.feature" | circleci tests run --command="xargs bundle exec cucumber --format junit,fileattribute=true --out ~/cucumber/junit.xml" --verbose --split-by=timings

- store_test_results:
    ~/cucumber

Ensure you are using xargs in your circleci tests run command to pass the list of test files/classnames via stdin to --command.

Update the glob command to match your use case. See the Cucumber section in the Collect Test Data document for details on how to output test results in an acceptable format for Cucumber. If your current job is using CircleCI’s intelligent test splitting, you must change the circleci tests split command to circleci tests run with the --split-by=timings parameter. If you are not using test splitting, --split-by=timings can be omitted.

Install Knapsack Pro in your project.

Add the following gem to your Gemfile:

gem 'minitest-ci'

Modify your test command to use circleci tests:

 - run:
     name: Minitest with Knapsack Pro
     command: |
       export KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE=/tmp/tests_to_run.txt

       # Retrieve the tests to run (all or just the failed ones), and let Knapsack Pro split them optimally.
       circleci tests glob "test/**/*_test.rb" | circleci tests run --index 0 --total 1 --command ">$KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE xargs -n1 echo" --verbose
       bundle exec rake "knapsack_pro:queue:minitest[--verbose --ci-report --no-ci-clean]"

 - store_test_results:
     path: test/reports

Ensure that the --verbose setting is enabled when invoking circleci tests run. This will display which tests circleci tests run is receiving on a "rerun".

Use store_artifacts to upload the JUnit XML that contains test results to CircleCI. This is the same file(s) that is being uploaded to CircleCI with store_test_results.

Manually inspect the newly uploaded JUnit XML via the Artifacts tab and ensure that there is a file= attribute or a classname attribute. If neither are present, you will see unexpected behavior when trying to rerun. Follow the instructions on this page to ensure that the test runner you are using is outputting its JUnit XML test results with a file (preferred) or classname attribute. Comment in our community forum if you are still stuck.

Ensure that xargs is present in the --command= argument.