Manage automated test data with the PractiTest orb

The software testing data provided by CI/CD tools is valuable, but it is not always comprehensive enough to give managers the insights they need to make improvements. To make effective business decisions, managers need visibility into the entire testing process, in a way that will help them understand what needs to be done and how.

Some visualization tools provide the option of editing your automation scripts and adding REST API calls to report runs and results. These tools can later visualize your testing data and results in a more comprehensive way. But adding these API calls can require a lot of work that might not fit to your organizational workflow and priorities. Plus, this doesn’t answer the need for a holistic view of your overall QA efforts.

In this tutorial, you will learn how to achieve greater visibility into your entire testing operation, including your automated tests, by exporting test data from your CI/CD pipeline to PractiTest.

Getting valuable test data directly to PractiTest

PractiTest is an end-to-end test management tool with the ability to show you everything you want to know about your entire testing operation, including:

• Execution progress run graphs
• Test run time reports
• Run-based reports
• And more

For CI/CD users, PractiTest created the Firecracker tool. The Firecracker automatically converts XML test results into tests in PractiTest, letting you organize the data dynamically and create customizable dashboard graphs and reports based on your test data.

With the PractiTest Firecracker orb, CircleCI users can easily integrate PractiTest with their pipelines. By adding the orb to your CircleCI config, all the run results will be populated in PractiTest and you will get a full dynamic view of your entire testing process. This will not require any code change to your automation scripts.

Prerequisites

To follow this tutorial, you will need:

• A CircleCI account
• A GitHub account
• A PractiTest account
• The latest Firecracker JAR version number

Getting started

In PractiTest, create an API Token. We recommend using a personal API token that can be enabled by the account owner and can be found on the personal settings page.

In your CircleCI settings, create a context and call it PT API Token (make sure to use the names exactly as they appear here). Add an environment variable named FIRE_EMAIL and put your email as the value (this will be the tester email that will appear later in the PractiTest run). Then add another environment variable named FIRE_API_TOKEN and put the PractiTest API token you retrieved earlier as the value.

Adding PractiTest to your continuous integration workflow

In this example, we will use a very basic test project in GitHub, which we will add to a CircleCI pipeline. The Firecracker orb will report all the runs and results to PractiTest, where you can create advanced, customizable displays of the test results.

To get started, fork the example GitHub project. Next, log in to your CircleCI account. On the Projects page, locate the new forked PractiTest project called circleci.test and press Set Up Project. Choose the existing config provided in the project repo. If you receive an error message stating that you need to enable the use of third-party orbs for your account, you can do so in the Organization Security settings.

Next, choose the master branch from the dropdown list and press Edit Config on the top right side.

![Edit config]2022-92-07-image2.jpg)

If you receive an error message because the parameters in the orb are not defined yet, you can ignore it. In the config file, you will find the parameters you can use to modify your PractiTest integration.

Let’s start with the required parameters:

• firecracker_version: Ensure you are using the latest Firecracker JAR version number.
• author_id: You can find the PractiTest user ID of the person running the tests in the URL under Personal Settings > click here to change your personal information. Copy the four numbers in the URL and paste them as the author_id.
• api_token and email: These will be defined automatically by the context you created in the prerequisites section.
• project_id: To find the ID of the project you want the results to be reported to, go to the Requirements, Test Library, Test Sets & Runs, or Issues module in PractiTest. The four numbers after the /p/ are the project ID.
• api_uri: If your PractiTest account is in an EU data center, please add https://eu1-prod-api.practitest.app/ to this parameter.

The other parameters, such as testset_name, will define the way the results appear in PractiTest, but for now we will keep them as they are.

Press Save and Run. CircleCI will automatically build the example project and update PractiTest with the test results. To find the results in PractiTest, go to the Test Sets & Runs module. You will see a new test set with the name Circleci_Testset, as defined in the config.

![Firecracker test set added]2022-92-07-image3.jpg)

Inside the test set, you will see the tests reported from this build and the results for each test and step.

Hurray! You have completed the basic steps to run the PractiTest Firecracker orb. In PractiTest, you can display your test data in various ways using dashboards and reports. You will see a few examples of this later in the tutorial. First, let’s see what happens when a test fails in CircleCI.

Recording tests results in PractiTest

Once you have added the Firecracker orb to your pipeline, every change to the project code will automatically trigger CircleCI to create a new build and update PractiTest with the results.

As an example, go to the project in GitHub, open the test folder, and edit the CLJ file called test_report to fail by changing the end is statement to {:pass 1, :fail 1, :error 3}. Then commit your changes.

If you go back to CircleCI, you will see it running, but this time the build will fail because of the failed test. In PractiTest, you will see that the same test set is now on status failed.

![Failed test results]2022-92-07-image6.jpg)

You can open the test set to see the exact instance that failed.

If you open the instance you will see two runs: the first one that passed and the second one that failed. By clicking the failed run, you can see more information about the results of this specific run and why it failed.

Now that you are familiar with the basic flow and configuration, you are ready to move on to more advanced settings.

Setting advanced parameters for detailed results

With the Firecracker orb, you can define extra parameters to receive more insightful results that will let you create extensive dashboard graphs and reports.

For some of the advanced parameters, we recommend you create target custom fields in PractiTest to match the test results that will be received from CircleCI. You can use these fields for the additional parameters explained below.

Below are a few of the extra parameters you can define in the config file. These can also be found in the CircleCI documentation.

If you don’t want to run CircleCI as a machine user, you can run Firecracker in a Docker container. Do that by inserting the parameter below as a step before the step that runs the tests (this way if a test fails, CircleCI will still execute Firecracker and upload the results to PractiTest):

setup_remote_docker:
  version: 19.03.13

If you want to map more test set fields from your build to PractiTest, use the additional_testset_fields parameter. For example, if you created a test set field in PractiTest called Branch and you want to add the branch value to it, add the rows as follows:

additional_testset_fields: |
  \"custom-fields\":{\"---f-field_id\":\"${CIRCLE_BRANCH}\"},\"system-fields\":{}

Replace field_id above with the field ID available in Project Settings> Fields> Branch field. The field ID is the five numbers at the end of the URL. Once this is done, the branch value from CircleCI will be automatically populated in the Branch field in PractiTest for every build.

If you want to map more test fields from your build to PractiTest, use the additional_test_fields parameter. For example, you can use a field from the XML test results (available in the artifacts of previous builds in CircleCI) called full-class-name and map it to the Preconditions test system field in PractiTest:

additional_test_fields: |
  \"custom-fields\":{},\"system-fields\":{\"preconditions\":\"?full-class-name\"}

This way, the values in the full-class-name field will be automatically populated in the Preconditions field for every build.

If you want to add more inline parameters, you can use the extra_params parameter. For example, if you want to add the command line display action log to your build, you can add extra_params: --display-action-logs.

The different options for the inline parameters are available in the Firecracker help when running Firecracker independently in the command line. For more information, please contact PractiTest support.

By default, Firecracker will create one test set for all the test results in PractiTest. If you wish to create multiple test sets based on the Suite-Name attribute, use multitestset: true and delete the testset_name parameter in the original config file.

By default, all XML test cases in the build are reported to PractiTest as one multistep test. If you want each XML test case to be reported as a single step test, add the test_case_as_pt_test_step: false parameter.

If you didn’t use the multitestset parameter, or if you set it to false, you can define the name of the single test set receiving the results using testset_name: CallMeAnything.

Visualizing your test data

In PractiTest, you can slice and dice your data to create insightful dashboards and reports. For example, you can create an instance tubular aggregated report based on any of your PractiTest filters and test fields.

You can also create a basic pie chart for your instances to present any of your test fields, such as the test area, branch, model, or phase.

If you want to track the execution progress, you can create an instance execution progress graph and filter the results to show exactly what you want to track.

Conclusion

Integrating PractiTest with your CircleCI is another step to centralize all your testing efforts without the overhead of using an API. PractiTest lets you integrate all the tools you are using to make your workflow seamless and efficient.

Having one place where you can organize your data dynamically and visualize it gives you the ability to better manage your testing and make wiser business decisions based on all your relevant information. Visit the PractiTest help center to learn more about how to use Firecracker and how to integrate the Firecracker tool with CircleCI. You can also find more information on the Firecracker orb in the CircleCI developer hub.