Building Postman test reports

In Testing an API with Postman, we set up automatic tests for API endpoints using Postman Collections and the Postman command-line utility, Newman. In this tutorial, we will build on that experience by using the Newman CLI tool to generate detailed, storable reports for our API tests.

Prerequisites

To follow this post, a few things are required:

1. Basic knowledge of JavaScript
2. Postman for Desktop installed on your system (you can download it here)
3. A CircleCI account
4. A GitHub account
5. Node.js installed on your system (version >= 10)
6. A collection and environment to report on

Note: If you completed the tasks in the Testing an API with Postman tutorial, you have a collection and environment you can use for reporting. If you are unfamiliar with Postman Collections, it may be helpful to complete that tutorial before starting this one.

Getting a Postman Collection public URL

Right now, the collection we are using to test APIs has to be updated each time we need to add more tests and requests. This manual step makes is cumbersome. You need to export the collection file and update the automated testing project. Fortunately, Postman runs a service that lets you upload and synch your collections.

Open the Postman desktop application and sign in. Your local collections are automatically uploaded and synched.

Once you have done this, you can get a public link to your collection. This link will point to the version of your collection hosted on the Postman service. Any changes to your collection will be immediately available from the link. As long as you’re logged in, Postman synchs all updates. That is the greatest benefit of using a link instead of exporting your collection as json.

To get your link, open your collection’s flyout menu and click Share. Copy your collection’s public link from the pop-up dialog, as shown in the following screenshot.

With this link, you no longer need to export and move collection files.

Creating the report project using Newman’s HTMLExtra Reporter

Next, we need to set up a project for running the API tests and for generating test reports. Newman supports several report generation packages. The one I have had the most success using is newman-reporter-htmlextra.

Although Newman and the reporter are usually installed and run globally, we will be doing something a little different for this project. We will install both packages locally so that we avoid global dependencies when we run the project.

Create a project folder and install both packages locally using the following commands:

mkdir postman-api-reports
cd postman-api-reports
npm init -y
npm install newman
npm install -S newman-reporter-htmlextra

Place your exported environment file at the root of the project. Now, the project has everything set up to generate reports for the API tests.

Generating reports locally

Next, we will run the command to test the API and generate the test reports. Use the following format, using information specific to your project:

npx newman run [YOUR_COLLECTION_PUBLIC_LINK] -e ./[POSTMAN_ENVIRONMENT_FILENAME].json -r htmlextra

This command invokes the local installation of newman with npx, then runs the tests in the collection. The results are passed to the htmlextra command from the newman-reporter-htmlextra package, which generates the reports.

You could run this command from the CLI to generate the reports, but it may not be practical to run a long command every time. Instead, you can add it as an npm script called report in the package.json file as shown in the following example:

"script" : {
    "report": "npx newman run https://www.getpostman.com/collections/778ce5ceb7c3ffbfd6bd -e ./My-Remote-API-Testing.postman_environment.json -r htmlextra"
}

Run the following command to run tests and generate reports:

npm run report

The first time you run this command, it creates a new folder, newman, at the root of the project, and adds a timestamped html file to the folder. Open the file in your browser to review the test report.

By default, the report displays a summary. The summary shows the number of test iterations, along with failed, passed, and skipped tests. Click any tab to get more detail about that category of results. You can even switch the display theme from light to dark mode.

Setting up a CircleCI project for reports automation

Fantastic – we have generated the test reports. Well done! Now it is time to automate the process. Begin by pushing your project to GitHub.

Add a .gitignore file, and instruct Git to ignore node_modules and the newman folder. In your .gitignore file, enter:

node_modules
newman

Make a commit to the project.

Now, go to the Add Projects page on the CircleCI dashboard to add the project.

Click Set Up Project.

Click Use Existing Config. This setting indicates that you are adding a configuration file manually, instead of using the sample.

Next, you are prompted to either download a configuration file or to start building.

Click Start Building. This build will fail because we have not set up our configuration file yet. We will set one up in the next step.

Automating report generation

It is now time to write the automation script. Create a folder named .circleci at the root of the project and add a configuration file named config.yml to it. In this file, enter the following code:

version: 2.1
jobs:
  build:
    working_directory: ~/repo
    docker:
      - image: circleci/node:10.16.3
    steps:
      - checkout
      - run:
          name: Update NPM
          command: "sudo npm install -g npm@5"
      - restore_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
      - run:
          name: Install Dependencies
          command: npm install
      - save_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
          paths:
            - ./node_modules
      - run:
          name: Generate Report
          command: npm run report
      - store_artifacts:
          path: ~/repo/newman

In the configuration above, the node.js Docker image is pulled in as the environment for running the tests. Then, npm is updated, and dependencies are installed and cached. With all dependencies in place, the custom report script is run using npm. This command runs the tests and generates the report. Finally, the reports folder (newman) is stored as an artifact. It can be found on the build’s Artifacts tab on the CircleCI console.

Push the new changes.

Click the build to display the summary.

Click Artifacts to find your generated reports.

Conclusion

In this tutorial, you have extended the end-to-end testing capabilities of Postman by making it generate accessible and detailed test reports. You can use these procedures to make sure that tests run, and that reports are generated whenever you push updates to your API.

Happy coding!