Performance testing measures how well systems perform when subjected to various workloads. The key qualities being tested are stability and responsiveness. Performance testing shows the robustness and reliability of systems in general, along with the specific potential breaking points. k6 is an open source framework built to make performance testing fun for developers.

In this tutorial, you will use k6 to do load testing on a simple API hosted on the Heroku platform. Then you will learn how to interpret the results obtained from the tests. This tutorial is a companion to HTTP request testing with k6.

Prerequisites

To follow this tutorial, you will need:

Our tutorials are platform-agnostic, but use CircleCI as an example. If you don’t have a CircleCI account, sign up for a free one here.

As a performance testing framework, k6 stands out because of the way it encapsulates usability and performance. Tools like a command-line interface to execute scripts and a dashboard to monitor the test run results add to its appeal.

k6 Logo

k6 is written in the goja programming language, which is an implementation of ES2015(ES6) JavaScript on pure Golang language. That means you can use JavaScript to write k6 scripts, although language syntax will only be compatible with the JavaScript ES2015 syntax.

Note: k6 does not run on a Node.js engine. It uses a Go JavaScript compiler.

Now that you know what runs under the k6 hood, why not take on setting up k6 to run your performance tests?

Cloning the repository

First, clone the sample application from the GitHub repository.

git clone https://github.com/CIRCLECI-GWP/api-performance-testing-with-k6.git

The next step is to install k6 on your machine.

Installing k6

Unlike JavaScript modules, k6 must be installed using a package manager. In macOS you can use Homebrew. In Windows OS, you can use Chocolatey. There are more installation options available for other operating systems in the k6 documentation. For this tutorial we will follow the macOS installation guide. Execute this command in Homebrew:

brew install k6

This is the only command we need to run to get started with writing k6 tests.

Writing your first performance test

Great work! You have installed k6 and know how it works. Next up is understanding what k6 measures and what metrics you can use to write your first load test. Load testing models the expected usage of a system by simulating multiple users accessing the system at the same time. In our case, we will be simulating multiple users accessing our API within specified time periods.

Load testing our application

The load tests in the examples measure the responsiveness and stability of our API system when subject to various groups of users. The example tests determine the breaking points of the system, and acceptable limits.

In this tutorial, we will be running scripts against a k6 test file that first creates a todo in an already hosted API application on Heroku. For this to be considered a load test, we must include factors like scalable virtual users, scalable requests, or variable time. In our case, we will focus on scaling users to the application as the tests run. This feature comes pre-built with the k6 framework, which makes it super easy for our implementation.

To simulate this, we will use the k6 concept of executors. Executors supply the horsepower in the k6 execution engine and are responsible for how the scripts execute. The runner may determine these:

  • Number of users to be added
  • Number of requests to be made
  • Duration of the test
  • Traffic received by the test application

In our first test, we will use a k6 executor with an approach that k6 refers to as ramping up. In the ramping up approach, k6 incrementally adds virtual users (VUS) to run our script until a peak is reached. It then gradually decreases the number within the stipulated time, until the execution time ends.

Write this load test to incorporate the idea of executors, virtual users, and the test itself. Here is an example:

import http from 'k6/http';
import { check, group } from 'k6';

export let options = {
   stages: [
       { duration: '0.5m', target: 3 }, // simulate ramp-up of traffic from 1 to 3 virtual users over 0.5 minutes.
       { duration: '0.5m', target: 4}, // stay at 4 virtual users for 0.5 minutes
       { duration: '0.5m', target: 0 }, // ramp-down to 0 users
     ],
};

export default function () {
   group('API uptime check', () => {
       const response = http.get('https://aqueous-brook-60480.herokuapp.com/todos/');
       check(response, {
           "status code should be 200": res => res.status === 200,
       });
   });

   let todoID;
   group('Create a Todo', () => {
       const response = http.post('https://aqueous-brook-60480.herokuapp.com/todos/',
       {"task": "write k6 tests"}
       );
       todoID = response.json()._id;
       check(response, {
           "status code should be 200": res => res.status === 200,
       });
       check(response, {
           "response should have created todo": res => res.json().completed === false,
       });
   })
});

In this script, the options object gradually increases the number of users within a given time in the defined stages. Though the executor has not been defined by default, k6 identifies the stages, durations, and targets in the options object, and determines that the executor is ramping-vus.

In the load test script, our target is a maximum of 4 concurrent users within a period of 1m 30 seconds. The script will therefore increase application users gradually and carry out as many create-todo request iterations as possible within that time frame.

In other words, we will attempt to make as many requests as possible within the specified time frame while varying the number of active virtual users with the provided stages. The success of this test will depend on the number of successful requests made by the application inside the time period. Here is a graph of virtual users of our test against time:

Performance testing vus

This ramp-up graph illustration shows that as time increases, virtual users are incrementally added to the load test until the peak number of users is reached. That number is reached in the final stage as the virtual users are completing their requests and exiting the system. For this tutorial, the peak number of users is 4. The ramping-vus executor type is not the only one available to run load tests. Other executors are available in k6. Usage depends on the need and nature of the performance test you want to execute.

The example performance test script shows that we can use k6 to define our test, the number of virtual users that our machine will run, and also create different assertions with check() under our group() blocks for the execution script.

Note: Groups in k6 let you combine a large load script to analyze results of the test. Checks are assertions within groups, but work differently from other types of assertions as they do not halt the execution of the load tests when they fail or pass.

Running performance tests using k6

Now you are ready get start executing tests that will help you understand the performance capabilities of your systems and applications. We will test the limits of our Heroku free dyno and the hosted Todo API application deployed on it. While this is not an ideal server for running load tests, it lets us identify when and under what circumstances the system reaches its breaking point. For this tutorial, the breaking point could appear in Heroku or our API application. This test will make a multiple HTTP requests with virtual users (VUS) to Heroku. Starting with one virtual user and ramping up to 4, the virtual users will create todo items for a duration of 1 minute and 30 seconds.

Note: We are making the assumption that Heroku is able to handle 4 concurrent sessions of users interacting with our API.

To run the performance test from the previous code snippet, just run this command on our terminal:

k6 run create-todo-http-request.js

When this test has finished, you will have your first results from a k6 test run.

Create todo performance test results

Use the results to verify that the performance test executed for 1 minute and 30.3 seconds and that all the iterations of the test passed. You can also verify that in the period that the test was executed, 4 virtual users made a total of 390 iterations (completed the process of creating todo items).

Other metrics you can obtain from the data include:

  • checks, the number of completed check() assertions declared on the test (all of them passed)
  • Total http_reqs, the number of all the HTTP requests sent out (780 requests)
  • Combination of the total number of iterations of creating new todo items and HTTP requests for checking the uptime state of our Heroku server

This test was successful, and we were able to obtain meaningful information about the application. However, we do not yet know if our system has any breaking points or if our application could start behaving strangely after a certain number of concurrent users starts using our API. This is what we will try to accomplish in the next section.

Adding another HTTP request will help us find out if we can break the Heroku server or even our application. We will also reduce the number of virtual users and the duration of the test to save on resources. These adjustments can be made in the configuration section by simply commenting out the last two stages of the execution.

Note: “Breaking the application” means overwhelming the system to the point that it returns errors instead of a successful execution, without changing resources or application code. That means the system returns errors based only on changing the number of users or the number of requests.

stages: [
       { duration: '0.5m', target: 3 }, // simulate ramp-up of traffic from 1 to 3 users over 0.5 minutes.
       // { duration: '0.5m', target: 4 }, -> Comment this portion to prevent execution
       // { duration: '0.5m', target: 0 }, -> Comment this portion to prevent execution

With this configuration, we have been able to shorten the duration of execution and reduce the number of virtual users used to execute our test. We will execute the same performance test we used previously but add an HTTP request to:

  • Fetch the created Todo
  • Verify that its todoID is consistent with what was created
  • Verify that its state of creation is completed: false

To execute this performance test, you will need the create-and-fetch-todo-http-request.js file, which is already in the root of the project directory in our cloned repository. Go to the terminal and execute this command:

k6 run create-and-fetch-todo-http-request.js

Once this command has completed execution, you will get the test results.

Create and fetch todo performance test results

These results show that our test actually broke the limits of the free Heroku dynos for our application. Although there were successful requests, not all of them returned responses in a timely manner. That then led to the failure of some requests. Using only 3 constant virtual users and executing tests for 30 seconds, there were 56 complete iterations. Out of those 56, 95.97% of checks were successful.

The failed checks were related to the response of fetching the todoID of our created Todo items. This could be the first indicator of a bottleneck in our API or our system. Finally, we can verify that our http_reqs total 168, which is the total of the requests from the creation of the todo items, fetching of the todo items data, and verifying the uptime of the Heroku server:

56 requests for each iteration * 3 - each of every request item

From these test runs, it is clear that performance tests metrics are not standard across the board for all systems and applications. These numbers may vary depending on the machine configuration, nature of the application, and even dependent systems for the applications under test. The goal is to ensure that bottlenecks are identified and fixed. You need to understand the type of load your system can handle so that you can plan for the future, including usage spikes and above-average demand for resources.

In the previous step, we were able to execute our performance tests for the application hosted on a Heroku free plan and analyze the command line results. With k6, you can also analyze the results of performance tests using the K6 cloud dashboard. The dashboard is a web-based interface that allows us to view the results of our performance tests.

k6 cloud configuration and output

Running k6 performance tests and getting the output on your terminal is great, but it is also cool to share the results and output with the rest of the team. In addition to the many cool analysis features, sharing data is where the k6 dashboard really shines.

To configure the output for the k6 dashboard, log into the k6 cloud with the account details you created earlier. Next, locate your access token.

k6 could API token

Once you are on the API token page, copy the access token so you can run the tests and upload them to the cloud.

Next, add K6_CLOUD_TOKEN as an environment variable. Run this command in your terminal to set the k6 cloud token:

export K6_CLOUD_TOKEN=<k6-cloud-token>

Note: Replace the cloud token value in the example with the k6 token you just copied from the dashboard.

Now you can execute your load tests. Use this command:

k6 run --out cloud create-and-fetch-todo-http-request.js

--out cloud tells k6 to output results to the cloud. k6 automatically creates dashboards with test results when the execution starts. You can use the dashboard to evaluate the results of your performance tests.

The dashboard makes interpretation of results more convenient and easier to share than terminal output. The first metrics from your uploaded run are the total number of request and how requests were made by different virtual users. There is also information about when different users were making requests.

k6 performance overview graph

This graph shows the total number of virtual users against the number of requests that were made, and the average response time of every request. It gets more interesting as you drill down on every request. The k6 dashboard makes this possible using groups() and checks().

Try this: select the Checks tab, then filter the results using the tree view.

k6 performance insight checks

All the requests from the test run are listed in chronological order and also assess the failures and success execution rates with the average time it took for all the requests and also the total number of requests executed.

To evaluate the individual requests and their times, select the HTTP tab in the same performance insight page. You can review all the requests made, the time they took to execute, and how the execution compares with other requests. For example, one request falls into the 95th percentile, while another one was in the 99th. The dashboard shows the maximum amount of time it took to execute the longest request with a standard deviation figure for reference purposes. The page provides you almost any kind of data point that you and your team can use to identify bottlenecks.

k6 performance insight http requests

You can also use the k6 dashboard to select a specific request and identify its performance against average run times. This would be useful in identifying issues like database bottlenecks caused by a large number of locked resources when multiple users are accessing the resources.

There is so much you and your team can benefit from using the power of the k6 dashboard.

Your next steps are triggering k6 performance tests with CircleCI, and configuring them to run on every deploy.

Setting up Git and pushing to CircleCI

Note: If you have already cloned the project repository, you can skip this part of the tutorial. I have added the steps here if you want to learn how to set up your own project.

To set up CircleCI, initialize a Git repository in the project by running the following command:

git init

Next, create a .gitignore file in the root directory. Inside the file add node_modules to keep npm-generated modules from being added to your remote repository. Then, add a commit and push your project to GitHub.

Log in to CircleCI and go to the Projects dashboard. You can choose the repository that you want to set up from a list of all the GitHub repositories associated with your GitHub username or your organization. The project for this tutorial is named api-performance-testing-with-k6. On the Projects dashboard, select the option to set up the project you want. Choose the option to use an existing configuration.

Note: After initiating the build, expect your pipeline to fail. You still need to add the customized .circleci/config.yml configuration file to GitHub for the project to build properly.

Setting up CircleCI

Create a .circleci directory in your root directory and then add a config.yml file. The config file holds the CircleCI configuration for every project. Execute your k6 tests using the CircleCI k6 orb in this configuration:

version: 2.1
orbs:
 k6io: k6io/test@1.1.0
workflows:
 load_test:
   jobs:
     - k6io/test:
         script: create-todo-http-request.js

Using third-party orbs

CircleCI orbs are reusable packages of YAML configurations that condense code into a single line. To allow the use of third-party orbs like python@1.2, you may need to:

  • Enable organization settings if you are the administrator, or
  • Request permission from your organization’s CircleCI admin.

Now, on further investigation, you can verify that running your performance tests in k6 was successful, and they have been integrated into CircleCI. Great work!

k6 performance tests execution on CircleCI

Now it is time to add metrics to measure the execution time of each endpoint.

Evaluating API request time

From the previous performance tests, it is clear that the test structure is less important than how the system reacts to the testing loads at any particular time. k6 comes with a metrics feature called Trend that lets you customize metrics for your console and cloud outputs. You can use Trend to find out the specific time of every request made to an endpoint by customizing how the timings will be defined. Here is an example code block:

import { Trend } from 'k6/metrics';

const uptimeTrendCheck = new Trend('/GET API uptime');
const todoCreationTrend = new Trend('/POST Create a todo');

export let options = {
   stages: [
       { duration: '0.5m', target: 3 }, // simulate ramp-up of traffic from 0 to 3Vus
   ],
};

export default function () {
   group('API uptime check', () => {
       const response = http.get('https://aqueous-brook-60480.herokuapp.com/todos/');
       uptimeTrendCheck.add(response.timings.duration);
       check(response, {
           "status code should be 200": res => res.status === 200,
       });
   });

To implement k6 Trend, import it from k6/metrics, then define each trend you want. For this part of the tutorial, you only need to check the response time for the API when doing an uptime check, or when creating a new todo. Once the trends are created, navigate to the specific tests and embed the data you need to the declared trends.

Execute your todo creation performance test file using the command:

k6 run create-todo-http-request.js

Check the console response. You can find this test file in the root directory of the cloned repository. Once this passes locally, commit and push the changes to GitHub.

k6 performance results with trend

Once your tests finish executing, you can review the descriptions of the trends that you added in the previous step. Each has the timings for the individual requests, including the average, maximum, and minimum execution times and their execution percentiles.

Conclusion

This tutorial introduced you to what k6 is and how to use it to run performance tests. You went through the process of creating a simple k6 test for creating a todo list item to be run against a Heroku server. I also showed you how to interpret k6 performance tests results in the command line terminal, and using the cloud dashboard. Lastly, we explored using custom metrics and k6 Trends. I encourage you to share this tutorial with your team, and continue to expand on what you have learned.

I hope you enjoyed this tutorial as much as I did preparing it!

__


Waweru Mwaura is a software engineer and a life-long learner who specializes in quality engineering. He is an author at Packt and enjoys reading about engineering, finance, and technology. You can read more about him on his web profile.

Read more posts by Waweru Mwaura