Smoke testing in CI/CD pipelines

Imagine you run a web application through your CI/CD pipeline and all of the tests pass. Based on those results, you confidently deploy it to a live target environment. Despite the successful tests, the application just does not function as expected. This is a common situation that plagues many development teams.

You can’t always predict what will happen when your application is pushed live. Many teams look to smoke tests to ensure that changes can be trusted. Smoke tests are designed to reveal these types of failures early by running test cases that cover the critical components and functionality of the application. They also ensure that the application will function as expected in a deployed scenario.

When implemented, smoke tests are often executed on every application build to verify that basic but critical functionality passes before jumping into more extensive and time-consuming testing. Smoke tests help create the fast feedback loops that are vital to the software development life cycle.

In this post, we’ll demonstrate how to add smoke testing to the deployment stage of a CI/CD pipeline. The smoke testing in our example will test simple aspects of the application post deployment.

What are smoke tests?

Smoke tests are a set of basic tests that verify the most critical functions of your application are working after deployment. They act as a first-pass check to catch major issues before running more comprehensive test suites. The term comes from hardware testing, where powering on a device and checking for smoke was a quick way to see if something was fundamentally broken.

Why use smoke tests?

Early detection: Quickly catch deployment or configuration issues that unit/integration tests might miss. Fast feedback: Provide rapid confirmation that the application is running and accessible in its target environment. Confidence in deployments: Ensure that the most essential user flows or endpoints are working before promoting a build to production. Cost and time savings: Prevent wasted resources on further testing or manual QA if the deployment is fundamentally broken.

When should you use smoke tests?

After deployment: Immediately after deploying to a staging or production environment. Before promotion: Before promoting a build to the next environment (e.g., staging → production). On every pipeline run: As part of your CI/CD pipeline, especially in automated deployments.

Smoke tests are great for exposing unexpected build errors, connection errors, and for validating a server’s expected response after a new release is deployed to a target environment. For example, a quick, simple smoke test could validate that an application is accessible and is responding with expected response codes like OK 200, 300, 301, 404, etc. The examples in this post will test that the deployed app responds with an OK 200 server code and will also validate that the default page content renders the expected text.

Prerequisites

To follow along, you’ll need the following:

• A GitHub account
• A CircleCI account
• Docker
• Kubernetes
• Basic familiarity with Google Kubernetes Engine (GKE)
• smoke.sh - a smoke testing framework
• Pulumi

Project Setup

This tutorial builds on configuration and code that are featured in a previous post, Automate releases from your pipelines using Infrastructure as Code. The full source code is available in this repo.

Clone the example repository

If you haven’t already, start by cloning the example repository:

git clone https://github.com/CIRCLECI-GWP/pulumi-project.git

cd pulumi-project

If you’re new to this series, or if you need to set up your Google Cloud Platform (GCP) project and credentials, please follow the setup instructions in the previous tutorial. This will walk you through creating a Python application and deploying it to a Google Kubernetes Engine (GKE) cluster using CircleCI and Pulumi.

Continuing from the previous tutorial

If you still have your original GCP project and Pulumi stack from the previous tutorial, you can simply add a tests folder to your project for smoke testing. You have two options for adding smoke tests to your project:

• Using a shell script (smoke.sh) or
• Using a Python script (with the requests library).

In this tutorial, we will explore both approaches.

If your original GCP project has been deleted or is no longer accessible, you will need to:

• Create a new GCP project and service account.
• Create a new Pulumi stack (for example, smoketest) to keep your infrastructure state clean and avoid errors from missing resources.
• Update your stack configuration (e.g., Pulumi.smoketest.yaml) with your new GCP project details.

Note: For the purpose of this tutorial, we will demonstrate using a new Pulumi stack and a new GCP project. This approach ensures a clean setup and is a best practice when your original cloud resources are no longer available.

Create a new Pulumi stack

If you are starting with a new GCP project or want to isolate your smoke test environment, create a new Pulumi stack named smoketest:

pulumi stack init smoketest

This command initializes a new Pulumi stack named smoketest. You can use this stack to manage your smoke test infrastructure separately from your main application stack.

Next, set the configuration for your new stack with your GCP project details. This will also generate the Pulumi.smoketest.yaml file in your project directory:

pulumi config set gcp:project <your-new-gcp-project-id> --stack smoketest
pulumi config set gcp:region <your-region> --stack smoketest
pulumi config set gcp:zone <your-zone> --stack smoketest
pulumi config set gke:name smoketest --stack smoketest

Your Pulumi.smoketest.yaml file will look like this:

config:
  gcp:project: <your-new-gcp-project-id>
  gcp:region: <your-region>
  gcp:zone: <your-zone>
  gke:name: smoketest

Adding smoke tests to your pipeline

In the previous tutorial, your CI/CD pipeline ran unit tests, built and pushed a Docker image, and used Pulumi to provision a GKE cluster and deploy your application. However, it did not include any smoke tests to verify that the application was actually running and accessible after deployment.

Without smoke tests, you have no automated way to confirm that your deployment was successful and that your application is functioning as expected in the new environment.

In this section, we’ll show you how to enhance your pipeline by adding smoke tests. These tests will automatically validate that your application is up and responding correctly after each deployment.

How do I write a smoke test?

The first step is to develop test cases that define the steps required to validate an application’s functionality. Identify the functionality that you want to validate, and then create scenarios to test it. In this tutorial, I’m intentionally describing a very minimal scope for testing. For our sample project, the main goal is to confirm that the application is accessible after deployment and that the default page renders the expected static text.

I prefer to outline and list the items I want to test because it suits my style of development. The outline shows the factors I considered when developing the smoke tests for this app. Here is an example of how I developed test cases for this smoke test:

• What language/test framework?
  • Bash
  • smoke.sh
• When should this test be executed?
  • After the GKE cluster has been created
• What will be tested?
  • Test: Is the application accessible after it is deployed?
    • Expected Result: Server responds with code 200
  • Test: Does the default page render the text “Welcome to CI/CD”
    • Expected Result: TRUE
  • Test: Does the default page render the text “Version Number: “
    • Expected Results: TRUE
• Post test actions (must occur regardless of pass or fail)
  • Write test results to standard output
  • Destroy the GKE cluster and related infrastructure
    • Run pulumi destroy

For this tutorial, we’ll demonstrate smoke tests using the open-source Bash framework smoke.sh, which is simple to implement and fits our needs well. You can also use other languages or frameworks, such as Python with the requests library. Let’s start by writing a test script with smoke.sh.

Create smoke test using smoke.sh

Start by creating a new folder in your project root:

mkdir -p tests

Add Bash-based smoke tests with smoke.sh

Download the smoke.sh framework and place it in the tests/ directory:

curl -o tests/smoke.sh https://raw.githubusercontent.com/asm89/smoke.sh/master/smoke.sh
chmod +x tests/smoke.sh

Now, create a new file named smoke_test in the tests/ directory. This file will contain the smoke test script that uses the smoke.sh framework to perform the tests.

#!/bin/bash

. tests/smoke.sh

TIME_OUT=300
TIME_OUT_COUNT=0
PULUMI_STACK="smoketest"
PULUMI_CWD="."
SMOKE_IP=$(pulumi stack --stack $PULUMI_STACK --cwd $PULUMI_CWD output app_endpoint_ip)
SMOKE_URL="http://$SMOKE_IP"

while true
do
  STATUS=$(curl -s -o /dev/null -w '%{http_code}' $SMOKE_URL)
  if [ $STATUS -eq 200 ]; then
    smoke_url_ok $SMOKE_URL
    smoke_assert_body "Welcome to CI/CD"
    smoke_assert_body "Version Number:"
    smoke_report
    echo "\n\n"
    echo 'Smoke Tests Successfully Completed.'
    echo 'Terminating the Kubernetes Cluster in 300 seconds...'
    sleep 300
    pulumi destroy --stack $PULUMI_STACK --cwd $PULUMI_CWD --yes
    break
  elif [[ $TIME_OUT_COUNT -gt $TIME_OUT ]]; then
    echo "Process has Timed out! Elapsed Timeout Count.. $TIME_OUT_COUNT"
    pulumi destroy --stack $PULUMI_STACK --cwd $PULUMI_CWD --yes
    exit 1
  else
    echo "Checking Status on host $SMOKE... $TIME_OUT_COUNT seconds elapsed"
    TIME_OUT_COUNT=$((TIME_OUT_COUNT+10))
  fi
  sleep 10
done

Make it executable:

chmod +x tests/smoke_test

Now that you’ve create the smoke_test script, here’s a summary of how it works and what each part does.

Line-by-line description of the smoke_test file

Start at the top of the file.

#!/bin/bash

. tests/smoke.sh

This snippet specifies the Bash binary to use and also specifies the file path to the core smoke.sh framework to import/include in the smoke_test script.

The next snippet defines environment variables that will be used throughout the smoke_test script. Here is a list of each environment variable and its purpose:

TIME_OUT=300
TIME_OUT_COUNT=0
PULUMI_STACK="k8s"
PULUMI_CWD="."
SMOKE_IP=$(pulumi stack --stack $PULUMI_STACK --cwd $PULUMI_CWD output app_endpoint_ip)
SMOKE_URL="http://$SMOKE_IP"

• PULUMI_STACK="smoketest" is used by Pulumi to specify the Pulumi app stack.
• PULUMI_CWD="." is the path to the Pulumi infrastructure code.
• SMOKE_IP=$(pulumi stack --stack $PULUMI_STACK --cwd $PULUMI_CWD output app_endpoint_ip) is the Pulumi command used to retrieve the public IP address of the application on the GKE cluster. This variable is referenced throughout the script.
• SMOKE_URL="http://$SMOKE_IP" specifies the url endpoint of the application on the GKE cluster.

while true
do
  STATUS=$(curl -s -o /dev/null -w '%{http_code}' $SMOKE_URL)
  if [ $STATUS -eq 200 ]; then
    smoke_url_ok $SMOKE_URL
    smoke_assert_body "Welcome to CI/CD"
    smoke_assert_body "Version Number:"
    smoke_report
    echo "\n\n"
    echo 'Smoke Tests Successfully Completed.'
    echo 'Terminating the Kubernetes Cluster in 300 seconds...'
    sleep 300
    pulumi destroy --stack $PULUMI_STACK --cwd $PULUMI_CWD --yes
    break
  elif [[ $TIME_OUT_COUNT -gt $TIME_OUT ]]; then
    echo "Process has Timed out! Elapsed Timeout Count.. $TIME_OUT_COUNT"
    pulumi destroy --stack $PULUMI_STACK --cwd $PULUMI_CWD --yes
    exit 1
  else
    echo "Checking Status on host $SMOKE... $TIME_OUT_COUNT seconds elapsed"
    TIME_OUT_COUNT=$((TIME_OUT_COUNT+10))
  fi
  sleep 10
done

This snippet is where all the magic happens. It’s a while loop that executes until a condition is true or the script exits. In this case, the loop uses a curl command to test if the application returns an OK 200 response code. Because this pipeline is creating a brand new GKE cluster from scratch, there are transactions in the Google Cloud Platform that need to be complete before we begin smoke testing.

• The GKE cluster and application service must be up and running.
• The $STATUS variable is populated with the results of the curl requests then tested for the value of 200. Otherwise, the loop increments the $TIME_OUT_COUNT variable by 10 seconds, then waits for 10 seconds to repeat the curl request until the application is responding.
• Once the cluster and app are up, running, and responding, the STATUS variable will produce a 200 response code and the remainder of the tests will proceed.

The smoke_assert_body "Welcome to CI/CD" and smoke_assert_body "Version Number: " statements are where I test that the welcome and version number texts are being rendered on the webpage being called. If the result is false, the test will fail, which will cause the pipeline to fail. If the result is true, then the application will return a 200 response code and our text tests will result in TRUE. Our smoke test will pass and execute the pulumi destroy command that terminates all of the infrastructure created for this test case. Since there is no further need for this cluster, it will terminate all the infrastructure created in this test.

This loop also has an elif (else if) statement that checks to see if the application has exceeded the $TIME_OUT value. The elif statement is an example of exception handling, which enables us to control what happens when unexpected results occur.

If the $TIME_OUT_COUNT value exceeds the TIME_OUT value, then the pulumi destroy command is executed and terminates the newly created infrastructure. The exit 1 command then fails your pipeline build process. Regardless of test results, the GKE cluster will be terminated because there really isn’t a need for this infrastructure to exist outside of testing.

Integrate smoke tests into the CI/CD pipeline

Add a new run step below the pulumi/update step of the deploy_to_gcp job in your .circleci/config.yml file. This will ensure that the smoke test runs after the application is deployed to the GKE cluster.

      ...
      - run:
          name: Run Smoke Test against GKE
          command: |
            echo 'Initializing Smoke Tests on the GKE Cluster'
            ./tests/smoke_test
            echo "GKE Cluster Tested & Destroyed"
      ...

This snippet demonstrates how to integrate and execute the smoke_test script into an existing CI/CD pipeline. Adding this new run block ensures that every pipeline build will test the application on a live GKE cluster and provide a validation that the application passed all test cases.

If you created a new stack for smoke testing (for example, smoketest), make sure to update the stack name in your pipeline configuration. Replace any references to the old stack (such as k8s) with your new stack name (smoketest):

- pulumi/refresh:
    stack: smoketest
    working_directory: ${HOME}/project/pulumi-project/
- pulumi/update:
    stack: smoketest
    working_directory: ${HOME}/project/pulumi-project/

This ensures that your pipeline uses the correct Pulumi stack and deploys to the intended GCP project and environment.

Once you’ve updated your configuration and added your smoke test scripts, save all your changes and push them to your repository.

Update CircleCI environment variables

If you created a new GCP project and service account, don’t forget to update the environment variables for your service account credentials in your CircleCI project settings. This ensures your pipeline can authenticate and deploy resources to your new GCP project.

![Smoke test success]

Using Python-based smoke test with pytest

Alternatively, you can write your smoke test in Python using pytest and the requests library. This approach is modern, easy to read, and integrates seamlessly with most CI/CD pipelines.

Create a file named test_smoke.py in the tests/ directory with the following content:

import requests

def test_homepage():
    resp = requests.get("http://<your-app-url>")
    assert resp.status_code == 200
    assert "Welcome to CI/CD" in resp.text

Replace with the actual endpoint or use an environment variable.

You can add this test to your CI/CD pipeline by including a step in the deploy_to_gcp job:

      ...
      - run:
          name: Run Smoke Test with pytest
          command: |
            pip install requests pytest
            pytest tests/test_smoke.py
      ...

This will install the necessary dependencies and run the smoke test after the deployment step.

More advanced smoke/API testing

If your application exposes APIs or you want to test more complex scenarios, consider using tools like Postman or k6. For a full API smoke testing tutorial, see Testing an API with Postman.

Wrapping up

In summary, you have demonstrated the advantages of using smoke tests and Infrastructure as Code in CI/CD pipelines to test builds in their target deployment environments. Testing an application in its target environment provides valuable confidence that it will behave as you expect when it’s deployed. Integrating smoke testing into CI/CD pipelines adds another layer of confidence in application builds.

Thanks for reading!