Using workflows to deploy an API to multiple environments

Deploying applications to a staging environment for testing before deploying to production is a common practice. Testing to make sure features are working properly first is a best-practice, and many developers, startups, and organizations deploy their applications this way.

I have found that it quickly becomes time consuming and inefficient to go through a manual process every time updates are made to my applications. How great would it be to automate the process? In this tutorial, we will deploy an API to both a staging and production environment. This process is also known as multi-deployment.

In our multi-deployment process, we will create a workflow that first deploys an API to a staging environment, and runs automated tests against the staged API using Postman. If the tests pass, our workflow will deploy the API to the production environment without our intervention. If the tests do not pass, the API will not be deployed to production.

Prerequisites

To follow this tutorial, a few things are required:

1. Node.js installed on your system
2. Postman for Desktop installed on your system (you can download it here)
3. Heroku or another application hosting platform
4. A CircleCI account
5. A GitHub account

With all these installed and set up, it is time to begin the tutorial.

Cloning the API project

To begin, you will need to clone the API project. We will be working with a simple Node.js API application with a root endpoint and two other endpoints for creating and fetching users. Clone the project by running this command:

git clone --single-branch --branch base-project https://github.com/coderonfleek/multi-deploy-api.git

When the cloning process is complete, go to the root of the project and install the dependencies:

cd multi-deploy-api
npm install

Next, run the application:

npm start

The application will start listening on a default port of 3000.

Open Postman and make a GET request to the http://localhost:3000/users/get endpoint. This will return an array of users.

Our API is ready to deploy.

Setting up staging and deploy environments on Heroku

The next task is to create the deployment environments. You will be creating one for staging (where the application’s updates will be deployed, for testing) and deploy (the production environment). We will create these environments on Heroku for this tutorial, but you can use the hosting platform you prefer.

Go to your Heroku account dashboard and click New, then Create new app. On the app creation page, create the staging application environment.

Now, repeat the same process to create the production app.

The environments are now set up for deploying the application. Finally, on Heroku, get your API Key from the Account tab on the Account Settings page.

Setting up API tests with Postman Collections

After it is deployed to the staging environment, we want the API to be tested. For this tutorial, we will use Postman to set up API tests that can be automated.

The first step is to create a dedicated environment for the API requests. From the Postman desktop, click the Manage Environments cog icon (it is at the top right corner). The Manage Environments dialog shows any environments that are already in place. Click Add to create a new environment.

In the new environment dialog, enter a name for the new environment. Fill in the staging environment API base URL (https://users-api-staging.herokuapp.com) as an environment variable (api_url). What you enter in INITIAL VALUE is duplicated for CURRENT VALUE. Keep it like that; using CURRENT VALUE is beyond the scope of this tutorial.

Click Add to finish creating the environment. Switch to the new environment using the dropdown at the top right of your screen.

Creating a collection

The next step is to create a Postman Collection for the user endpoints of the API you will be testing.

Click Collections from the left sidebar. Then click New Collection.

On the New Collection dialog, fill in the name (Users) of your collection. You can also add a description if you want to add more information about the collection.

Click Create to finish setting up the collection. The new collection is immediately shown on the left sidebar under Collections.

Adding requests to the collection

Now it is time to add requests to your collection. The API consists of two endpoints:

• {{api_url}}/users/get (GET): Fetches a list of user profiles
• {{api_url}}/users/create (POST): Creates a new user profile

You will be adding a request for each endpoint. To get started, click the flyout menu beside the collection (the arrow icon).

Click Add requests. On the New Request dialog, create a request for the {{api_url}}/users/get endpoint. Click Save to Users-API-Collection to save it to the collection you created earlier.

Now that the request is created, a new request tab is loaded. Using the api_url variable you created, enter the endpoint for the request in the address bar ({{api_url}}/users/get). Make sure to select GET as the request method. Click Save.

Next, create a request for the {{api_url}}/users/create endpoint. We need to get random values so that the POST request can create test users. For that, we need a Pre-request script.

Open the Pre-request Script tab and add the following script:

let random = +new Date();

pm.globals.set("name", `Test-User-${random}`);

This script uses the current timestamp to randomly create names for each fired request. The random name variable is set as a global variable for the request instance.

Now write the request body with the dynamic name variable.

A dynamic name parameter will be used for each request to the /users/create endpoint.

Adding tests

It is time to add some tests to the requests you just created.

Click the {{api_url}}/users/get request (in the left sidebar) to make sure it is loaded. Click the Tests tab. In the window, add:

pm.test("Request is successful with a status code of 200", function () {
  pm.response.to.have.status(200);
});

pm.test("Check that it returns an array", function () {
  var jsonData = pm.response.json();
  pm.expect(jsonData).to.be.an("array");
});

The first test checks that the request returns successfully with a status code of 200. The second test makes sure that it also returns an array.

Click the user creation request Test tab to add:

pm.test("User creation was successful", function () {
  pm.expect(pm.response.code).to.be.oneOf([200, 201, 202]);
});

pm.test("Confirm response message", function () {
  var jsonData = pm.response.json();
  pm.expect(jsonData.message).to.eql("User successfully registered");
});

The first test here checks that user creation is successful by asserting a status code of either 200, 201 or 202. The second test makes sure that the correct response message is returned.

Note: Be sure to click Save whenever you make a change to either of the requests in this collection.

Configuring automated testing with Newman

Next, we will use Postman’s CLI tool, Newman, to automate how the tests run.

Your collection will need a public URL. This link will point to the version of your collection hosted on the Postman service. The major advantage of using this link instead of exporting your collection as json is that changes to your collection will always be available using the link as long as you’re logged into Postman. When you sign in on the desktop, Postman syncs all your local collections to your Postman account. If you need a Postman account, the sign-in pagewill guide you through creating one.

To get your public URL, click Share from your collection’s flyout menu. Your collection’s public link is displayed under the Get Link tab.

Using this link, you no longer need to export and move collection files. You do need to export the environment file when it is updated, but this happens less frequently.

Download your Postman environment by clicking the Manage Environments cog icon. Click Download to get the file. The filename will be something like User-API-Tests.postman_environment.json (depending on how you named the environment). Add this environment file to the root of your project.

Next, install the Newman CLI tool at the root of your project by running:

npm install --save-dev newman

Replace the test script in package.json with:

"scripts": {
    ...
    "test": "npx newman run [YOUR_COLLECTION_PUBLIC_URL] -e ./[YOUR_ENVIRONMENT_FILENAME].json"
}

This test script uses npx to run newman against the collection’s URL. The collection URL uses the environment’s filename to specify where the tests will run. Replace the values [YOUR_COLLECTION_PUBLIC_URL] and [YOUR_ENVIRONMENT_FILENAME] with your collection public URL and environment filename, respectively. Newman runs the tests defined in the collection against the requests and picks up any pre-request scripts that have been defined.

Now your tests are set up for automation.

Connecting your project to CircleCI

Begin by pushing your project to GitHub.

Now, go to the Add Projects page on the CircleCI dashboard to add the project.

Click Set Up Project.

On the setup page, click Use Existing Config to indicate that you are setting up a configuration file manually and not using the sample displayed. Next, you get a prompt to either download a configuration file for the pipeline, or to start building.

Click Start Building. This build will fail because we have not set up our configuration file yet. We’ll do this in the next step.

Before you leave the CircleCI console, you need to set up environment variables to deploy the application to both environments (staging and production) on Heroku.

From the Pipelines page, select your application, then click Project Settings.

From the Project Settings side-menu, click Environment Variables, then clickAdd Environment Variable.

Add these variables:

• HEROKU_STAGING_APP_NAME: The Heroku app name for the staging environment (in this case users-api-staging)
• HEROKU_PRODUCTION_APP_NAME: The Heroku app name for the production environment (in this case users-api-production)
• HEROKU_API_KEY: Your Heroku API key

Now you are ready to deploy to both the staging and production environments on Heroku.

Configuring a stage -> test -> deploy workflow

As promised at the beginning of this tutorial, our goal is to create an automated workflow in which an application is:

1. Deployed to a staging environment
2. Tested on the staging address and (if tests pass)
3. Deployed to the production environment

Our next task is to write a deployment pipeline script that performs those steps. Start by creating a folder named .circleci at the root of the project. Add a configuration file named config.yml inside the folder you just created. Enter the following code:

version: 2.1
orbs:
  heroku: circleci/heroku@0.0.10
jobs:
  stage:
    executor: heroku/default
    steps:
      - checkout
      - heroku/install
      - heroku/deploy-via-git:
          app-name: $HEROKU_STAGING_APP_NAME

  test:
    working_directory: ~/repo
    docker:
      - image: circleci/node:10.16.3
    steps:
      - checkout
      - run:
          name: Update NPM
          command: "sudo npm install -g npm@5"
      - restore_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
      - run:
          name: Install Dependencies
          command: npm install
      - save_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
          paths:
            - ./node_modules
      - run:
          name: Run tests
          command: npm run test

  deploy:
    executor: heroku/default
    steps:
      - checkout
      - heroku/install
      - heroku/deploy-via-git:
          app-name: $HEROKU_PRODUCTION_APP_NAME

workflows:
  stage_test_deploy:
    jobs:
      - stage
      - test:
          requires:
            - stage
      - deploy:
          requires:
            - test

That is quite a script, right? Let me break it down and add some detail. First, the heroku orb is pulled in:

orbs:
  heroku: circleci/heroku@0.0.10

Then we create three jobs:

• stage: Checks out the project code from the repo and uses the Heroku CircleCI orb to deploy the application to the staging environment.
• test: Installs the project dependencies needed to have the newman CLI installed. It then runs the test script in package.json, invoking newman to run the tests in the collection against the staging application that was just deployed.
• deploy: Deploys the application to the production environment on Heroku.

Orbs are reusable pipeline packages that abstract common tasks. The Heroku orb that we are using abstracts the process of installing and setting up the heroku CLI tool to deploy the application.

Finally, the script defines the stage_test_deploy workflow. This workflow runs the stage job. Once that job is run successfully, the test job runs tests against the application that was just deployed to staging. When all tests on the staging application have passed, the script runs the deploy job to deploy the application to the production environment. It is a smooth, simple process.

It is time to test our workflow. Commit all changes and push the code to the remote repository to run the pipeline script.

The currently active job is running while the others wait.

A successful result runs the jobs one after the other.

Click the workflow (stage_test_deploy) link to review the steps in the process and their duration.

You can also click each build for details about how it ran.

Great work!

Conclusion

Automating real-life manual tasks can help you save both time and effort, boosting your productivity. In this tutorial, we have automated the process of staging, testing, and deploying an application to a production environment. You can apply what you have learned here to even more complex scenarios and further increase efficiency in your workflows.

Happy coding!