TutorialsSep 11, 201910 min read

Build, test, and deploy a Go application to AWS ECS

DevOps Engineer at Andela

In this tutorial example, we will deploy a simple Go application to Amazon EC2 Container Service (ECS). Then we will automatically build, test, and deploy subsequent versions of the app using CircleCI. In order to ensure a good grasp of the technologies used, we are going to do this gradually, with the major steps being:

Create a security group
Create an ECS cluster with 1 instance
Create an ECS task definition
Create a service that runs the task definition
Create and configure an Amazon Elastic Load Balancer (ELB) and target group that will associate with our cluster’s ECS service
Use the DNS name on our ELB to access the application (to test that it works)
Configure CircleCI using the circleci/aws-ecr@6.2.0 orb to build and push an updated image to Amazon Elastic Container Registry (ECR)
Configure CircleCI using the circleci/aws-ecs@0.0.11 orb to deploy the updated image to the cluster we created earlier

To fully appreciate the benefits of Amazon ECS, you first need to understand Docker. Knowledge of Docker and containerization is assumed throughout this tutorial. Additionally, you’ll need an intorductory-level understanding of continuous integration and continuous deployment (CI/CD). In the next section, we’ll cover some of the technologies and terms that we’ll be using.

Summary of the technologies and terms used

ECS recap: ECS is a cloud computing service in Amazon Web Services (AWS) that manages containers. It enables developers to deploy and manage scalable applications that run on groups of servers, called clusters, through application programming interface (API) calls and task definitions. Essentially, it is a task scheduler. The tasks that it creates map to running Docker containers. It determines, based on available resources, where to run your tasks on the resources in your cluster. While other container technologies exist (LXC, rkt, etc.), because of Docker’s massive adoption, ECS was designed to work natively with Docker containers.
Task definition: Look at it as a recipe describing how to run your containers. It has information such as the ports to expose on the container, the memory and CPU to allocate, as well as the Docker image from which to launch the container. In our example, it would be one container, the image to use, the CPU and memory to allocate, and the ports to expose.
Task: An ECS task is a unit of running containers instantiated from a task definition. They are a logical grouping of 1 to N containers that run together on the same instance, with N defined by you between 1 and 10. Multiple tasks can be created by one task definition, as demand requires.
Service: An ECS service is used to guarantee that you always have some number of tasks running at all times. If a task’s container exits due to error, or the underlying EC2 instance fails and is replaced, the ECS service will replace the failed task. We create clusters so that the service has plenty of resources in terms of CPU, memory, and network ports to use. To us, it doesn’t really matter which instance tasks run on, so long as they run. A service configuration references a task definition. A service is responsible for creating tasks.
Cluster: An ECS cluster is a grouping of (container) instances (or tasks, in Fargate) that lie within a single region, but can span multiple Availability Zones. ECS handles the logic of scheduling, maintaining, and handling scaling requests to these instances. It also takes away the work of finding the optimal placement of each task based on your CPU and memory needs. A cluster can run many services. If you have multiple applications that are a part of your product, you may wish to put several of them on one cluster. This makes efficient use of the resources available and minimizes setup time.
Container Instance: This is an Amazon Elastic Compute Cloud (EC2) instance that is running the Amazon ECS container agent. It has a specifically defined IAM policy and role and has been registered to a cluster. When you run tasks with Amazon ECS, your tasks, using the EC2 launch type, are placed on your active container instances.
CircleCI orbs: Orbs are packages of CircleCI configuration that can be shared across projects. Orbs allow you to make a single bundle of jobs, commands, and executors that can reference each other and can be imported into a CircleCI build configuration and invoked in their own namespace.

A simple Go application

The following will be the directory structure for our project:

.
├── .circleci
│   └── config.yml
├── Dockerfile
├── README.md
├── ecs-service.json
├── main.go
└── task-definition.json

1 directory, 6 files

The complete application can be found in this repo. To follow along, clone it to the desired location with this line in your terminal:

$ git clone https://github.com/daumie/circleci-ecs.git

If you are only interested in the Dockerfile, you can find that here. The Go application’s main.go file can be found here.

Let’s put it all together. First things first, create and activate an AWS Account. Then, install and configure the AWS CLI on your local machine. We will use it to interact with AWS from the command line interface.

Next, we will use the default Virtual Private Cloud (VPC) that is automatically created when we created our AWS account. If it is not available, you can create a default VPC by running:

$ aws ec2 create-default-vpc

Confirm that we have a VPC that we can work with by running:

$ aws ec2 describe-vpcs

After confirming that we have a default VPC, let’s create a security group that we’ll use later:

$ aws ec2 create-security-group --group-name circleci-demo-sg --description "Circle CI Demo Security Group"

Next, we will be creating an ECS Cluster and the associated EC2 instance. We will call the cluster circleci-demo-cluster. We need to attach the circleci-demo-sg security group that we created in earlier.

Cluster name: circleci-demo-cluster
EC2 instance type: t2.medium
Networking: Use default VPC with all of its subnets
Security group: (circleci-demo-sg) you will use its id
Container Instance IAM Role: ecsInstanceRole

Wait a few minutes and then confirm that the container instance has successfully registered to the circleci-demo-cluster. You can confirm it by clicking the ECS Instances tab under Clusters/my-cluster.

Create the application image and push it to AWS ECR

Create a Docker image locally and push it to ECR:

$ docker build -t circleci-ecs:v1 .

Step 1/14 : FROM golang:latest as builder
 ---> be63d15101cb
...

Create an image repository on ECR by following these instructions. Name it circleci-demo:

AWS accounts have unique ID’s. Change 634223907656 in the following command appropriately. After getting the repository name, we can now tag the image accordingly:

$ docker tag circleci-ecs:v1 634223907656.dkr.ecr.eu-west-2.amazonaws.com/circleci-demo:latest

You can authenticate AWS ECR repositories for Docker CLI with credential helper. Let’s use the command below to authenticate (change the region as appropriate).

$ aws ecr get-login --no-include-email --region eu-west-2 | bash

Then, push the image to the ECR repository:

$ docker push 634223907656.dkr.ecr.eu-west-2.amazonaws.com/circleci-demo:latest

Now that we have an image in the ECR registry, we need a task definition that will be our blueprint to start the Go application. Our task-definition.json file in our project’s root has these lines of code:

{
    "family": "circleci-demo-service",
    "containerDefinitions": [
        {
            "name": "circleci-demo-service",
            "image": "634223907656.dkr.ecr.eu-west-2.amazonaws.com/circleci-demo:latest",
            "cpu": 128,
            "memoryReservation": 128,
            "portMappings": [
                {
                    "containerPort": 8080,
                    "protocol": "tcp"
                }
            ],
            "command": [
                "./main"
            ],
            "essential": true
        }
    ]
}

Note: Remember to change the image to the one you pushed to ECR.

Let’s register the task definition from the command line interface with:

$ aws ecs register-task-definition --cli-input-json file://task-definition.json

Confirm that the task definition successfully registered in the ECS Console:

Create an ELB and a target group to later associate with our ECS service

We are creating an ELB because we eventually want to load balance requests across multiple containers and we also want to expose our Go application to the internet for testing. For this, we will use the AWS Console. Go to EC2 Console > Load Balancing > Load Balancers and click Create Load Balancer and select Application Load Balancer.

Configure the load balancer

Name it circleci-demo-elb and select internet-facing.
Under listeners, use the default listener with a HTTP protocol and port 80.
Under Availability Zone, chose the VPC that was used during cluster creation and choose the subnets that you want.

Configure security settings

Skip the warning as we won’t be using SSL.

Configure security groups

Create a new security group named circleci-demo-elb-sg and open up port 80 and source 0.0.0.0/0 so anything from the outside world can access the ELB on port 80.

Configure routing

Create a new target group name circleci-demo-target-group with port 80.

Register targets

Review

Review the load balancer details

The circleci-demo-elb-sg security group opens the circleci-demo-elb load balancer’s port 80 to the world. Now, we need to make sure that the circleci-demo-sg security group associated with the ECS instance allows traffic from the load balancer. To allow all ELB traffic to hit the container instance, run the following:

$ aws ec2 authorize-security-group-ingress --group-name circleci-demo-sg --protocol tcp --port 1-65535 --source-group circleci-demo-elb-sg

Confirm that the rules were added to the security groups via the EC2 Console:

inbound rules

outbound rules

With these security group rules:

Only port 80 on the ELB is exposed to the outside world.
Any traffic from the ELB going to a container instance with the circleci-demo-target-group group is allowed.

Create a service

The next step is to create a service that runs the circleci-demo-service task definition (defined in the task-definition.json file). Our ecs-service.json file in our project’s root has these lines of code:

{
    "cluster": "circleci-demo-cluster",
    "serviceName": "circleci-demo-service",
    "taskDefinition": "circleci-demo-service",
    "loadBalancers": [
        {
            "targetGroupArn": "arn:aws:elasticloadbalancing:eu-west-2:634223907656:targetgroup/circleci-demo-target-group/a5a0f047c845fcbb",
            "containerName": "circleci-demo-service",
            "containerPort": 8080
        }
    ],
    "desiredCount": 1,
    "role": "ecsServiceRole"
}

To find the targetGroupArn that was created when creating the circleci-demo-elb load balancer, go to EC2 Console > Load Balancing > Target Groups and click the circleci-demo-target-group. Copy it and substitute the one in targetGroupArn in the ecs-service.json file.

Now, create the circleci-demo-service ECS service:

$ aws ecs create-service --cli-input-json file://ecs-service.json

From the ECS console go to Clusters > circleci-demo-cluster > circleci-demo-service and view the Tasks tab. Confirm that the container is running:

Test that everything is working

Verify the ELB publicly available DNS endpoint with curl:

$ curl circleci-demo-elb-129747675.eu-west-2.elb.amazonaws.com; echo

Hello World!

The same can be confirmed from the browser with:

Configuring CircleCI to build, test, and deploy

After successfully deploying our Go application to ECS, we now want to redeploy the app on every update. By using CircleCi orbs, we will save massive amounts of time by importing pre-built commands, jobs, and executors into our configuration file. This will also reduce the lines of code in our config greatly by eliminating much of the bash scripting required for AWS deployments. We will invoke the following orbs in this project using the orbs key:

circleci/aws-ecr@6.2.0: An orb for working with Amazon’s ECR to build and push and updated image
circleci/aws-ecs@0.0.11: An orb for working with Amazon’s ECS to deploy the updated image to the cluster created earlier

Orbs consist of the following elements:

Commands
Jobs: A set of executable commands or steps
Executors: These define the environment in which the steps of a job will be run, e.g., Docker, Machine, macOS, etc., in addition to any other parameters of that environment

To use CircleCI, we need a configuration file that CircleCI will use to order the operations of building, testing, and deploying. For this project, the config.yml file contains the following lines of code:

version: 2.1

orbs:
  aws-ecr: circleci/aws-ecr@6.2.0
  aws-ecs: circleci/aws-ecs@0.0.11

workflows:
# Log into AWS, build and push image to Amazon ECR
  build_and_push_image:
    jobs:
      - aws-ecr/build-and-push-image:
          account-url: AWS_ECR_ACCOUNT_URL
          aws-access-key-id: AWS_ACCESS_KEY_ID
          aws-secret-access-key: AWS_SECRET_ACCESS_KEY
          create-repo: true
          # Name of dockerfile to use. Defaults to Dockerfile.
          dockerfile: Dockerfile
          # AWS_REGION_ENV_VAR_NAME
          region: AWS_DEFAULT_REGION
          # myECRRepository
          repo: '${MY_APP_PREFIX}'
          # myECRRepoTag
          tag: "$CIRCLE_SHA1"
      - aws-ecs/deploy-service-update:
          requires:
            - aws-ecr/build-and-push-image
          aws-region: AWS_DEFAULT_REGION
          family: '${MY_APP_PREFIX}-service'
          cluster-name: '${MY_APP_PREFIX}-cluster'
          container-image-name-updates: 'container=${MY_APP_PREFIX}-service,tag=${CIRCLE_SHA1}'

We will be using GitHub and CircleCI. Create a CircleCI account, if you don’t have one. Sign up with GitHub. From the CircleCI dashboard click Add Project and add the project from the list shown.

Add the following environment variables:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION
AWS_ECR_ACCOUNT_URL (In this example, “634223907656.dkr.ecr.eu-west-2.amazonaws.com”)
MY_APP_PREFIX (In this example, “circleci-demo”)

Let’s change this line in the main.go file:

html := "Hello World!"

html := "Hello World! Now updated with CircleCI"

Commit and push your changes to GitHub.

You can confirm that the changes were applied from the terminal by running:

$ curl circleci-demo-elb-129747675.eu-west-2.elb.amazonaws.com ; echo

Hello World! Now updated with CircleCI

The same can be confirmed from the browser:

Conclusion

We have built a simple GO application and deployed it to ECR. Now, we can add tests to our application to make sure that those tests are passed before updating our ECS instances. While this tutorial used a basic application, this is a mature deployment pipeline that works for many real-world situations.

Additionally, using CircleCI orbs improves productivity by simplifying how we write our CircleCI configuration. Orbs can also be shared, which saves time by using pre-built commands, jobs, and executors over and over in our configuration files. Orbs are not limited to CircleCI + ECS deployments. You can go through the full list of available orbs in the Orb Registry to find the ones that align with your choice of cloud platform, programming language, and more.