Start Building for Free

Configuring Nomad Client Metrics

1 week ago4 min read
Server v2.x
Server Admin
On This Page

Nomad Metrics is a helper service used to collect metrics data from the Nomad server and clients running on the Services and Nomad instances respectively. Metrics are collected and sent using the DogStatsD protocol and sent to the Services machine.

Nomad Metrics Server

The Nomad Metrics container is run on the services host using the server flag and is installed as part of the CircleCI server installation process, requiring no additional configuration.

Nomad Metrics Client

The Nomad Metrics client is installed and run on all Nomad client instances. You will need to update your AWS Launch Configuration in order to install and configure it. Additionally, you will need to modify the AWS security group to ensure that UDP port 8125 is open on the Services machine. Steps for both configuration changes are explained below.

Updating the Services machine Security Group

  1. Select the Instances link located under the Instances group in the left sidebar.

  2. Select the Services Box Instance. The name tag typically resembles circleci_services.

  3. In the description box at the bottom, select the users security group link located next to the Security Groups section. It typically resembles *_users_sg.

  4. This will take you straight to the Security Group page highlighting the users security group. In the description box at the bottom, select the Inbound tab followed by the Edit button.

  5. Select the Add a Rule button. From the drop-down, select Custom UDP Rule. In the Port Range field enter 8125.

  6. The source field gives you a few options. However, this ultimately depends on how you have configured the VPC and subnet. Below are some more common scenarios.

    1. (Suggested) Allow traffic from the nomad client subnet. You can usually match the entries used for ports 4647 or 3001. For example,

    2. Allow all traffic to UDP port 8125 using

  7. Press the Save Button

Updating the AWS Launch Configuration


AWS EC2 Launch Configuration ID
  1. Select the Auto Scaling Groups (ASG) link in the sidebar on the left.

  2. Locate the ASG with a name tag similar to`*_nomad_clients_asg`

  3. The Launch Configuration name is next to the ASG name IE terraform-20180814231555427200000001

AWS EC2 Services Box Private IP Address
  1. Select the Instances link located under the Instances group in the left sidebar

  2. Select the Services Box Instance. The name tag typically resembles circleci_services

  3. In the description box at the bottom of the page, make note of the private IP address.

Updating the Launch Configuration

  1. Select the Launch Configurations link located under Auto Scaling in the sidebar to the left. Select the Launch Configuration you retrieved in the previous steps.

  2. In the description pane at the bottom, select the Copy launch configuration button.

  3. Once the configuration page opens, select 3. Configure details link located at the top of the page.

  4. Update the Name field to something meaningful IE nomad-builder-with-metrics-lc-DATE.

  5. Select the Advanced Details drop down.

  6. Copy and paste the launch configuration script from below in the text field next to User data.

  7. IMPORTANT: Enter the private IP address of the services box at Line 10. For example, export SERVICES_PRIVATE_IP="".

  8. Select the Skip to review button and then the Create launch configuration button.


set -exu

export http_proxy=""
export https_proxy=""
export no_proxy=""
export aws_instance_metadata_url=""
export PUBLIC_IP="$(curl $aws_instance_metadata_url/latest/meta-data/public-ipv4)"
export PRIVATE_IP="$(curl $aws_instance_metadata_url/latest/meta-data/local-ipv4)"
export DEBIAN_FRONTEND=noninteractive
UNAME="$(uname -r)"
export CONTAINER_NAME="nomad_metrics"
export CONTAINER_IMAGE="circleci/nomad-metrics:0.1.198-5f5befe"
export NOMAD_METRICS_PORT="8125"

echo "-------------------------------------------"
echo "     Performing System Updates"
echo "-------------------------------------------"
apt-get update && apt-get -y upgrade

echo "--------------------------------------"
echo "        Installing NTP"
echo "--------------------------------------"
apt-get install -y ntp

# Use AWS NTP config for EC2 instances and default for non-AWS
if [ -f /sys/hypervisor/uuid ] && [ `head -c 3 /sys/hypervisor/uuid` == ec2 ]; then
cat <<EOT > /etc/ntp.conf
driftfile /var/lib/ntp/ntp.drift
disable monitor

restrict default ignore
restrict mask
restrict nomodify notrap

server prefer iburst

service ntp restart

echo "--------------------------------------"
echo "        Installing Docker"
echo "--------------------------------------"
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL | apt-key add -
add-apt-repository "deb [arch=amd64] $(lsb_release -cs) stable"
apt-get install -y "linux-image-$UNAME"
apt-get update
apt-get -y install docker-ce=5:18.09.9~3-0~ubuntu-xenial

# force docker to use userns-remap to mitigate CVE 2019-5736
apt-get -y install jq
mkdir -p /etc/docker
[ -f /etc/docker/daemon.json ] || echo '{}' > /etc/docker/daemon.json
cp /etc/docker/daemon.json /etc/docker/daemon.json.orig
jq '.["userns-remap"]="default"' /etc/docker/daemon.json > "$tmp" && mv "$tmp" /etc/docker/daemon.json

sudo echo 'export http_proxy="${http_proxy}"' >> /etc/default/docker
sudo echo 'export https_proxy="${https_proxy}"' >> /etc/default/docker
sudo echo 'export no_proxy="${no_proxy}"' >> /etc/default/docker
sudo service docker restart
sleep 5

echo "--------------------------------------"
echo " Populating /etc/circleci/public-ipv4"
echo "--------------------------------------"
if ! (echo $PUBLIC_IP | grep -qP "^[\d.]+$")
  echo "Setting the IPv4 address below in /etc/circleci/public-ipv4."
  echo "This address will be used in builds with \"Rebuild with SSH\"."
  mkdir -p /etc/circleci
  echo $PRIVATE_IP | tee /etc/circleci/public-ipv4

echo "--------------------------------------"
echo "         Installing nomad"
echo "--------------------------------------"
apt-get install -y zip
curl -o
mv nomad /usr/bin

echo "--------------------------------------"
echo "      Creating config.hcl"
echo "--------------------------------------"
export INSTANCE_ID="$(curl $aws_instance_metadata_url/latest/meta-data/instance-id)"
mkdir -p /etc/nomad
cat <<EOT > /etc/nomad/config.hcl
log_level = "DEBUG"
name = "$INSTANCE_ID"
data_dir = "/opt/nomad"
datacenter = "default"
advertise {
    http = "$PRIVATE_IP"
    rpc = "$PRIVATE_IP"
    serf = "$PRIVATE_IP"
client {
    enabled = true
    # Expecting to have DNS record for nomad server(s)
    servers = ["$SERVICES_PRIVATE_IP:4647"]
    node_class = "linux-64bit"
    options = {"driver.raw_exec.enable" = "1"}

telemetry {
    publish_node_metrics = true
    statsd_address = "$SERVICES_PRIVATE_IP:8125"

echo "--------------------------------------"
echo "      Creating nomad.conf"
echo "--------------------------------------"
cat <<EOT > /etc/systemd/system/nomad.service
ExecStart=/usr/bin/nomad agent -config /etc/nomad/config.hcl

echo "--------------------------------------"
echo "   Creating ci-privileged network"
echo "--------------------------------------"
docker network create --driver=bridge --opt ci-privileged

echo "--------------------------------------"
echo "      Starting Nomad service"
echo "--------------------------------------"
service nomad restart

echo "--------------------------------------"
echo "      Setting up Nomad metrics"
echo "--------------------------------------"
docker pull $CONTAINER_IMAGE
docker rm -f $CONTAINER_NAME || true

docker run -d --name $CONTAINER_NAME \
    --rm \
    --net=host \
    --userns=host \
    start --nomad-uri=http://localhost:4646 --statsd-host=$SERVICES_PRIVATE_IP --statsd-port=$NOMAD_METRICS_PORT --client

Updating the Auto Scaling Group

  1. Select the Auto Scaling Groups (ASG) link in the sidebar on the left.

  2. Select the ASG with a name tag similar to *_nomad_clients_asg.

  3. In the description box at the bottom, select the Edit button.

  4. Select the newly created Launch Configuration from the drop-down.

  5. Press the Save button.

  6. At this point, the older Nomad client instances will begin shutting down. They will be replaced with newer Nomad clients running Nomad Metrics.

StatsD Metrics





1 if the last poll of the Nomad agent failed; 0 otherwise. This gauge is set independent of circle.nomad.client_agent.poll_failure when nomad-metrics is operating in --client and --server modes simultaneously.


Total number of pending jobs across the cluster.


Total number of running jobs across the cluster.


Total number of complete jobs across the cluster.


Total number of dead jobs across the cluster.





1 if the last poll of the Nomad agent failed; 0 otherwise.


(See below)



(See below)



(See below)


(See below)



(See below)



(See below)


(See below)



(See below)



(See below)


(See below)



(See below)



(See below)

Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.