Search Results for ""

Monitoring Your Installation

System Monitoring

To enable metrics forwarding to either AWS Cloudwatch or Datadog, navigate to your CircleCI Management Console, select Settings from the menu bar and scroll down to enable the provider of your choice (your-circleci-hostname.com:8800/settings#cloudwatch_metrics).

VM Service and Docker Metrics

VM Service and Docker services metrics are forwarded via Telegraf, a plugin-driven server agent for collecting and reporting metrics.

Following are the enabled metrics:

Nomad Job Metrics

Nomad job metrics are enabled and emitted by the Nomad Server agent. Five types of metrics are reported:

Metric Description

circle.nomad.server_agent.poll_failure

Returns 1 if the last poll of the Nomad agent failed, otherwise it returns 0.

circle.nomad.server_agent.jobs.pending

Returns the total number of pending jobs across the cluster.

circle.nomad.server_agent.jobs.running

Returns the total number of running jobs across the cluster.

circle.nomad.server_agent.jobs.complete

Returns the total number of complete jobs across the cluster.

circle.nomad.server_agent.jobs.dead

Returns the total number of dead jobs across the cluster.

When the Nomad metrics container is running normally, no output will be written to standard output or standard error. Failures will elicit a message to standard error.

CircleCI Metrics

Introduced in CircleCI Server v2.18

circle.backend.action.upload-artifact-error

Tracks how many times an artifact has failed to upload.

circle.build-queue.runnable.builds

Tracks how many builds flowing through the system are considered runnable.

circle.dispatcher.find-containers-failed

Tracks how many 1.0 builds

circle.github.api_call

Tracks how many api calls CircleCI is making to github

circle.http.request

Tracks the response codes to CircleCi requests

circle.nomad.client_agent.*`

Tracks nomad client metrics

circle.nomad.server_agent.*

Tracks how many nomad servers there are.

circle.run-queue.latency

Tracks how long it takes for a runnable build to be accepted.

circle.state.container-builder-ratio

Keeps track of how many containers exist per builder ( 1.0 only ).

circle.state.lxc-available

Tracks how many containers are available ( 1.0 only )

circle.state.lxc-reserved

Tracks how many containers are reserved/in use ( 1.0 only ).

circleci.cron-service.messaging.handle-message

Provides timing and counts for RabbitMQ messages processed by the cron-service

circleci.grpc-response

Tracks latency over the system grpc system calls.

Supported Platforms

We have two built-in platforms for metrics and monitoring: AWS CloudWatch and DataDog. The sections below detail enabling and configuring each in turn.

AWS CloudWatch

To enable AWS CloudWatch complete the following:

  1. Navigate to the settings page within your Management Console. You can use the following URL, substituting your CircleCI URL: your-circleci-hostname.com:8800/settings#cloudwatch_metrics.

  2. Check Enabled under AWS CloudWatch Metrics to begin configuration.

    AWS CloudWatch
    Figure 1. Enable Cloudwatch

AWS CloudWatch Configuration

There are two options for configuration:

  • Use the IAM Instance Profile of the services box and configure your custom region and namespace.

    Configuration IAM
    Figure 2. CloudWatch Region and Namespace
  • Alternatively, you may use your AWS Access Key and Secret Key along with your custom region and namespace.

    Configuration Alt
    Figure 3. Access Key and Secret Key

After saving you can verify that metrics are forwarding by going to your AWS CloudWatch console.

DataDog

To enable Datadog complete the following:

  1. Navigate your Management Console Settings. You can use the following URL, substituting your CircleCI hostname: your-circleci-hostname.com:8800/settings#datadog_metrics

  2. Check Enabled under Datadog Metrics to begin configuration.

    Enable DataDog
    Figure 4. Enable Datadog Metrics
  3. Enter your DataDog API Key. You can verify that metrics are forwarding by going to your DataDog metrics summary.

    DataDog API Key
    Figure 5. Enter Datadog API key

Custom Metrics

Custom Metrics using a Telegraf configuration file may be use as an alternative to DataDog or AWS Cloudwatch. Using Telegraf allows for more fine grained control.

Customizing Metrics

Following are the steps required to customize which of these metrics you wish to receive:

  1. Check to enable Use Custom Telegraf Metrics from the Management Console settings

    Custom Metrics
    Figure 6. Custom Metrics
  2. SSH into the Services machine

  3. Add the following to /etc/circleconfig/telegraf/statsd.conf

    [[inputs.statsd]]
            service_address = ":8125"
            parse_data_dog_tags = true
            metric_separator = "."
            namepass = []
  4. Under namepass add any metrics you wish to receive, the example below shows choosing to configure just the first 4 from the list above:

    [[inputs.statsd]]
            service_address = ":8125"
            parse_data_dog_tags = true
            metric_separator = "."
            namepass = [
                "circle.backend.action.upload-artifact-error",
                "circle.build-queue.runnable.builds",
                "circle.dispatcher.find-containers-failed",
                "circle.github.api_call"
              ]
  5. Restart the telegraf container by running: sudo docker restart telegraf

  6. Navigate to your Management Console settings page, scroll down to save and restart your installation.

Configuring Custom Metrics

Configuration options are based on Telegraf’s documented output plugins. See their documentation here. For example, if you would like to use the InfluxDB Output Plugin you would need to follow these steps:

  1. SSH into the Servics Machine

  2. cd /etc/circleconfig/telegraf/influxdb.conf

  3. Adding the desired outputs, for example:

    [[output.influxdb]]
      url = "http://52.67.66.155:8086"
      database = "testdb"
  4. Run docker restart telegraf to restart the container to load or reload any changes.

You may check the logs by running docker logs -f telegraf to confirm your output provider (e.g. influx) is listed in the configured outputs. Additionally, if you would like to ensure that all metrics in an installation are tagged against an environment you could place the following code in your config:

[global_tags]
Env="<staging-circleci>"

Please see the InfluxDB documentation for default and advanced installation steps.

Any changes to the config will require a restart of the CircleCI application which will require downtime.