Search Results for ""

Monitoring Your Installation

This section includes information on metrics for monitoring your CircleCI Server installation.

System Monitoring

To enable system and Docker metrics forwarding to either AWS Cloudwatch or Datadog, navigate to your CircleCI Management Console, select Settings from the menu bar and scroll down to enable the provider of your choice (your-circleci-hostname.com:8800/settings#cloudwatch_metrics).

VM Service and Docker Metrics

VM Host and Docker services metrics are forwarded via Telegraf, a plugin-driven server agent for collecting and reporting metrics.

Following are the enabled metrics:

Nomad Job Metrics

Nomad job metrics are enabled and emitted by the Nomad Server agent. Five types of metrics are reported:

Metric Description

circle.nomad.server_agent.poll_failure

Returns 1 if the last poll of the Nomad agent failed, otherwise it returns 0.

circle.nomad.server_agent.jobs.pending

Returns the total number of pending jobs across the cluster.

circle.nomad.server_agent.jobs.running

Returns the total number of running jobs across the cluster.

circle.nomad.server_agent.jobs.complete

Returns the total number of complete jobs across the cluster.

circle.nomad.server_agent.jobs.dead

Returns the total number of dead jobs across the cluster.

When the Nomad metrics container is running normally, no output will be written to standard output or standard error. Failures will elicit a message to standard error.

Supported Platforms

We have two built-in platforms for metrics and monitoring: AWS CloudWatch and DataDog. The sections below detail enabling and configuring each in turn.

AWS CloudWatch

To enable AWS CloudWatch complete the following:

  1. Navigate to the settings page within your Management Console. You can use the following URL, substituting your CircleCI URL: your-circleci-hostname.com:8800/settings#cloudwatch_metrics.

  2. Check Enabled under AWS CloudWatch Metrics to begin configuration.

    AWS CloudWatch
    Figure 1. Enable Cloudwatch

AWS CloudWatch Configuration

There are two options for configuration:

  • Use the IAM Instance Profile of the services box and configure your custom region and namespace.

    Configuration IAM
    Figure 2. CloudWatch Region and Namespace
  • Alternatively, you may use your AWS Access Key and Secret Key along with your custom region and namespace.

    Configuration Alt
    Figure 3. Access Key and Secret Key

After saving you can verify that metrics are forwarding by going to your AWS CloudWatch console.

DataDog

To enable Datadog complete the following:

  1. Navigate your Management Console Settings. You can use the following URL, substituting your CircleCI hostname: your-circleci-hostname.com:8800/settings#datadog_metrics

  2. Check Enabled under Datadog Metrics to begin configuration.

    Enable DataDog
    Figure 4. Enable Datadog Metrics
  3. Enter your DataDog API Key. You can verify that metrics are forwarding by going to your DataDog metrics summary.

    DataDog API Key
    Figure 5. Enter Datadog API key

Custom Metrics

Custom Metrics using a Telegraf configuration file may be configured in addition to the predefined CloudWatch and Datadog metrics described above. Telegraph can also be used instead of CloudWatch and Datadog for more fine grained control.

Custom
Figure 6. Custom Metrics

Configuring Custom Metrics

Configuration options are based on Telegraf’s documented output plugins. See their documentation here. For example, if you would like to use the InfluxDB Output Plugin you would need to follow these steps:

  1. SSH into the Servics Machine

  2. cd /etc/circleconfig/telegraf/influxdb.conf

  3. Adding the desired outputs, for example:

    [[output.influxdb]]
      url = "http://52.67.66.155:8086"
      database = "testdb"
  4. Run docker restart telegraf to restart the container to load or reload any changes.

You may check the logs by running docker logs -f telegraf to confirm your output provider (e.g. influx) is listed in the configured outputs. Additionally, if you would like to ensure that all metrics in an installation are tagged against an environment you could place the following code in your config:

[global_tags]
Env="<staging-circleci>"

Please see the InfluxDB documentation for default and advanced installation steps.

Any changes to the config will require a restart of the system.