This document is intended for system administrators of self-hosted installations of CircleCI Server.
This section outlines how to set up and customize VM service for your CircleCI installation, to be used for
machine executor (Linux and Winows images) and remote Docker jobs.
|This is only available for installations on AWS, please contact your CircleCI account representative to request this for a static installation.|
|Any changes to management console settings require downtime while the CircleCI application restarts.|
To configure VM service, it is best practice to select the AWS EC2 option in the Management Console Settings. This will allow CircleCI to run remote Docker and
machine executor jobs using dedicated EC2 instances.
You can provide custom Amazon Machine Image (AMIs) for VM service, as described in the sections below. If you do not provide any custom images, all
machine executor and remote Docker jobs will be run on instances built with one of our default AMIs, which have Ubuntu 16.04, Docker version 18.06.3 and a selection of common languages, tools, and frameworks. See the
picard-vm-image branch of our image-builder repository for details. To run Windows jobs you must supply a Windows AMI, without this Windows jobs will fail to run.
Customizing the VM service images for your installation of CircleCI will allow you to specify versions of Docker and Docker Compose, as well as install any additional dependencies that may be part of your CI/CD pipeline. You can create separate AMIs for jobs that use remote Docker or the
machine executor, and for
machine you can specify separate AMIs for Linux and Windows. If you choose not to use a custom Linux image, you will likely need to run additional install and update steps on every commit as part of your
From v2.18, you can either provide a single custom Linux AMI to use for both
machine and remote Docker jobs using just the field marked '1' below, or, by providing a second custom AMI in the field marked '2', you can use different settings for each
To build custom VM service images:
Clone our image builder repo: https://github.com/circleci/image-builder/tree/picard-vm-image
aws-vm.jsonin your editor and fill in the required groups. An access key and secret key are required to upload. Handle the key and secret process according to your requirements, but consider restricting the
ami_groupsto only within your organization
packer build aws-vm.json
Refer to https://packer.io/docs/builders/amazon-ebs.html#ami_groups for more information and see https://github.com/circleci/image-builder/blob/picard-vm-image/provision.sh for details about settings.
You will need to associate the
circleci user with the image you want to use as shown in this example.
To enable the use of these custom images for your installation, you will need to provide the individual AMI IDs in the Settings accessible from the Management console (for example,
Introduced in CircleCI Server v2.18.3
Creating a Windows image and specifying it under the VM Service settings lets your users run jobs on dedicated Windows VMs. To create your Windows image run through the steps listed in our image builder repo, then copy the generated AMI ID and paste into the Custom Windows VM AMI field in your Management Console settings, under VM Provider (for example,
|Windows images are built on CircleCI, so we suggest you run through this process once your installation is up and running. Alternatively you can use any other CircleCI account – including on our managed Cloud service – to generate the image.|
There are two fields for defining the instance types you wish to use. The first is for the default instance type, and the second is to set the instance type to use when a Job specifies the
large resource class.
Remote Docker and
machine executor instances are spun up on demand. It is also possible to preallocate instances to remain up and running, ready for remote Docker and
machine jobs to be run (see the last two fields in figure 9).
If Docker Layer Caching (DLC) is to be used, VM Service instances need to be spun up on-demand. To ensure this can happen, either ensure any preallocated instances are in use, or set both remote Docker and
|When using preallocated instances be aware that a cron job is scheduled to cycle through these instances once per day to ensure they don’t end up in an unworkable state.|
Jobs run using the remote Docker Environment, or the
machine executor are scheduled and dispatched by the Nomad server to your Nomad clients and passed on to remote Docker or
machine from there. This means jobs run on remote Docker and the
machine executor can be monitored in the usual way, using the Nomad CLI. See our Introduction to Nomad Cluster Operation for more about Nomad commands and terminology.
|A cron job in scheduled to cycle all default and preallocated instanced at least once per day to ensure instanced don’t end up in a dead/bad state.|
By default, private IP addresses are used to communicate with VM service instances. If you need to grant wider access, for example, to allow developers SSH access, this can be set using the checkbox in the VM Provider Advanced Settings.