Setting Up Docker Hub Pull Through Mirror

Overview

Starting November 1, 2020, Docker Hub will impose rate limits for anonymous pulls based on the originating IP. To avoid service disruption, it is recommended to authenticate Docker pulls with Docker Hub. You can authenticate using build config (see Using Docker Authenticated Pulls).

Alternatively, you can set up a Docker Hub pull through registry mirror pre-configured with Docker Hub account credentials. Using a pull through registry mirror is potentially simpler than making many build config modifications. It may also bring additional performance improvements since network roundtrips to Docker Hub are reduced.

These setup instructions are divided into two parts: Setting up a pull through registry mirror, and configuring Nomad clients to use the mirror.

Set up a pull through cache registry (proxy for Docker Hub)

  1. Set up a Docker Hub account which will be used by the pull through cache registry to access Docker Hub.

    Setting up a paid account can be an option to ease the rate limiting further. See also: https://www.docker.com/pricing
  2. Set up an independent Linux server that has Docker installed.

    We set up this registry as an independent server (i.e. outside of CircleCI boxes) to avoid load on this cache server affecting other CircleCI Server services.

    Assume that the IP address for this server is 192.0.2.1.

  3. Run the command below to start the cache server.

    Replace DOCKER_HUB_USERNAME and DOCKER_HUB_ACCESS_TOKEN with the username for the Docker Hub account and the access token you obtained through https://hub.docker.com/settings/security, respectively.

    sudo docker run -d -p 80:5000 --restart=always --name=through-cache -e REGISTRY_PROXY_REMOTEURL="https://registry-1.docker.io" -e REGISTRY_PROXY_USERNAME=DOCKER_HUB_USERNAME -e REGISTRY_PROXY_PASSWORD=DOCKER_HUB_ACCESS_TOKEN registry

  4. Finally, make sure that the TCP port 80 (i.e., HTTP) is open and accessible. For better security, we recommend you open the port only to Nomad clients.

    On AWS, for example, you will need to make sure that both Security Groups and the firewall at the OS level are configured correctly to accept connections from Nomad clients over HTTP.

Configure Nomad clients to use the pull through cache server (run this for each Nomad client)

  1. Run the command below to configure the registry-mirrors option for the Docker daemon.

    Replace 192.0.2.1 with the IP address of your pull through cache server.

    sudo bash -c 'cat <<< $(jq ".\"registry-mirrors\" = [\"http://192.0.2.1\"]" /etc/docker/daemon.json) > /etc/docker/daemon.json'

  2. Reload Docker daemon to apply the configuration.

    sudo systemctl restart docker.service

Now you should be able to see that Docker images from Docker Hub are downloaded through the cache server. As a side effect, you should be able to see that private images for the user, configured for the cache server can be downloaded without explicit authentications.

How to revert the setup?

Follow the steps below on each Nomad client.

  1. Remove registry-mirrors option in /etc/docker/daemon.json. When you edit the file, make sure that /etc/docker/daemon.json is still a valid JSON file.

  2. Run sudo systemctl restart docker.service to apply the change.

Resources