Additional considerations
Non-TLS Docker registry installations
When configuring your air-gapped Docker registry, it is recommended to use TLS certificates to encrypt traffic. If using a non-TLS, or self-signed installation, the following additional steps will need to be taken.
On machines that access the Docker registry using Docker, the Docker daemon config must be updated (located on Linux at /etc/docker/daemon.json).
The insecure-registries section must be added to the file (if it exists), or the file must be created with the following if it does not. Make sure to include the full hostname and port of your registry, but do not include the protocol (http:// or https://).
{
"insecure-registries":["docker.example.internal:5000"]
}This file will need to be configured on the following machines:
-
All Nomad nodes in the air-gapped environment
-
Potentially all K3s nodes in the air-gapped environment, if using Docker-backed K3s
In addition, on each K3s node, the following file must be configured at /etc/rancher/k3s/registries.yaml. Take note to include the protocol where referenced.
mirrors:
"docker.example.internal:5000":
endpoint:
- "http://docker.example.internal:5000"
configs:
"docker.example.internal:5000":
tls:
insecure_skip_verify: trueService type load balancers in K3s
CircleCI server makes use of Service Type: Load Balancer Kubernetes resources to listen to traffic on multiple ports. In order to function, the cluster needs to.
If using a K3s installation, MetalLB can be used to create a virtual load balancer on the K3s node, to allow ingress traffic to CircleCI server.
Once installed, the following steps need to be followed:
A ConfigMap resource needs to be created to create an address pool for MetalLB.
apiVersion: "v1"
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- <<k3s_internal_ip_range_start>>-<<k3s_internal_ip_range_end>>The address pool can be named something other than "default", but the annotations in values.yaml will need to be updated. If there is only one k3s node, the address range should have the same IP repeated (for example, 10.0.0.5-10.0.0.5).
Once this ConfigMap resource is applied to the cluster (kubectl apply -f metallb-configmap.yaml), the address pool name can be updated in the values.yaml for the Helm installation.
# Additional nginx annotations
nginx:
annotations:
# This example uses MetalLB as a k3s load balancer
metallb.universe.tf/allow-shared-ip: defaultAfter installing the Helm chart, the circleci-proxy service must be patched to use the internal IP of the desired k3s node to act as the load balancer (this IP should be in the range entered in the ConfigMap above). The example below uses the IP address 10.0.0.5.
kubectl patch svc circleci-proxy -p '{"spec": {"type": "LoadBalancer", "externalIPs":["10.0.0.5"]}}'Once complete, DNS records can be created for your server installation (server.internal.example.com) and (*.server.internal.example.com) for 10.0.0.5.
Importing trusted TLS certificates
When using a GitHub Enterprise instance with self-signed or custom certificate authority certificates, CircleCI server can be configured to trust these certificates using two methods, as described below.
| Values provided for either of these methods are supported for GitHub Enterprise certificates only. No other VCS application is supported at this time. |
Configuring the TLS imports value
In the values.yaml file, the tls.imports value can be configured to include a list of hostname:port combinations from which to retrieve and trust TLS certificates.
tls:
...
import:
- github.airgap.example.com:443For each hostname:port combination, CircleCI server will, during installation, retrieve the public certificate for the particular GitHub Enterprise instance and trust it to establish connections with that instance.
Configuring the TLS certificates array
Instead of providing a list of hostname:port combinations for certificates to import, the public certificate chains of the corresponding TLS certificates to trust can be provided in the values.yaml file, in the tls.certificates value, as a list of base64 encoded certificates strings.
tls:
...
certificates:
- <<base64-encoded-public-tls-certificate-chain>>