Troubleshooting and support

Server v4.3

Server Admin

Generate support bundle

A support bundle is used by CircleCI engineers to diagnose and fix any issues you are experiencing. They are typically requested when you open a ticket.

To generate a support bundle, follow the steps below.

Prerequisites

First, make sure circleci-server is deployed and you have access to the cluster/namespace through kubectl.

# To check if you have access to cluster/namespace
kubectl -n <namespace> get pods

Next, install Krew.
Install support-bundle (kubectl plugin) to your local development machine.

# To install support-bundle plugin
kubectl krew install support-bundle

Generating support bundle

When ready, run the support bundle from the current directory and wait for it to finish.

kubectl support-bundle https://raw.githubusercontent.com/CircleCI-Public/server-scripts/main/support/support-bundle.yaml

Managing pods

Verify pod readiness and status

Check the READY column as well as STATUS. Even if the STATUS is Running, pods are not ready to serve user requests. Some pods may take some time to become ready.

kubectl get pods -n <namespace>
NAME READY STATUS RESTARTS AGE
api-service-5c8f557548-zjbsj 1/1 Running 0 6d20h
audit-log-service-77c478f9d5-5dfzv 1/1 Running 0 6d20h
builds-service-v1-5f8568c7f5-62h8n 1/1 Running 0 6d20h
circleci-mongodb-0 1/1 Running 0 6d20h
circleci-nomad-0 1/1 Running 6 6d20h
…

To show only pods with a status besides Running, you can use the --field-selector option.

kubectl get pods --field-selector status.phase!=Running -n <namespace>
NAME READY STATUS RESTARTS AGE
nomad-server 0/1 Error 0 5d22h

Verify pod settings and status

To show detailed settings and status of pods, use the following command:

kubectl describe pods <pod-name> -n <namespace>

Get pod logs

To show logs of pods, use the following command:

kubectl logs <pod-name> -n <namespace>

Restart pods

To restart specific pods, the easiest way is remove the pod. Kubernetes automatically recreates the pod:

kubectl delete pod <pod-name> -n <namespace> --now

Utilizing REPL in an air-gapped environment

If you are in an air-gapped environment, you will first need to download the lein REPL dependencies, commit them, and upload it to your private Docker registry.

docker run -it clojure lein repl :start
docker commit $(docker ps -aq | head -1) <your-private-registry>/clojure

Now you are ready to access the REPL. From a system with access to the private Docker registry and kubectl access to the cluster, run the script below. Be sure to modify it to match your namespace if it is not circleci-server.

SVC_NAME=frontend
INET_ADDR="$(ip -4 route get 192.0.2.1 | grep -o 'src [0-9.]\{1,\}' | awk '{ print $2 }')"

# Start port-forwarding in background
kubectl port-forward "$(kubectl get po -l app="${SVC_NAME}" -n circleci-server -o jsonpath='{.items[0].metadata.name}')" --address "${INET_ADDR}" 6005 -n circleci-server &

# Start nREPL
sudo docker run --rm -it clojure lein repl :connect "${INET_ADDR}":6005

# End port-forwarding
kill $!