Troubleshooting and support
This document describes an initial set of troubleshooting steps to take if you are experiencing problems with your CircleCI server v4.x installation. If your issue is not addressed below, you can generate a support bundle or contact your CircleCI account team.
Generate support bundle
A support bundle is used by CircleCI engineers to diagnose and fix any issues you are experiencing. They are typically requested when you open a ticket.
To generate a support bundle, follow the steps below.
Prerequisites
-
First, make sure circleci-server is deployed and you have access to the cluster/namespace through kubectl.
# To check if you have access to cluster/namespace
kubectl -n <namespace> get pods-
Next, install Krew.
-
Install support-bundle (kubectl plugin) to your local development machine.
# To install support-bundle plugin
kubectl krew install support-bundleGenerating support bundle
When ready, run the support bundle from the current directory and wait for it to finish.
kubectl support-bundle https://raw.githubusercontent.com/CircleCI-Public/server-scripts/main/support/support-bundle.yamlManaging pods
Verify pod readiness and status
Check the READY column as well as STATUS. Even if the STATUS is Running, pods are not ready to serve user requests. Some pods may take some time to become ready. |
kubectl get pods -n <namespace>
NAME READY STATUS RESTARTS AGE
api-service-5c8f557548-zjbsj 1/1 Running 0 6d20h
audit-log-service-77c478f9d5-5dfzv 1/1 Running 0 6d20h
builds-service-v1-5f8568c7f5-62h8n 1/1 Running 0 6d20h
circleci-mongodb-0 1/1 Running 0 6d20h
circleci-nomad-0 1/1 Running 6 6d20h
…To show only pods with a status besides Running, you can use the --field-selector option.
kubectl get pods --field-selector status.phase!=Running -n <namespace>
NAME READY STATUS RESTARTS AGE
nomad-server 0/1 Error 0 5d22hVerify pod settings and status
To show detailed settings and status of pods, use the following command:
kubectl describe pods <pod-name> -n <namespace>Get pod logs
To show logs of pods, use the following command:
kubectl logs <pod-name> -n <namespace>Restart pods
To restart specific pods, the easiest way is remove the pod. Kubernetes automatically recreates the pod:
kubectl delete pod <pod-name> -n <namespace> --nowUtilizing REPL in an air-gapped environment
If you are in an air-gapped environment, you will first need to download the lein REPL dependencies, commit them, and upload it to your private Docker registry.
docker run -it clojure lein repl :start
docker commit $(docker ps -aq | head -1) <your-private-registry>/clojureNow you are ready to access the REPL. From a system with access to the private Docker registry and kubectl access to the cluster, run the script below. Be sure to modify it to match your namespace if it is not circleci-server.
SVC_NAME=frontend
INET_ADDR="$(ip -4 route get 192.0.2.1 | grep -o 'src [0-9.]\{1,\}' | awk '{ print $2 }')"
# Start port-forwarding in background
kubectl port-forward "$(kubectl get po -l app="${SVC_NAME}" -n circleci-server -o jsonpath='{.items[0].metadata.name}')" --address "${INET_ADDR}" 6005 -n circleci-server &
# Start nREPL
sudo docker run --rm -it clojure lein repl :connect "${INET_ADDR}":6005
# End port-forwarding
kill $!Disable concurrency limits
We recommend disabling concurrency limits for your CircleCI server installation. Disable concurrency limits by setting the following in your values.yaml:
Concurrency limits are disabled by default from server v4.1.5. From server v4.1.6, the distributor_dispatcher.disable_concurrency flag has been removed and concurrency limits are permanently disabled. |
distributor_dispatcher:
...
disable_concurrency: trueYou can check whether you are hitting these limits by checking the distributor-disptacher logs for a message that mentions concurrency limits.