Start Building for Free
CircleCI.comAcademyBlogCommunitySupport

Troubleshooting and support

2 months ago1 min read
Server v4.1
Server Admin
On This Page

This document describes an initial set of troubleshooting steps to take if you are experiencing problems with your CircleCI server v4.x installation. If your issue is not addressed below, you can generate a support bundle or contact your CircleCI account team.

Generate support bundle

A support bundle is used by CircleCI engineers to diagnose and fix any issues you are experiencing. They are typically requested when you open a ticket.

To generate a support bundle, follow the steps below.

Prerequisites

  1. First, make sure circleci-server is deployed and you have access to the cluster/namespace through kubectl.

# To check if you have access to cluster/namespace
kubectl -n <namespace> get pods
# To install support-bundle plugin
kubectl krew install support-bundle

Generating support bundle

When ready, run the support bundle from the current directory and wait for it to finish.

kubectl support-bundle https://raw.githubusercontent.com/CircleCI-Public/server-scripts/main/support/support-bundle.yaml

Managing pods

Verify pod readiness and status

kubectl get pods -n <namespace>
NAME READY STATUS RESTARTS AGE
api-service-5c8f557548-zjbsj 1/1 Running 0 6d20h
audit-log-service-77c478f9d5-5dfzv 1/1 Running 0 6d20h
builds-service-v1-5f8568c7f5-62h8n 1/1 Running 0 6d20h
circleci-mongodb-0 1/1 Running 0 6d20h
circleci-nomad-0 1/1 Running 6 6d20h
…

To show only pods with a status besides Running, you can use the --field-selector option.

kubectl get pods --field-selector status.phase!=Running -n <namespace>
NAME READY STATUS RESTARTS AGE
nomad-server 0/1 Error 0 5d22h

Verify pod settings and status

To show detailed settings and status of pods, use the following command:

kubectl describe pods <pod-name> -n <namespace>

Get pod logs

To show logs of pods, use the following command:

kubectl logs <pod-name> -n <namespace>

Restart pods

To restart specific pods, the easiest way is remove the pod. Kubernetes automatically recreates the pod:

kubectl delete pod <pod-name> -n <namespace> --now

Utilizing REPL in an air-gapped environment

If you are in an air-gapped environment, you will first need to download the lein REPL dependencies, commit them, and upload it to your private Docker registry.

docker run -it clojure lein repl :connect 6005
docker commit $(docker ps -aq | head -1) <your-private-registry>/clojure

Now you are ready to access the REPL. From a system with access to the private Docker registry and kubectl access to the cluster, run the script below. Be sure to modify it to match your namespace if it is not circleci-server.

SVC_NAME=frontend
INET_ADDR="$(ip -4 route get 192.0.2.1 | grep -o 'src [0-9.]\{1,\}' | awk '{ print $2 }')"

# Start port-forwarding in background
kubectl port-forward "$(kubectl get po -l app="${SVC_NAME}" -n circleci-server -o jsonpath='{.items[0].metadata.name}')" --address "${INET_ADDR}" 6005 -n circleci-server &

# Start nREPL
sudo docker run --rm -it clojure lein repl :connect "${INET_ADDR}":6005

# End port-forwarding
kill $!

Disable concurrency limits

We recommend disabling concurrency limits for your CircleCI server installation. Disable concurrency limits by setting the following in your values.yaml:

distributor_dispatcher:
  ...
  disable_concurrency: true

You can check whether you are hitting these limits by checking the distributor-disptacher logs for a message that mentions concurrency limits.


Suggest an edit to this page

Make a contribution
Learn how to contribute