Troubleshoot

To enable usage of uncertified orbs, go to your organization’s settings page, and click the Security tab. Then, click yes to enable Allow Uncertified Orbs.

Command:

Error:

To resolve this error, run circleci config process on your configuration and then save that configuration to disk. You then should run circleci local execute against the processed configuration.

You may not be an organization owner/admin.

Organizations can only claim a single namespace. In order to claim a namespace for an organization the authenticating user must have owner/admin privileges within the organization.

If you do not have the required permission level you might see an error similar to below:

Read more in the Orb CLI permissions matrix.

If you see the following error message, check the steps below to remediate the issue.

If your scheduled pipeline is not running, verify the following things:

In the CircleCI application, check the individual job and workflow views for error messages. More often than not, the error is because of formatting errors in your .circleci/config.yml file.

See the YAML Introduction page for more details.

After checking your .circleci/config.yml for formatting errors, search for your issue in the CircleCI support center.

A job might end up being queued because of a concurrency limit being imposed due your organization’s plan. If your jobs are queuing often, you can consider upgrading your plan.

In order to keep the system stable for all CircleCI customers, we implement different soft concurrency limits on each of the Resource classes. If you are experiencing queuing on your jobs, it is possible you are hitting these limits. Contact CircleCI support to request raises on these limits.

If you are not seeing a project you would like to build, and it is not currently building on CircleCI, check your org in the top left corner of the CircleCI application. For instance, if the top left shows your user my-user, only projects belonging to my-user will be available under Projects. If you want to build the project your-org/project, you must switch your organization on the application’s organization switcher menu to your-org.

CircleCI currently supports pulling (and pushing with Docker Engine) Docker images from Docker Hub. For official images, you can pull by simply specifying the name of the image and a tag:

For public images on Docker Hub, you can pull the image by prefixing the account or team username:

It is best practice not to use the latest tag for specifying image versions. It is also best practice to use a specific version and tag, for example cimg/ruby:3.0.4-browsers, to pin down the image and prevent upstream changes to your containers when the underlying base distribution changes. For example, specifying only cimg/ruby:3.0.4 could result in unexpected changes from browsers to node. For more context, refer to Docker image best practices, and CircleCI image best practices.

You can set the timezone in Docker images with the TZ environment variable. A sample .circleci/config.yml with a defined TZ variable would look like the following:

In this example, the timezone is set for both the primary image and an additional mySQL image.

A full list of available timezone options is available on Wikipedia.

The task remains in the Preparing Environment step while the pod has a warning attached, noting that volume mounting fails due to a lack of disk space.

You should ensure there is sufficient disk space.

If the node a pod is hosted on runs out of memory, the task will fail with a failure step named Runner Instance Failure, and a message:

The pod will have a status of OOMKilled when viewed in Kubernetes with kubectl. You can use task pod configuration to control memory allocation for the job itself.

If the node is full it will have a node.kuberenetes.io/disk-pressure taint, which will prevent new task pods from being scheduled. If all valid nodes for the pod have the same taint, or other conditions that prevent scheduling, the task pod will sit in a pending state until an untainted valid node becomes available. This will show the job as stuck in the Preparing Environment step in the UI.

You need to scale your cluster more effectively to avoid this state.

When container runner is hosted on a separate node, the task will still look like it is running in the CircleCI UI until there is a timeout for it. kubectl will also still show the pod as running until the cluster’s liveness probe timeout is hit. The pod will then enter a terminating state that it will become wedged in. At this point the pod will need to be forcefully removed. If force is not used it may cause kubectl to hang:

If the entrypoint specified for the image is invalid, the task will fail with an error:

Container runner and CircleCI cloud set the entrypoint of the primary container in different ways. On cloud, the entrypoint of the primary container is ignored unless it is preserved using the com.circleci.preserve-entrypoint=true LABEL instruction (see: Adding an entrypoint). In contrast, container runner will always default to a shell (/bin/sh), or the entrypoint specified in the job configuration, if set.

Specify an entrypoint using the Adding an entrypoint documentation to mitigate this error. You can set the entrypoint explicitly as described in Using custom built Docker images.

If an image for a job uses a different architecture than the node it is deployed on, container runner will give an error:

The task pod will also show an error status. This will show as a failed job in the CircleCI UI with the error:

You should correct the underlying architecture used for nodes with jobs to match the architecture for images being used by jobs.

If the task pod for a resource class is misconfigured, the task will fail once claimed. In the UI the error will be in a Runner Instance Failure step with a message resembling:

No pod has been created in the Kubernetes cluster. You will need to correct the task pod configuration as described on the Container runner page.

Bash is required for custom images used in jobs executed with a container runner.

If you see the message "Oops, there was an issue with your infrastructure. Verify your self-hosted runner infrastructure is operating and try re-running the job. If the issue persist, see our troubleshooting guide" on the job’s page, or if there is no content in the task lifecycle step (as shown), you should consider the potential causes described below:

You can check the logs for any of the previous runs using the command kubectl logs -n <namespace> <full pod name> --previous `.

We suggest connecting to the pod using the command kubectl exec --stdin --tty -n circleci < full pod name > — /bin/sh and then running a ping test for an extended period of time. We also recommend checking the connection to the links on our FAQ includes a section about the connectivity required for CircleCI’s self-hosted runners.

Refer to the Kubernetes documentation for details of external resource monitoring tools.

The task pod may fail with an Task agent appears not running: /bin/sh: 1: kill: No such process error due a failing liveness probe.

The liveness probe on the task pod checks to see if the PID provided by task agent is running using kill -0 $PID. Task agent will output its PID to a file used by the liveness probe to confirm task agent is running. This probe may fail if the task agent process fails to start, no longer exists or takes longer to initiate than the liveness probe’s default timeouts. You may adjust the liveness probe defaults in the values.yaml for container runners’s Helm chart.

In the example below we have set the liveness probe to allow a 2.5 minute startup time before probing(initialDelaySecond) and 2.5 minutes of failures before the liveness probe fails (the probe will wait 30 seconds between each probe and allow for 5 failure responses before failing the probe) and the task pod is terminated.

In the event that the liveness probe fails or the task pod terminates, there is a prestop hook, which attempts to kill any existing task agent by its provided PID. This is to ensure there are no orphaned task agent processes. However, if the PID on file does not map to an existing process, this will not throw and error and will instead log PreStop hook: task agent appears never started or already stopped.

When a task pod fails, it is cleaned up by the container-agent almost immediately. However, sometimes you may want the pod to stick around longer so that you may review the logs and diagnose the failure. You can disable task pod deletion by adding the environment variable KUBE_DISABLE_POD_DELETION_ON_TASK_CLEANUP=true to your container-agent values.yaml.

example:

In some cases, you may need to update the execution permission for the launch-agent so it is executable by root. Try running the following two commands:

Cancel the job and rerun it. If your job is still not running, file a support ticket.

CircleCI’s machine runners support rerunning a job with SSH for debugging purposes. Instructions on using this feature can be found at Debugging with SSH.

If you experience that OIDC token environment variables ($CIRCLE_OIDC_TOKEN, $CIRCLE_OIDC_TOKEN_V2) are missing from jobs, a common cause can be that the default temporary directory (for example, /tmp) is not writable or is mounted as noexec. The system’s temporary directory needs to be writable and allow execution for the OIDC plugin to be downloaded and executed from it.

The time required for a Helm-based restore version to complete successfully is dependent on the specific configuration of the target component. For example, a large number of replicas will lead to a longer duration, which could cause a timeout. It is possible to specify a different timeout by adding the circleci.com/operation-timeout annotation to the Rollout or Deployment. The default value for this is 10 minutes. For steps see the Configure your Kubernetes components page.

Check whether the component has been undeployed. If there are currently no live versions for a component, the Restore Version button will not be visible for that component until at least one version has been deployed.

Check whether the release is a Rollback. If this is the case, then this is a known issue that will be solved in a future update to the CircleCI release agent.

Check if the error message is “invalid or missing project ID“. In this case the component is missing a valid Project ID.