Debug with SSH
Troubleshoot problems in a job using Rerun job with SSH. Using this feature you can inspect log files, running processes, and directory paths.
Introduction
CircleCI gives you the option to access all jobs via SSH. Read our blog post on debugging CI/CD pipelines with SSH.
When you log in with SSH, you are running an interactive login shell. You may be running the command on top of the directory where the command failed the first time, or you may be running the command from the directory one level up from where the command failed (for example, ~/project/ or ~/). Either way, you will not be initiating a clean run. You may wish to execute pwd or ls to ensure that you are in the correct directory.
Note that a default CircleCI pipeline executes steps in a non-interactive shell. There is a possibility that running steps using an interactive login may succeed, but fail in non-interactive mode.
Steps
| The steps described in this section apply to all CircleCI accounts. If you are using Bitbucket Cloud, or your account is integrated through the GitHub OAuth app, you can add an SSH key to your VCS account ( GitHub or Bitbucket) rather than in the CircleCI app under User Settings if you prefer. |
-
If you have not already done so, add an SSH key to your User Settings. This key will be used to connect to container or VM that is running your job.
-
Navigate to the job view for the job you want to rerun.
-
From the CircleCI web app dashboard, select your project from the dropdown menu
-
Expand the workflow that contains your job
-
Select the job name to enter the job view
-
-
To start a job with SSH enabled, select the Rerun job with SSH option from the Rerun dropdown menu.
The
Rerun job with SSHfeature is intended for debugging purposes. These jobs will be created inside of the same pipeline as the original job.
-
To see the connection details, expand the Enable SSH section in the job output where you will see the SSH command needed to connect. The details are displayed again in the Wait for SSH section at the end of the job.
After clicking Rerun Job With SSH you may notice that your job did not populate the Enable SSH step. This could indicate that you have not added an SSH key to your User Settings. 
-
Run the connection command given in the job output and follow the instructions to SSH into the running job (using the same SSH key that you use for GitHub or Bitbucket) to perform whatever troubleshooting you need to.
If you are using the Windows executor you will need to pass in the shell you want to use when using SSH. For example, to run powershell in your build you would run: ssh -p <remote_ip> -- powershell.exe.
The job virtual machine (VM) will remain available for an SSH connection for 10 minutes after the pipeline finishes running and then automatically shut down (or you can cancel it). After you SSH into the job, the connection will remain open for one hour for customers on the Free Plan, or two hours for all other customers.
If your job has parallel steps, CircleCI launches more than one VM to perform them. You will see more than one 'Enable SSH' and 'Wait for SSH' section in the job output.
If you rerun a workflow that contains a job which was previously re-run with SSH, the new workflow will be run with SSH enabled for that job, even after SSH capability has been disabled at the project level.
Debugging: permission denied (publickey)
If you run into permission issues trying to SSH to your job, try the following in the sections below.
Ensure authentication with GitHub/Bitbucket
A single command can be used to test that your keys are set up as expected.
For GitHub, run:
ssh git@github.comOr, for Bitbucket, run:
ssh -Tv git@bitbucket.orgYou should see both the following in the output:
$ Hi :username! You've successfully authenticated...$ logged in as :username.If you do not see output like above, you can try troubleshooting with the following:
Ensure authenticating as the correct user
The user that triggered the rerun is the user that is required to authenticate. You will need to make sure that your username is in the 'Enable SSH' step.
If you have multiple accounts, double-check that you are authenticated as the right one. In order to SSH into a CircleCI build, the username must be one which has access to the project being built.
If you are authenticating as the wrong user, you can try to resolve this by offering a different SSH key with ssh -i. See the next section for guidance on confirming which key is being offered.
Ensure the correct key is offered to CircleCI
If you have verified that you can authenticate as the correct user, but you are still getting "Permission denied" from CircleCI, you may be offering the wrong credentials to CircleCI.
Check which key is being offered that authenticates you, by running:
$ ssh -v git@github.comOr,
$ ssh -v git@bitbucket.comIn the output, look for a sequence like this:
debug1: Offering public key: /Users/me/.ssh/id_ed25519_github
<...>
debug1: Authentication succeeded (publickey).This sequence indicates that the key /Users/me/.ssh/id_ed25519_github is the one which your VCS accepted.
Next, run the SSH command for your CircleCI job, but add the -v flag. In the output, look for one or more lines like this:
debug1: Offering public key: ...Make sure that the key which your VCS accepted (in our example, /Users/me/.ssh/id_ed25519_github) was also offered to CircleCI.
If it was not offered, you can specify it via the -i command-line argument to SSH. For example:
$ ssh -i /Users/me/.ssh/id_ed25519_github -p 64784 54.224.97.243When you add the -v flag, you can also run multiple options in verbose mode to get more details, for example:
$ ssh -vv git@github.comor the maximum of
$ ssh -vvv git@github.com