Debugging with SSH Access#

Often, the best way to troubleshoot failures and bugs in your pipelines is to SSH into a job and inspect log files, running processes, and directory paths. Semaphore gives you the option to access all running jobs via SSH, restart your jobs in debug mode, or to start on-demand virtual machines to explore the CI/CD environment.

Before you begin, you'll need to install Semaphore CLI.

Restarting a job in debug mode#

Setting up a pipeline can be challenging if you are not familiar with the software stack installed in Semaphore virtual machines. Starting a debug session for your job is a great place to start exploring.

To start a debug session for your job, run:

sem debug job [job-id]

This will start a new interactive job based on the specifications of the last one, export the same environment variables, inject the same secrets, and connect to the same git commit.

Commands are not executed automatically in debug mode, instead they are stored in ~/commands.sh. This allows you to execute them step-by-step, and inspect the changes in the environment.

By default, the duration of the SSH session is limited to one hour. To run longer debug sessions, pass the duration flag to the previous command as shown below:

sem debug job [job-id] --duration 3h

By default, a debug session does not include the contents of the repository related to your Semaphore project. Run checkout in the debug session to clone your repository.

Error: project or job not found#

If you get this error and you're part of multiple organizations, check to see if you're currently in the right one by running sem context. If not, change context to the organization of the desired project or job by running sem context <id>.

Inspecting the state of a running job#

Often, the best way to inspect failures is to SSH into a running job, explore the running processes, inspect the environment variables, and take a peek at the log files.

Semaphore allows you to SSH into any running job with:

sem attach [job-id]

Use sem get jobs to see a list running jobs.

To find the root cause of a failed job, Semaphore allows you to restart your job in debug mode with the following command:

sem debug job [job-id]

Port forwarding your web server and UI debugging issues#

Sometimes SSH access to your CI/CD environment is not enough to fully explore the problem. For example, Selenium-based tests will fail if html elements are not visible on the screen when you are running them.

Semaphore allows you to forward ports to your local machine and explore the UI of your application from your browser. If your application is running on port 3000, you can port forward it to your local port 6000 with:

sem port-forward [job-id] 6000 3000

The http://localhost:6000 should now be accessible in your browser.

Note: Port-Forwarding only works for Virtual Machine-based CI/CD environments.

Stopping a debug session#

The debug session will automatically end once you exit the SSH session. To manually stop a debug session execute sem stop job [job-id] from your local machine. You can find the Job ID of your debug job using the sem get jobs command.

Debug sessions for self-hosted jobs#

If the job you are trying to debug ran in a self-hosted agent type, the Semaphore CLI will not automatically log into the machine where the Semaphore agent is running the debug job. Instead, it will just display the self-hosted agent name running that debug job and wait until the debug session can be stopped.

After logging into the machine running the debug job:

  • Make sure you are logged in as the same user the Semaphore agent is using. If you are using the agent-aws-stack, that user is semaphore.
  • In self-hosted environments, the Semaphore agent does not automatically load environment variables exposed to the job through shell profiles, so you need to source the environment variables exposed to the job. The file with all the environment variables is located at /tmp/.env-*.

See also#