Frequently asked questions#

Issues we stumble upon regularly, across all parts of Semaphore 2.0

Troubleshooting#

How to solve "Fail: Could not parse object" during bundle install?

If the bundle install output looks like this:

Fetching gem metadata from http://rubygems.org/.......
Fetching gem metadata from http://rubygems.org/..
Updating git://github.com/some/gem.git
fatal: Could not parse object 'a84dd3407eaf064064cca9650c354cb163384467'.
Git error: command <code>git reset --hard a84dd3407eaf064064cca9650c354cb163384467</code> in directory /home/runner/somehash/vendor/bundle/ruby/1.9.1/bundler/gems/gem-a84dd3407eaf has failed.
If this error persists you could try removing the cache directory '/home/runner/somehash/vendor/bundle/ruby/1.9.1/cache/bundler/git/gem-cbe2ee16ed53098079007f06cd77ed0890d0d752'
This problem occurs when there have been changes like force-pushes to a git repo which is referenced in a Gemfile. You can solve it by following these steps:

- Comment that gem line in the Gemfile
- Run bundle install
- Uncomment the gem line in the Gemfile
- Run bundle install again

The Gemfile.lock will now reference a valid git revision.

How to change the timezone?

The default timezone in the virtual machine is set to UTC. The timezone can be changed in 2 ways: - Assign a different value to TZ environment variable:

export TZ=Europe/Belgrade
- Create a symlink in /etc/localtime to one of the available timezones:
sudo ln -sf /usr/share/zoneinfo/Europe/Belgrade /etc/localtime

How to build a project with git submodules?

- Add the following commands as a prologue:

git submodule init
git submodule update
- Add the following command as an epilogue:
git submodule deinit --force .
Make sure that Semaphore has permissions to clone your submodules repository. In our private dependencies page you can find more information about setting permissions for private repositories.

How to use a self-signed certificate with private Docker registry?

If you have a private Docker registry that uses a self-signed SSL certificate and pulling the Docker images does not work. The solution is to:

- Add a self-signed certificate as a secret on Semaphore
- Save it under the name of domain.crt
- Add the following command to your pipeline:

sudo mv $SEMAPHORE_GIT_DIR/domain.crt /etc/docker/certs.d/myregistrydomain.com:5000/ca.crt

This will allow the connection to a private remote registry using the self-signed certificate.

Is it possible to attach to an ongoing SSH session?

It's possible to use sem attach to an ongoing SSH session but you'd need to attach to the job ID of the SSH session. To get the job ID you can use sem get jobs to get the list of all running jobs.

How to change the Postgres locale?

Semaphore uses sem-service to provide different versions of databases. The sem-service tool uses Docker containers instead of traditional Linux services. So, the traditional way of changing locales no longer works since it does not affect containers.

The following recipe provides an altered version of the container to sem-service. The database should be available as before, without modifying your application in any way:

1. Create a Dockerfile with the following:

FROM postgres:9.6
RUN localedef -i pt_BR -c -f UTF-8 -A /usr/share/locale/locale.alias pt_BR.UTF-8
ENV LANG pt_BR.UTF-8
2. Rebuild the Postgres image using the locale:
docker build - -t postgres:[lang] < Dockerfile
3. Start the newly created image:
docker run --rm --net host -d -e POSTGRES_PASSWORD=semaphore --name postgres -v /var/run/postgresql:/var/run/postgresql postgres:[lang]

How can I remove Semaphore Status checks on pull requests?

You can disable Semaphore as a required status check through the repository settings page in your GitHub account.

How to troubleshoot a stalling job?

The most common reason for stalled builds is a process that refuses to shut down properly. Either a debug statement or a cleanup procedure in the catch procedure. Reproducing this can be hard sometimes. These are the steps we recommend:

1. Start a build on a branch and let it get stale.
2. Attach to a running job: sem attach [job-id].
3. Now, you should be in the instance of the job's virtual machine.

In the running instance, you can:

- List the running processes with ps aux or top. Is there any suspicious process running?
- Run a strace on the running process: sudo strace -p to see the last kernel instruction that it is waiting for. For example, select(1, ... can mean the process is waiting for user's input.
- Look into the system metrics at /tmp/system-metrics. This tracks memory and disk usage. Lack of disk space or free memory can introduce unwanted stalling into jobs.
- Look into the Agent logs at /tmp/agent_logs. The logs could indicate waiting for some conditions.
- Look into the Job logs at /tmp/job_logs.json. The logs could also indicate waiting for some conditions.
- Check the syslog as it can be also a valuable source of information: tail /var/log/syslog. It can indicate 'Out of memory' conditions.

While this issue is ongoing, you might consider using a shorter execution_time_limit in your pipelines. This will prevent stale builds to run for a full hour, and fail sooner.

Why is my test suite failing if all the tests pass?

This usually happens because code coverage tools, for instance simplecov, can be set to fail the test suite if a minimum coverage is not achieved.
Besides the above, some dependencies can configure an at_exit hook and will change the final exit code of the suite.

How to solve the "fatal: expected flush after ref listing" error?

If a commands fails with this:

error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: expected flush after ref listing
It means the communication between Semaphore and Github was interrupted. As a workaround, you may add retry to the failed command:
retry -t 5 <command>
You may find more information about the retry tool here.

Jobs & Workflows#

Why my jobs don't start?

You might be hitting the quota limitation. Check your organization's quota in the Activity Monitor by clicking on the initial of your organization in the top right corner of the page. More information about quota and how to ask for an increase here.

You may also run sem get jobs to display all running jobs so you may confirm how much quota is being used. More information about sem get here.

Why does my job break after changing the shell configuration?

Adding any of the following to your shell is not supported and will cause the jobs to immediately fail:

set -e
set -o pipefail
set -euxo pipefail

This also applies when sourcing a script that contains the previous settings:

source ~/my_script
. ~/my_script

Why are my workflows not running in parallel?

When pushing several commits into the same branch, Semaphore won't run parallel workflows. This means that pushing several times into a branch won't create parallel workflows, instead, Semaphore assigns the new workflows to the queue and run one workflow at a time. However, it's possible to push commits to different branches and they will be run in parallel.

The only way to push several commits to a single branch and not wait for the workflows to finish one by one is to enable the auto_cancel feature.

Billing#

Why are you still charging my old credit card when I added a new default credit card?

If you’ve added a new credit card to the subscription, but the old one is still being charged, it means that the new credit card wasn't marked for usage. Here’s how to do that:

1. Click on the initial of your organization in the top right corner of the page,
2. In the dropdown menu, choose Plans & Billing,
3. Next to the Payment details, click on Credit card info,
4. Go to Subscription tab
5. Click on Manage
6. Go to Update Payment Method
7. Click on the Use this button next to the credit card you'd like to use

After that, you can also remove the old credit card if you don't need it anymore.

Why can't I remove the old credit card after adding a new one?

If you run into this situation, it means that the old credit card is still in use. In order to mark the new credit card for usage, you can:

1. Click on the initial of your organization in the top right corner of the page,
2. In the dropdown menu, choose Plans & Billing
3. Next to the Payment details, click on Credit card info,
4. Go to Subscription tab
5. Click on Manage
6. Go to Update Payment Method
7. Click on the Use this button next to the credit card you'd like to use

After that, you’ll be able to remove the old credit card.

Account management#

How can I add repositories that belong to my GitHub organization?

In order to be able to do that, the access for Semaphore 2.0 needs to be granted within your GitHub organization. You can grant the access here. If it has already been granted, there should be a green checkmark next to the name of your organization.

If not, you should either grant access or request it from the organization's owner.