FAQ: Frequently asked questions#

Issues we stumble upon regularly, in all areas of Semaphore 2.0

How can I solve "Fail: Could not parse object" during bundle install?#

Click for details

If the bundle install output looks like this:

Fetching gem metadata from http://rubygems.org/.......
Fetching gem metadata from http://rubygems.org/..
Updating git://github.com/some/gem.git
fatal: Could not parse object 'a84dd3407eaf064064cca9650c354cb163384467'.
Git error: command <code>git reset --hard a84dd3407eaf064064cca9650c354cb163384467</code> in directory /home/runner/somehash/vendor/bundle/ruby/1.9.1/bundler/gems/gem-a84dd3407eaf has failed.
If this error persists you should try removing the cache directory '/home/runner/somehash/vendor/bundle/ruby/1.9.1/cache/bundler/git/gem-cbe2ee16ed53098079007f06cd77ed0890d0d752'
This problem occurs when there have been changes like force-pushes to a git repo which is referenced in a Gemfile. You can solve it by following these steps:

- Comment the gem line in the Gemfile
- Run bundle install
- Uncomment the gem line in the Gemfile
- Run bundle install again

The Gemfile.lock will now reference a valid git revision.

I get "Revision: COMMIT_SHA not found .... Exiting" error. What is the cause?#

Click for details

The reason for that error are changes to repository tree structure in git VCS, which become effective in remote when your pipeline is still running. It usually happens when you modify or remove commits (for example, with git rebase or git commit --amend commands) and then push with --force flag shortly after the previous push.

If you want to avoid executing previous pipelines after new pushes, check out how to set up auto-cancel strategy. If pipeline hasn't started after you pushed changes, you may also want to check out if GitHub delivered webhooks to Semaphore.

How can I change timezone?#

Click for details

The default timezone in the virtual machine is set to UTC. The timezone can be changed in 2 ways: - Assign a different value to TZ environment variable:

export TZ=Europe/Belgrade
- Create a symlink in /etc/localtime to one of the available timezones:
sudo ln -sf /usr/share/zoneinfo/Europe/Belgrade /etc/localtime

How can I build a project with git submodules?#

Click for details

- Add the following commands as a prologue:

git submodule init
git submodule update
- Add the following command as an epilogue:
git submodule deinit --force .
Make sure that Semaphore has permissions to clone your submodules repository. You can find more information about setting permissions for private repositories on our private dependencies page.

How can I use a self-signed certificate with a private Docker registry?#

Click for details

If you have a private Docker registry that uses a self-signed SSL certificate and pulling the Docker images does not work, the solution is to:

- Add a self-signed certificate as a secret on Semaphore
- Save it under the name domain.crt
- Add the following command to your pipeline:

sudo mv $SEMAPHORE_GIT_DIR/domain.crt /etc/docker/certs.d/myregistrydomain.com:5000/ca.crt

This will allow a connection to a private remote registry using the self-signed certificate.

Is it possible to attach to an ongoing SSH session?#

Click for details

It's possible to use sem attach to an ongoing SSH session, but you need to attach to the job ID of the SSH session. To get the job ID, you can use sem get jobs to get a list of all running jobs.

How can I change the Postgres locale?#

Click for details

Semaphore uses sem-service to provide different versions of databases. The sem-service tool uses Docker containers instead of traditional Linux services. So, the traditional way of changing locales no longer works, as it does not affect containers.

The following recipe provides an altered version of the container to sem-service. The database should be available as before, without modifying your application in any way:

1. Create a Dockerfile with the following:

FROM postgres:9.6
RUN localedef -i pt_BR -c -f UTF-8 -A /usr/share/locale/locale.alias pt_BR.UTF-8
ENV LANG pt_BR.UTF-8
2. Rebuild the Postgres image using the locale:
docker build - -t postgres:[lang] < Dockerfile
3. Start the newly-created image:
docker run --rm --net host -d -e POSTGRES_PASSWORD=semaphore --name postgres -v /var/run/postgresql:/var/run/postgresql postgres:[lang]

How can I remove Semaphore Status checks on pull requests?#

Click for details

You can disable Semaphore as a required status check on the repository settings page of your GitHub account.

How can I troubleshoot a stalling job?#

Click for details

The most common reason for stalled builds is a process that refuses to shut down properly. This is most likely a debug statement or a cleanup procedure in the catch procedure. Reproducing this can be hard sometimes. These are the steps we recommend:

1. Start a build on a branch and let it become stale.
2. Attach the following to a running job: sem attach [job-id].
3. Now, you should be in the instance of the job's virtual machine.

In the running instance, you can:

- List the running processes with ps aux or top. Are there any suspicious processes running?
- Run a strace on the running process: sudo strace -p to see the last kernel instruction that it is waiting for. For example, select(1, ... can mean that the process is waiting for user input.
- Check the system metrics at /tmp/system-metrics. This tracks memory and disk usage. Lack of disk space or free memory can result in stalling.
- Check the Agent logs at /tmp/agent_logs. The logs could indicate that an agent is waiting for certain conditions.
- Check the Job logs at /tmp/job_logs.json. The logs could indicate that a job is waiting for certain conditions.
- Check the syslog as it can be also a valuable source of information: tail /var/log/syslog. It can indicate 'Out of memory' conditions.

While an issue is ongoing, you might consider using a shorter execution_time_limit in your pipelines. This will prevent stale builds from running for a full hour.

Why is my test suite failing if all the tests passed?#

Click for details

This usually happens because of code coverage tools, e.g. simplecov, which can be set to fail the test suite if a minimum coverage level is not achieved.
Furthermore, some dependencies can configure an at_exit hook that will change the final exit code of the suite.

How can I avoid a "fatal: expected flush after ref listing" error?#

Click for details

If a commands fails with this:

error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: expected flush after ref listing
It means that communication between Semaphore and Github was interrupted. As a workaround, you can add retry to the failed command:
retry -t 5 <command>
You can find more information about the retry tool here.

Why are tests passing locally but not on Semaphore?#

Click for details

The main reason for this behavior is differences in the stacks. As a first step, ensure that the same versions of languages, services, tools, and frameworks such as Selenium, browser drivers, Capybara, Cypress are used both locally and in the CI environment. To achieve this, use sem-service, sem-version, and the operating systems' package manager. Environment variables can also lead to unexpected behaviors, for instance, Semaphore 2.0 will set CI=true by default.

If you are using Docker containers when performing tests, it's possible that, while the command itself runs instantly, the process will not be completely started, leading to certain endpoints not being available. Using a minimum sleep 10 can help in this scenario. Cypress has a wait-on module that provides similar functionality.

Finally, when tests have different outcomes between reruns, using the same commit or in an SSH session, then this is a case of flaky tests. The following articles should be of help:
https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests
https://semaphoreci.com/blog/2017/08/03/tips-on-treating-flakiness-in-your-test-suite.html

How can I insert multiline commands?#

Click for details

You can divide a command into several lines by writing them in the folded style > and by stripping the line break in the yaml file -. To do this, we can start the command with a line containing only >- and write the command in multiple lines below it:

commands:
  - >-
    if [ "foo" = "foo" ];
    then commands...;
    else commands...;
    fi;

Block Style Indicator: The block style indicates how new lines inside the block should behave. If you want to keep each line as a new line, use the literal style, indicated by a pipe |. If you want them to be replaced by spaces instead, use the folded style, indicated by a right angle bracket >.

Block Chomping Indicator: The chomping indicator controls what should happen with new lines at the end of the string. The default, clip, puts a single new line at the end of the string. To remove all new lines, strip them by putting a minus sign - after the style indicator. Both clip and strip ignore how many new lines are actually at the end of the block; to keep them all, put a plus sign + after the style indicator.

Why won't my jobs start?#

Click for details

You might be hitting the quota limitation. Check your organization's quota in the Activity Monitor by clicking on your organization's initials in the top right corner of the page. Find more information about the quota and how to ask for an increase here.

You can also run sem get jobs to display all running jobs to confirm how much of the quota is being used. More information about sem get can be found here.

Why did my job break after changing the shell configuration?#

Click for details

Adding any of the following to your shell is not supported and will cause the jobs to immediately fail:

set -e
set -o pipefail
set -euxo pipefail

This also applies when sourcing a script that contains the previous settings:

source ~/my_script
. ~/my_script

Why does my job fail when I specify "exit 0" in commands?#

Click for details

Using `exit` command closes the PTY and causes undefined behaviour for the command. As a workaround, you can use `return 0` or `echo "exit 0" | sh` instead.

Why are my workflows not running in parallel?#

Click for details

When pushing several commits into the same branch, Semaphore won't run parallel workflows. This means that pushing multiple times into a branch won't create parallel workflows, rather Semaphore will assign the new workflows into the queue and run them one at a time. However, it's possible to push commits to different branches and they will be run in parallel.

The only way to push several commits to a single branch and not wait for the workflows to finish one by one is to enable the auto_cancel feature.

How can I redeliver webhooks from Github to Semaphore?#

Click for details

This is not the most common problem, but occasionally Semaphore does not receive a webhook from Github for some reason. This results in a workflow not being triggered. You can redeliver the webhook, however, and this should trigger the workflow. These are the steps to redeliver webhooks from Github:

1. Go to your repository on GitHub
2. Click Settings
3. Click Webhooks
4. Click Edit for the webhook you want to redeliver
5. Scroll down to Recent Deliveries and search for the one that failed
6. Click the ... symbol, then click Redeliver

Why did my workflow stop without explanation?#

Click for details

The workflow might have been stopped by the auto_cancel feature. There are two auto-cancel strategies: running and queued.

The running strategy stops all pipelines in the queue as soon as a new one appears.

The queued strategy will only cancel pipelines that are waiting in the queue and have not yet started.

What can I use to split my parallel tests?#

Click for details

We recommend using semaphore_test_boosters gem. This gem spreads tests across parallel jobs based on a configuration file or uniform file distribution (default behavior). Execution time-based distribution is not supported yet.

Other options are also supported, e.g. Knapsack (both free and pro versions).

Knapsack Rspec example:

jobs:
  - name: Knapsack RSpec
    parallelism: 5
    commands:
      - CI_NODE_TOTAL=$SEMAPHORE_JOB_COUNT CI_NODE_INDEX=$((SEMAPHORE_JOB_INDEX-1)) bundle exec rake 'knapsack:rspec'

Knapsack Pro Rspec example:

jobs:
  - name: Knapsack Pro RSpec
    parallelism: 5
    commands:
      - bundle exec rake 'knapsack_pro:queue:rspec'


You can find a more detailed example in the official documentation.

Why aren't workflows triggering on pull requests?#

Click for details

Make sure to enable pull requests in the project Settings.

If the configuration is correct, check if the pull request can be merged or if there are conflicts.
Semaphore uses the merge commit to run the workflows, so there will be no merge commit if there is a conflict in the pull request.

How can I use different machines in the same pipeline?#

Click for details

In certain scenarios, it's advantageous to use different machine types for different jobs in a pipeline. For instance, some operations require fewer resources and it would be wasteful to use a bigger machine or a test suite that has to run in multiple environments.

Semaphore 2.0 provides the agent in task feature that allows mixing and matching of various machine types and even Docker-based CI/CD:

version: v1.0
name: Tests
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: 'MacOS tests'
    task:
      agent:
        machine:
          type: a1-standard-4
          os_image: macos-xcode14
      jobs:
        - name: 'MacOS test'
          commands:
            - echo 'Testing MacOS'

  - name: Docker tests
    task:
      agent:
        machine:
          type: e1-standard-4
        containers:
          - name: main
            image: 'registry.semaphoreci.com/ruby:2.6'
      jobs:
        - name: Docker test
          commands:
            - echo 'Testing Docker'
  - name: Ubuntu tests
    task:
      jobs:
        - name: Ubuntu test
          commands:
            - echo 'Testing Ubuntu'

Why does my code contain tags that have already been deleted?#

Click for details

When using checkout with the --use-cache parameter the code will contain older changes that have not yet been propagated, because this particular update only happens every 3 days by default.

Reducing the value of SEMAPHORE_GIT_CACHE_AGE before performing the checkout ensures that changes are brought into the cache more often and should help mitigate this behavior:

export SEMAPHORE_GIT_CACHE_AGE=43200
The previous value is for 12 hours and is a good baseline but, depending on your development workflow, it might need to be lowered more.

Why is my Heroku deployment timing out?#

Click for details

Deploying to Heroku might timeout at the authentication step. If this occurs, the .netrc file might have expired.

In this case, we recommend to regenerate the file and recreate the secret.

How to comment on Github's pull requests from a workflow?#

Click for details

You can use the Github API to comment on pull requests. An example is shown below:

curl -X POST -H "Authorization: token <OAUTH_TOKEN>" https://api.github.com/repos/<owner>/<repo-name>/issues/<number>/comments -d '{"body":"body"}'

Why are you still charging my old credit card when I added a new default credit card?#

Click for details

If you’ve added a new credit card to the subscription, but the old one is still being charged, it means that the new credit card wasn't properly marked for usage. Here’s how to do that:

1. Click on the initials of your organization in the top right corner of the page,
2. In the dropdown menu, choose Plans & Billing,
3. Next to Payment details, click on Credit card info,
4. Go to the Subscription tab
5. Click on Manage
6. Go to Update Payment Method
7. Click the Use this button next to the credit card you'd like to use

After that, you can also remove the old credit card if you don't need it anymore.

Why can't I remove my old credit card after adding a new one?#

Click for details

If you run into this situation, it means that the old credit card is still in use. In order to mark the new credit card for usage, you can:

1. Click on the initial of your organization in the top right corner of the page,
2. In the dropdown menu, choose Plans & Billing
3. Next to the Payment details, click on Credit card info,
4. Go to the Subscription tab
5. Click on Manage
6. Go to Update Payment Method
7. Click on the Use this button next to the credit card you'd like to use

After that, you’ll be able to remove the old credit card.

How can I add repositories that belong to my GitHub organization?#

Click for details

In order to be able to do that, Semaphore 2.0 needs to be granted access within your GitHub organization. You can grant access here. If it has already been granted, there should be a green checkmark next to the name of your organization.

If not, you should either grant access or request it from the organization's owner.