Frequently asked questions#

Issues we stumble upon regularly, across all parts of Semaphore 2.0

How to solve "Fail: Could not parse object" during bundle install?#

Click for details

If the bundle install output looks like this:

Fetching gem metadata from http://rubygems.org/.......
Fetching gem metadata from http://rubygems.org/..
Updating git://github.com/some/gem.git
fatal: Could not parse object 'a84dd3407eaf064064cca9650c354cb163384467'.
Git error: command <code>git reset --hard a84dd3407eaf064064cca9650c354cb163384467</code> in directory /home/runner/somehash/vendor/bundle/ruby/1.9.1/bundler/gems/gem-a84dd3407eaf has failed.
If this error persists you could try removing the cache directory '/home/runner/somehash/vendor/bundle/ruby/1.9.1/cache/bundler/git/gem-cbe2ee16ed53098079007f06cd77ed0890d0d752'
This problem occurs when there have been changes like force-pushes to a git repo which is referenced in a Gemfile. You can solve it by following these steps:

- Comment that gem line in the Gemfile
- Run bundle install
- Uncomment the gem line in the Gemfile
- Run bundle install again

The Gemfile.lock will now reference a valid git revision.

How to change the timezone?#

Click for details

The default timezone in the virtual machine is set to UTC. The timezone can be changed in 2 ways: - Assign a different value to TZ environment variable:

export TZ=Europe/Belgrade
- Create a symlink in /etc/localtime to one of the available timezones:
sudo ln -sf /usr/share/zoneinfo/Europe/Belgrade /etc/localtime

How to build a project with git submodules?#

Click for details

- Add the following commands as a prologue:

git submodule init
git submodule update
- Add the following command as an epilogue:
git submodule deinit --force .
Make sure that Semaphore has permissions to clone your submodules repository. In our private dependencies page you can find more information about setting permissions for private repositories.

How to use a self-signed certificate with private Docker registry?#

Click for details

If you have a private Docker registry that uses a self-signed SSL certificate and pulling the Docker images does not work. The solution is to:

- Add a self-signed certificate as a secret on Semaphore
- Save it under the name of domain.crt
- Add the following command to your pipeline:

sudo mv $SEMAPHORE_GIT_DIR/domain.crt /etc/docker/certs.d/myregistrydomain.com:5000/ca.crt

This will allow the connection to a private remote registry using the self-signed certificate.

Is it possible to attach to an ongoing SSH session?#

Click for details

It's possible to use sem attach to an ongoing SSH session but you'd need to attach to the job ID of the SSH session. To get the job ID you can use sem get jobs to get the list of all running jobs.

How to change the Postgres locale?#

Click for details

Semaphore uses sem-service to provide different versions of databases. The sem-service tool uses Docker containers instead of traditional Linux services. So, the traditional way of changing locales no longer works since it does not affect containers.

The following recipe provides an altered version of the container to sem-service. The database should be available as before, without modifying your application in any way:

1. Create a Dockerfile with the following:

FROM postgres:9.6
RUN localedef -i pt_BR -c -f UTF-8 -A /usr/share/locale/locale.alias pt_BR.UTF-8
ENV LANG pt_BR.UTF-8
2. Rebuild the Postgres image using the locale:
docker build - -t postgres:[lang] < Dockerfile
3. Start the newly created image:
docker run --rm --net host -d -e POSTGRES_PASSWORD=semaphore --name postgres -v /var/run/postgresql:/var/run/postgresql postgres:[lang]

How can I remove Semaphore Status checks on pull requests?#

Click for details

You can disable Semaphore as a required status check through the repository settings page in your GitHub account.

How to troubleshoot a stalling job?#

Click for details

The most common reason for stalled builds is a process that refuses to shut down properly. Either a debug statement or a cleanup procedure in the catch procedure. Reproducing this can be hard sometimes. These are the steps we recommend:

1. Start a build on a branch and let it get stale.
2. Attach to a running job: sem attach [job-id].
3. Now, you should be in the instance of the job's virtual machine.

In the running instance, you can:

- List the running processes with ps aux or top. Is there any suspicious process running?
- Run a strace on the running process: sudo strace -p to see the last kernel instruction that it is waiting for. For example, select(1, ... can mean the process is waiting for user's input.
- Look into the system metrics at /tmp/system-metrics. This tracks memory and disk usage. Lack of disk space or free memory can introduce unwanted stalling into jobs.
- Look into the Agent logs at /tmp/agent_logs. The logs could indicate waiting for some conditions.
- Look into the Job logs at /tmp/job_logs.json. The logs could also indicate waiting for some conditions.
- Check the syslog as it can be also a valuable source of information: tail /var/log/syslog. It can indicate 'Out of memory' conditions.

While this issue is ongoing, you might consider using a shorter execution_time_limit in your pipelines. This will prevent stale builds to run for a full hour, and fail sooner.

Why is my test suite failing if all the tests pass?#

Click for details

This usually happens because code coverage tools, for instance simplecov, can be set to fail the test suite if a minimum coverage is not achieved.
Besides the above, some dependencies can configure an at_exit hook and will change the final exit code of the suite.

How to solve the "fatal: expected flush after ref listing" error?#

Click for details

If a commands fails with this:

error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: expected flush after ref listing
It means the communication between Semaphore and Github was interrupted. As a workaround, you may add retry to the failed command:
retry -t 5 <command>
You may find more information about the retry tool here.

Why are tests passing locally but not on Semaphore?#

Click for details

The main reason for this behavior is differences in the stacks. As a first step, ensure the same versions of languages, services, tools and frameworks such as Selenium, browser drivers, Capybara, Cypress are used both locally and in the CI environment. To achieve this make use of sem-service , sem-version and also the operating systems' package manager. Environment variables can also lead to unexpected behaviors, for instance, Semaphore 2.0 will set CI=true by default.

If you are using Docker containers when performing the tests, in some cases it's possible that while the command itself runs instantly the process will not be completely started, leading to certain endpoints not being available. Using sleep 10 or a larger value can help in this scenario. Cypress has the wait-on module that provides a similar functionality.

Finally, when tests have different outcomes between reruns using the same commit or in an SSH session then this is a case of flaky tests. The following articles should help in this regard:
https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests
https://semaphoreci.com/blog/2017/08/03/tips-on-treating-flakiness-in-your-test-suite.html

How can I insert multiline commands?#

Click for details

You can divide a command in several lines by writing the line in the folded style > and by stripping the line break in the yaml file -. To do this, we can start the command with line containing only >- and write the command in more lines below it:

commands:
  - >-
    if [ "foo" = "foo" ];
    then commands...;
    else commands...;
    fi;

Block Style Indicator: The block style indicates how new lines inside the block should behave. If you want to keep them as new lines, use the literal style, indicated by a pipe |. If instead you want them to be replaced by spaces, use the folded style, indicated by a right angle bracket >.

Block Chomping Indicator: The chomping indicator controls what should happen with new lines at the end of the string. The default, clip, puts a single new line at the end of the string. To remove all new lines, strip them by putting a minus sign - after the style indicator. Both clip and strip ignore how many new lines are actually at the end of the block; to keep them all put a plus sign + after the style indicator.

Why my jobs don't start?#

Click for details

You might be hitting the quota limitation. Check your organization's quota in the Activity Monitor by clicking on the initial of your organization in the top right corner of the page. More information about quota and how to ask for an increase here.

You may also run sem get jobs to display all running jobs so you may confirm how much quota is being used. More information about sem get here.

Why does my job break after changing the shell configuration?#

Click for details

Adding any of the following to your shell is not supported and will cause the jobs to immediately fail:

set -e
set -o pipefail
set -euxo pipefail

This also applies when sourcing a script that contains the previous settings:

source ~/my_script
. ~/my_script

Why are my workflows not running in parallel?#

Click for details

When pushing several commits into the same branch, Semaphore won't run parallel workflows. This means that pushing several times into a branch won't create parallel workflows, instead, Semaphore assigns the new workflows to the queue and run one workflow at a time. However, it's possible to push commits to different branches and they will be run in parallel.

The only way to push several commits to a single branch and not wait for the workflows to finish one by one is to enable the auto_cancel feature.

How to redeliver webhooks from Github to Semaphore?#

Click for details

Even if this is not a common problem, it might happen that Semaphore does not receive a webhook from Github. This results in a workflow not being triggered. You can redeliver the webhook and this should trigger the workflow. These are the steps to redeliver webhooks from Github:

1. Go to your repository on GitHub
2. Click Settings
3. Click Webhooks
4. Click Edit for the webhook you want to redeliver
5. Scroll down to Recent Deliveries and search for the failed one
6. Click the ... symbol, then click Redeliver

Why does my workflow stop without explanation?#

Click for details

The workflow might have been stopped by the auto_cancel feature. There are two auto-cancel strategies: running and queued.

The running strategy stops all pipelines in the queue as soon as a new one appears.

The queued strategy will only cancel pipelines that are waiting in the queue and have not yet started to run.

What can I use to split my parallel tests?#

Click for details

The recommended way is by using the semaphore_test_boosters gem. Other options are also supported, for instance Knapsack, both free and pro versions.

Knapsack Rspec example:

jobs:
  - name: Knapsack RSpec
    parallelism: 5
    commands:
      - CI_NODE_TOTAL=$SEMAPHORE_JOB_COUNT CI_NODE_INDEX=$((SEMAPHORE_JOB_INDEX-1)) bundle exec rake 'knapsack:rspec'

Knapsack Pro Rspec example:

jobs:
  - name: Knapsack Pro RSpec
    parallelism: 5
    commands:
      - bundle exec rake 'knapsack_pro:queue:rspec'


You can find a more detailed example in the official documentation.

Why workflows don't trigger on pull requests?#

Click for details

Make sure to enable pull requests in the project Settings.

If the configuration is correct, check if the pull request can be merged, or if there are conflicts.
Semaphore uses the merge commit to run the workflows and there is no merge commit if there is a conflict on the pull request.

How can I use different machines in the same pipeline?#

Click for details

In certain scenarios it's advantageous to use different machine types for certain jobs in a pipeline. For instance, some operations require less resources and it would be wasteful to use a bigger machine or a test suite has to run in multiple environments.

Semaphore 2.0 provides the agent in task feature that allows mixing and matching of various machine types and even Docker based CI/CD:

version: v1.0
name: Tests
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: 'MacOS tests'
    task:
      agent:
        machine:
          type: a1-standard-4
          os_image: macos-xcode12
      jobs:
        - name: 'MacOS test'
          commands:
            - echo 'Testing MacOS'

  - name: Docker tests
    task:
      agent:
        machine:
          type: e1-standard-4
        containers:
          - name: main
            image: 'registry.semaphoreci.com/ruby:2.6'
      jobs:
        - name: Docker test
          commands:
            - echo 'Testing Docker'
  - name: Ubuntu tests
    task:
      jobs:
        - name: Ubuntu test
          commands:
            - echo 'Testing Ubuntu'

Why does my code contain tags that have already been deleted?#

Click for details

When using checkout with the --use-cache parameter the code will contain older changes that have not been propagated yet since by default this update only happens every 3 days.

Reducing the value of SEMAPHORE_GIT_CACHE_AGE before performing the checkout ensures changes are brought into the cache more often and should help mitigate the behavior:

export SEMAPHORE_GIT_CACHE_AGE=43200
The above value is for 12 hours and is a good baseline but depending on your development workflow it might require to be lowered even more.

Why are you still charging my old credit card when I added a new default credit card?#

Click for details

If you’ve added a new credit card to the subscription, but the old one is still being charged, it means that the new credit card wasn't marked for usage. Here’s how to do that:

1. Click on the initial of your organization in the top right corner of the page,
2. In the dropdown menu, choose Plans & Billing,
3. Next to the Payment details, click on Credit card info,
4. Go to Subscription tab
5. Click on Manage
6. Go to Update Payment Method
7. Click on the Use this button next to the credit card you'd like to use

After that, you can also remove the old credit card if you don't need it anymore.

Why can't I remove the old credit card after adding a new one?#

Click for details

If you run into this situation, it means that the old credit card is still in use. In order to mark the new credit card for usage, you can:

1. Click on the initial of your organization in the top right corner of the page,
2. In the dropdown menu, choose Plans & Billing
3. Next to the Payment details, click on Credit card info,
4. Go to Subscription tab
5. Click on Manage
6. Go to Update Payment Method
7. Click on the Use this button next to the credit card you'd like to use

After that, you’ll be able to remove the old credit card.

How can I add repositories that belong to my GitHub organization?#

Click for details

In order to be able to do that, the access for Semaphore 2.0 needs to be granted within your GitHub organization. You can grant the access here. If it has already been granted, there should be a green checkmark next to the name of your organization.

If not, you should either grant access or request it from the organization's owner.