FAQ: Frequently asked questions#
Issues we stumble upon regularly, in all areas of Semaphore 2.0
How can I solve "Fail: Could not parse object" during bundle install?#
Click for details
If the bundle install output looks like this:
Fetching gem metadata from http://rubygems.org/.......
Fetching gem metadata from http://rubygems.org/..
Updating git://github.com/some/gem.git
fatal: Could not parse object 'a84dd3407eaf064064cca9650c354cb163384467'.
Git error: command <code>git reset --hard a84dd3407eaf064064cca9650c354cb163384467</code> in directory /home/runner/somehash/vendor/bundle/ruby/1.9.1/bundler/gems/gem-a84dd3407eaf has failed.
If this error persists you should try removing the cache directory '/home/runner/somehash/vendor/bundle/ruby/1.9.1/cache/bundler/git/gem-cbe2ee16ed53098079007f06cd77ed0890d0d752'
- Comment the gem line in the Gemfile
- Run bundle install
- Uncomment the gem line in the Gemfile
- Run bundle install again
The Gemfile.lock will now reference a valid git revision.
I get "Revision: COMMIT_SHA not found .... Exiting" error. What is the cause?#
Click for details
The reason for that error are changes to repository tree structure in git VCS,
which become effective in remote when your pipeline is still running.
It usually happens when you modify or remove commits (for example, with
git rebase or git commit --amend commands)
and then push with --force flag shortly after the previous push.
If you want to avoid executing previous pipelines after new pushes, check out how to set up auto-cancel strategy. If pipeline hasn't started after you pushed changes, you may also want to check out if GitHub delivered webhooks to Semaphore.
How can I change timezone?#
Click for details
The default timezone in the virtual machine is set to UTC.
The timezone can be changed in 2 ways:
- Assign a different value to TZ environment variable:
export TZ=Europe/Belgrade
/etc/localtime to one of the available timezones:
sudo ln -sf /usr/share/zoneinfo/Europe/Belgrade /etc/localtime
How can I build a project with git submodules?#
Click for details
- Add the following commands as a prologue:
git submodule init
git submodule update
git submodule deinit --force .
How can I use a self-signed certificate with a private Docker registry?#
Click for details
If you have a private Docker registry that uses a self-signed SSL certificate and pulling the Docker images does not work, the solution is to:
- Add a self-signed certificate as a secret on Semaphore
- Save it under the name domain.crt
- Add the following command to your pipeline:
sudo mv $SEMAPHORE_GIT_DIR/domain.crt /etc/docker/certs.d/myregistrydomain.com:5000/ca.crt
This will allow a connection to a private remote registry using the self-signed certificate.
Is it possible to attach to an ongoing SSH session?#
Click for details
It's possible to use sem attach to an ongoing SSH session, but you need to attach to the job ID of the SSH session.
To get the job ID, you can use sem get jobs to get a list of all running jobs.
How can I change the Postgres locale?#
Click for details
Semaphore uses sem-service to provide different versions of databases. The sem-service tool uses Docker containers instead of traditional Linux services.
So, the traditional way of changing locales no longer works, as it does not affect containers.
The following recipe provides an altered version of the container to sem-service.
The database should be available as before, without modifying your application in any way:
1. Create a Dockerfile with the following:
FROM postgres:9.6
RUN localedef -i pt_BR -c -f UTF-8 -A /usr/share/locale/locale.alias pt_BR.UTF-8
ENV LANG pt_BR.UTF-8
docker build - -t postgres:[lang] < Dockerfile
docker run --rm --net host -d -e POSTGRES_PASSWORD=semaphore --name postgres -v /var/run/postgresql:/var/run/postgresql postgres:[lang]
How can I remove Semaphore Status checks on pull requests?#
Click for details
You can disable Semaphore as a required status check on the repository settings page of your GitHub account.
How can I troubleshoot a stalling job?#
Click for details
The most common reason for stalled builds is a process that refuses to shut down properly. This is most likely a debug statement or a cleanup procedure in the catch procedure. Reproducing this can be hard sometimes. These are the steps we recommend:
1. Start a build on a branch and let it become stale.2. Attach the following to a running job:
sem attach [job-id].
3. Now, you should be in the instance of the job's virtual machine.
In the running instance, you can:
- List the running processes with ps aux or top. Are there any suspicious processes running?
- Run a strace on the running process: sudo strace -p
to see the last kernel instruction that it is waiting for.
For example, select(1, ... can mean that the process is waiting for user input.
- Check the system metrics at /tmp/system-metrics. This tracks memory and disk usage.
Lack of disk space or free memory can result in stalling.
- Check the Agent logs at /tmp/agent_logs. The logs could indicate that an agent is waiting for certain conditions.
- Check the Job logs at /tmp/job_logs.json. The logs could indicate that a job is waiting for certain conditions.
- Check the syslog as it can be also a valuable source of information: tail /var/log/syslog. It can indicate 'Out of memory' conditions.
While an issue is ongoing, you might consider using a shorter execution_time_limit in your pipelines.
This will prevent stale builds from running for a full hour.
Why is my test suite failing if all the tests passed?#
Click for details
This usually happens because of code coverage tools, e.g. simplecov, which can be set to fail the test suite if a minimum coverage level is not achieved.
Furthermore, some dependencies can configure an at_exit hook that will change the final exit code of the suite.
How can I avoid a "fatal: expected flush after ref listing" error?#
Click for details
If a commands fails with this:
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: expected flush after ref listing
retry to the failed command:
retry -t 5 <command>
retry tool here.
Why are tests passing locally but not on Semaphore?#
Click for details
The main reason for this behavior is differences in the stacks. As a first step, ensure that the same versions of languages, services, tools, and frameworks such as Selenium, browser drivers, Capybara, Cypress are used both locally and in the CI environment.
To achieve this, use sem-service, sem-version, and the operating systems' package manager.
Environment variables can also lead to unexpected behaviors, for instance, Semaphore 2.0 will set CI=true by default.
If you are using Docker containers when performing tests, it's possible that, while the command itself runs instantly,
the process will not be completely started, leading to certain endpoints not being available. Using a minimum sleep 10 can help in this scenario.
Cypress has a wait-on module that provides similar functionality.
https://semaphoreci.com/community/tutorials/how-to-deal-with-and-eliminate-flaky-tests
https://semaphoreci.com/blog/2017/08/03/tips-on-treating-flakiness-in-your-test-suite.html
How can I insert multiline commands?#
Click for details
You can divide a command into several lines by writing them in the folded
style > and by stripping the line break in the yaml file
-. To do this, we can start the command with a line containing only
>- and write the command in multiple lines below it:
commands:
- >-
if [ "foo" = "foo" ];
then commands...;
else commands...;
fi;
Block Style Indicator: The block style indicates how new lines inside the block
should behave. If you want to keep each line as a new line, use the literal style,
indicated by a pipe |. If you want them to be replaced by
spaces instead, use the folded style, indicated by a right angle bracket >.
Block Chomping Indicator: The chomping indicator controls what should happen
with new lines at the end of the string. The default, clip, puts a single new
line at the end of the string. To remove all new lines, strip them by putting a
minus sign - after the style indicator. Both clip and strip ignore how many new
lines are actually at the end of the block; to keep them all, put a plus sign +
after the style indicator.
Why won't my jobs start?#
You might be hitting the quota limitation. Check your organization's quota
in the You can also run Click for details
Activity Monitor by clicking on your organization's initials
in the top right corner of the page. Find more information about the quota and how to ask for
an increase here.sem get jobs to display all running jobs
to confirm how much of the quota is being used.
More information about sem get can be found here.
Why did my job break after changing the shell configuration?#
Click for details
Adding any of the following to your shell is not supported and will cause the jobs to immediately fail:
set -e
set -o pipefail
set -euxo pipefail
This also applies when sourcing a script that contains the previous settings:
source ~/my_script
. ~/my_script
Why are my workflows not running in parallel?#
Click for details
When pushing several commits into the same branch, Semaphore won't run parallel workflows. This means that pushing multiple times into a branch won't create parallel workflows, rather Semaphore will assign the new workflows into the queue and run them one at a time. However, it's possible to push commits to different branches and they will be run in parallel.
The only way to push several commits to a single branch and not wait for the workflows to finish one by one is to enable the auto_cancel feature.
How can I redeliver webhooks from Github to Semaphore?#
Click for details
This is not the most common problem, but occasionally Semaphore does not receive a webhook from Github for some reason. This results in a workflow not being triggered. You can redeliver the webhook, however, and this should trigger the workflow. These are the steps to redeliver webhooks from Github:
1. Go to your repository on GitHub
2. Click Settings
3. Click Webhooks
4. Click Edit for the webhook you want to redeliver
5. Scroll down to Recent Deliveries and search for the one that failed
6. Click the ... symbol, then click Redeliver
Why did my workflow stop without explanation?#
Click for details
The workflow might have been stopped by the auto_cancel feature. There are two auto-cancel strategies: running and queued.
The running strategy stops all pipelines in the queue as soon as a new one appears.
The queued strategy will only cancel pipelines that are waiting in the queue and have not yet started.
What can I use to split my parallel tests?#
Click for details
We recommend using semaphore_test_boosters gem. This gem spreads tests across parallel jobs based on a configuration file or uniform file distribution (default behavior). Execution time-based distribution is not supported yet.
Other options are also supported, e.g. Knapsack (both free and pro versions).
Knapsack Rspec example:
jobs:
- name: Knapsack RSpec
parallelism: 5
commands:
- CI_NODE_TOTAL=$SEMAPHORE_JOB_COUNT CI_NODE_INDEX=$((SEMAPHORE_JOB_INDEX-1)) bundle exec rake 'knapsack:rspec'
Knapsack Pro Rspec example:
jobs:
- name: Knapsack Pro RSpec
parallelism: 5
commands:
- bundle exec rake 'knapsack_pro:queue:rspec'
You can find a more detailed example in the official documentation.
Why aren't workflows triggering on pull requests?#
Click for details
Make sure to enable pull requests in the project Settings.
If the configuration is correct, check if the pull request can be merged or if there are conflicts.
Semaphore uses the merge commit to run the workflows, so there will be no merge commit if there is a conflict in the pull request.
How can I use different machines in the same pipeline?#
Click for details
In certain scenarios, it's advantageous to use different machine types for different jobs in a pipeline. For instance, some operations require fewer resources and it would be wasteful to use a bigger machine or a test suite that has to run in multiple environments.
Semaphore 2.0 provides the agent in task feature that allows mixing and matching of various machine types and even
Docker-based CI/CD:
version: v1.0
name: Tests
agent:
machine:
type: e1-standard-2
os_image: ubuntu1804
blocks:
- name: 'MacOS tests'
task:
agent:
machine:
type: a1-standard-4
os_image: macos-xcode13
jobs:
- name: 'MacOS test'
commands:
- echo 'Testing MacOS'
- name: Docker tests
task:
agent:
machine:
type: e1-standard-4
containers:
- name: main
image: 'registry.semaphoreci.com/ruby:2.6'
jobs:
- name: Docker test
commands:
- echo 'Testing Docker'
- name: Ubuntu tests
task:
jobs:
- name: Ubuntu test
commands:
- echo 'Testing Ubuntu'
Why does my code contain tags that have already been deleted?#
Click for details
When using checkout with the
--use-cache parameter
the code will contain older changes that have not yet been propagated, because this particular update only happens every 3 days by default.
Reducing the value of SEMAPHORE_GIT_CACHE_AGE before performing the
checkout ensures that changes are brought into the cache more
often and should help mitigate this behavior:
export SEMAPHORE_GIT_CACHE_AGE=43200
Why is my Heroku deployment timing out?#
Click for details
Deploying to Heroku might timeout at the authentication step.
If this occurs, the .netrc file might have expired.
In this case, we recommend to regenerate the file and recreate the secret.
How to comment on Github's pull requests from a workflow?#
Click for details
You can use the Github API to comment on pull requests. An example is shown below:
curl -X POST -H "Authorization: token <OAUTH_TOKEN>" https://api.github.com/repos/<owner>/<repo-name>/issues/<number>/comments -d '{"body":"body"}'
Why are you still charging my old credit card when I added a new default credit card?#
Click for details
If you’ve added a new credit card to the subscription, but the old one is still being charged, it means that the new credit card wasn't properly marked for usage. Here’s how to do that:
1. Click on the initials of your organization in the top right corner of the page,
2. In the dropdown menu, choose Plans & Billing,
3. Next to Payment details, click on Credit card info,
4. Go to the Subscription tab
5. Click on Manage
6. Go to Update Payment Method
7. Click the Use this button next to the credit card you'd like to use
After that, you can also remove the old credit card if you don't need it anymore.
Why can't I remove my old credit card after adding a new one?#
Click for details
If you run into this situation, it means that the old credit card is still in use. In order to mark the new credit card for usage, you can:
1. Click on the initial of your organization in the top right corner of the page,
2. In the dropdown menu, choose Plans & Billing
3. Next to the Payment details, click on Credit card info,
4. Go to the Subscription tab
5. Click on Manage
6. Go to Update Payment Method
7. Click on the Use this button next to the credit card you'd like to use
After that, you’ll be able to remove the old credit card.
How can I add repositories that belong to my GitHub organization?#
Click for details
In order to be able to do that, Semaphore 2.0 needs to be granted access within your GitHub organization. You can grant access here. If it has already been granted, there should be a green checkmark next to the name of your organization.
If not, you should either grant access or request it from the organization's owner.