Pipeline queues allow you to control which pipelines Semaphore runs sequentially and/or in parallel. For example, you can configure consecutive pipelines to run in parallel on the main branch, while allowing only one deployment to production to be running at any given time.
You can configure this behaviour via the queue property in the pipeline's YAML configuration file.
By default, Semaphore assigns pipelines to the same queue and runs them sequentially if they are initiated from the same branch/tag/pull request with the same configuration, e.g. multiple pushes or multiple promotions.
Using queues for deployments#
You can use queues to prevent parallel runs of deployment pipelines that might cause issues in the environment you are deploying to.
If you are deploying projects independently one from another, you can use
separate queues for each project by setting
scope: project in the queue
In the following example, we configure the deployment pipeline to run
sequentially in a queue called
version: v1.0 name: Production deployment agent: machine: type: e1-standard-2 os_image: ubuntu1804 queue: name: Deployment queue scope: project blocks: - name: Deploy task: jobs: - commands: - make deploy
In cases where multiple projects are deployed to the same environment (e.g.
to the same Kubernetes cluster) you need the deployment pipelines of those projects to
queue together. To set that up, use queue configurations with equal values
scope set to organization in all related projects.
The following example illustrates a queue configuration that can be used for projects that need to have a shared deployment queue.
version: v1.0 name: Project A deployment agent: machine: type: e1-standard-2 os_image: ubuntu1804 queue: name: Shared deployment queue scope: organization blocks: - name: Deploy task: jobs: - commands: - make deploy
Running pipelines in parallel#
In cases where pipeline runs are completely independent one from another, you might want to run them in parallel to receive faster feedback.
You can configure this via the
processing property of the queue configuration.
This is mostly useful for pipelines that are only running tests and when you want to have results for each push to GitHub, e.g. on the main branch.
In the event that you only need the result of the pipeline from the latest push, you might want to consider the auto-cancel feature.
The following is an example of a queue configuration that allows parallel pipelines:
version: v1.0 name: Tests agent: machine: type: e1-standard-2 os_image: ubuntu1804 queue: processing: parallel blocks: - name: Unit Tests task: jobs: - commands: - make test
Conditional queue configurations#
In some cases, it is useful to have different queue behaviours for different branches of the same pipeline. Similarly, you might only need to assign tag-triggered pipelines to a separate queue and let the rest to behave normally.
The solution for such cases is to have multiple queue configurations that are only applicable if certain conditions are met. Semaphore supports this option via conditional queue configurations. Here's an example:
version: v1.0 name: Example project agent: machine: type: e1-standard-2 os_image: ubuntu1804 queue: - when: "branch = 'master'" name: Production scope: organization - when: "tag =~ '.*'" name: Image build queue - when: true processing: parallel blocks: - name: Unit Tests task: jobs: - commands: - make test - name: Build and push image run: when: "branch = 'master' or tag =~ '.*'" task: jobs: - commands: - make image.build - make image.push - name: Deploy to production run: when: "branch = 'master'" task: jobs: - commands: - make deploy
In this example, if the pipeline is triggered by a push to the master branch, it
will be assigned to the queue called
Production. This queue is defined within the
scope of the organization and can be shared with other projects.
With this queue configuration, only one production deployment is possible at any given time for the whole organization.
If the pipeline is triggered by a git tag, however, it will be
assigned to the queue called
Image build queue, which is specific to that project.
This should ensure that images are built and pushed in chronological order.
The third configuration has a "catch-all" condition that is applied if none of the conditions from the previous configurations is true.
This is the case for all pipelines triggered by a push to any non-master branch or any pull request, and it means that those pipelines will be run in parallel.