Skip to main content

Pipelines

A pipeline is a group of connected blocks. This page explains what pipelines are, how they organize workflow execution order, and what settings are available.

Overview​

Pipelines are groups of blocks that can be connected via dependencies to define their execution order.

Pipelines are also the unit of configuration. Each pipeline is encoded as separate a YAML file in the .semaphore folder.

For reference, here is an example pipeline with its respective YAML.

Example job with a single test

Block execution order​

In the same way that a block is a group of jobs, a pipeline is a group of blocks. Blocks can be connected using dependencies, which Semaphore uses to determine the execution order.

Take the following example:

  • Blocks B and C depend on Block A
  • Block D depends on both Blocks B and C

Pipeline execution order

In this scenario, Block B and C wait until Block A is done. Block D in turn, waits for Blocks B and C to be finished.

You can reorder blocks by changing their dependencies using the visual editor.

Video Tutorial: How to reorder blocks
What if we removed all dependencies?
If we removed dependencies between blocks, then all of them would run in parallel.

Pipeline initialization​

Before Semaphore can start running the jobs in the pipeline, the pipeline YAML file needs to be retrieved from the repository. As a first step, Semaphore request the file via the GitHub or BitBucket API and inspects its contents.

There are two types of pipelines:

  • Static: they don't require runtime evaluations and can be used as-is
  • Dynamic: contain elements that must be evaluated at runtime

Dynamic pipelines contain at least one of these elements:

Dynamic pipelines are evaluated in the initialization job.

Initialization job​

The initialization job only takes place for dynamic pipelines. It runs in a dedicated initialization agent and performs the following steps:

  1. Clones the repository using Git
  2. Parses and evaluates conditions in the input YAML file using Semaphore Pipeline Compiler (spc)
  3. Executes organization pre-flight checks, if any
  4. Executes project pre-flight checks, if any
  5. Saves the job output log
info

The Semaphore Pipeline Compiler (spc) is an open-source component. You can find the code in the spc repository

How to change the agent for init jobs​

You can change the agent that runs the initialization job in two ways:

  • For the organization: affects all projects in the organization. See organization init agent to learn how to change this setting
  • For the project: changes the agent running initialization for a single project. See project pre-flight checks to learn how to change this setting

How to access init logs​

Semaphore shows an Initializing message for pipelines with an initialization job. You can see the log by clicking on the See log link at the top of the pipeline.

See logs link

Here you can see the how spc evaluated the pipeline and all the actions taken during initialization.

Example init job log

Connecting pipelines with promotions​

Your project can have multiple pipelines to perform different tasks such as build, release, or test. Promotions connect pipelines. Multiple pipelines can be chained to create branching workflows to automate almost any task.

The workflow always starts with the default pipeline (located at .semaphore/semaphore.yml) and flows from left to right following promotions.

A workflow with 3 pipelines

For more information, see the Promotions page.

Pipeline settings​

Pipeline settings are applied to all its blocks. You can change pipeline settings with the editor or directly in the YAML.

Agents​

An agent is the machine and operating system where a job run. Semaphore keeps a pool of warm agents to ensure there's always one ready to work.

Semaphore Cloud provides the following agent types in x86 and ARM architectures:

You can add your own machines by installing self-hosted agents.

To select the agent running your jobs in a pipeline:

  1. Select the pipeline
  2. Select the Environment Type
  3. Select the Operating System
  4. Select the machine type

The available hardware changes depending on the type of environment you selected.

Agent Selection

Docker containers​

tip

If you want to build and run Docker images in your jobs, check the working with Docker page.

Jobs can run inside Docker containers. This allows you to define a custom-build environment with pre-installed tools and dependencies needed for your project. You can enable this setting in the pipeline agent or in the block agent override.

You can run multiple containers at the same time. The job runs in the first container (called main) and attaches the other containers to the same network. This is similar to how containers inside a Kubernetes pod communicate.

The network addresses of all containers are mapped to their names. Let's say you have two containers, "main" and "mysql", you can connect to the database from main with:

container 'main'
mysql --host=mysql --user=root

To run the job inside a Docker container:

  1. Select the pipeline
  2. In Environment Types select Docker Container(s)
  3. Select the machine type
  4. Type the Image name for this container
  5. Optionally, add environment variables
  6. Optionally, add more containers

Setting Docker environments

Prologue​

Commands in the prologue run before the jobs in the block start. Semaphore prepends these commands to every job contained in the pipeline. This works like the block prologue but for all blocks in the pipeline.

Pipeline prologue

Epilogue​

Commands in the epilogue run at the end of the jobs in all blocks. Semaphore appends these commands to every job contained in the pipeline. This works like the block epilogue but for all blocks in the pipeline.

Pipeline epilogue

Execution time limit​

The time limit for job execution. Defaults to 1 hour. Jobs running longer that the limit are forcibly terminated.

Execution time limit set 1 hour

Fail-fast​

Changes Semaphore behavior when a job fails. The possible actions are:

  • Stop: stop all running jobs
  • Cancel: already-running jobs are allowed to finish

The available strategies are:

  • Do nothing: fail fast is disabled
  • Stop all remaining jobs: stops the jobs
  • Cancel pending jobs: cancels the jobs that have not yet started
  • Stop remaining jobs, unless on the master branch: stops the jobs when the current branch is not "master"
  • Run a custom fail-fast strategy: define custom conditions for stop and cancel. See conditions DSL

Setting the fail fast strategy

Auto-cancel​

Define what happens when changes are pushed to the remote repository while a pipeline is running. By default, all pipelines will enter a First-In-First-Out (FIFO) queue and be executed in turn.

The possible strategies when a new pipeline is queued are:

  • running: stops and cancels all pipelines in the queue. Starts the new pipeline immediately
  • queued: cancels queued but no-yet-started pipelines. Waits for running pipelines to finish before starting the new one

The options are:

  • Do nothing: disables auto-cancel
  • Cancel all pipelines, both running and queued: stops running pipelines and cancel queued pipelines
  • Cancel only queued pipelines: cancels queued pipelines, wait for already-started pipelines to finish
  • On the master branch cancel only queued pipelines, on others cancel both running and queued: a mix of the last two strategies
  • Run a custom auto-cancel strategy: define custom conditions for stop and cancel. Uses the conditions DSL

Defining an auto-cancel strategy

YAML file path​

This option overrides the location of the pipeline file. This option is not available for the default pipeline (located at .semaphore/semaphore.yml).

Changing the pipeline configuration path

After pipeline jobs​

You can configure jobs to run once a pipeline stops. After pipeline jobs always run, even when jobs are canceled or have failed.

After-pipeline jobs are executed in parallel. Typical use cases for after-pipeline jobs are sending notifications, collecting test reports, or submitting metrics to an external server.

You can add after-pipeline jobs using YAML or the editor.

  1. Press +Add After Jobs
  2. Type the name of the job
  3. Add your commands
  4. Optionally, you can add more jobs
  5. To delete them, click the X next to the job

Adding an after pipeline job with the visual editor

Pipeline queues​

Queues allow you to control the order in which pipelines run. Semaphore pipelines can run sequentially or in parallel. For example, you can run CI pipelines in parallel on the main branch, while limiting deployment pipelines to run one at at time to prevent conflicts or race conditions.

Default and named queues​

Semaphore creates a queue for each Git push or pull requests. All workflows sharing the same commit SHA belong in the same queue and run sequentially.

In other words, every time you re-run a workflow, create a pull request, push a tag, or start a promotion, the pipeline is added to the end of the same-commit queue.

Default queue behavior

The downside of this strategy is that it may create conflicts due to pipelines running in parallel. In the example above, two deploy pipelines may try to deploy conflicting versions into the same environment, leading to instability.

You can avoid conflicts with named queues. Named queues allow you to manually assign pipelines to specific queues to they always run in sequence.

Named queued used to avoid deployment conflicts

In the example above we have two queues. The "main" queue runs CI pipelines for all commits. The possibly-disrupting deployment pipelines are assigned to a separate "deployment" queue. Thus, deployments are forced to run in sequence, avoiding conflicts due to parallelism.

Queue scopes​

Queues can be configured on two scopes:

  • Project (the default): pipelines belonging to the same project
  • Organization: pipelines belonging to all projects withing an organization

How to assign named queues​

Queues can only be defined using the pipeline YAML. There is currently no support for queue management using the visual editor.

To assign a pipeline to a named queue, follow these steps:

  1. Open the pipeline file
  2. Add queue, name and scope at the root level of the YAML
  3. Save the file and push it to the repository

The following example shows how to assign the pipeline to project-level queue called "Deployment queue".

Pipeline assigned to a project-level queue
version: v1.0
name: Production deployment
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu2004
queue:
  name: Deployment queue
  scope: project
blocks:
  - name: Deploy
    task:
      jobs:
      - commands:
          - make deploy

We can do the same with organization-level queues. This example shows how to assign the pipeline a queue called "Shared deployment queue" for all projects in the organization.

Pipeline assigned to an organization-level queue
version: v1.0
name: Project A deployment
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu2004
queue:
  name: Shared deployment queue
  scope: organization
blocks:
  - name: Deploy
    task:
      jobs:
      - commands:
          - make deploy

How to disable queues​

You can force pipelines to run in parallel by disabling queuing. This can help to obtain faster feedback when pipelines are completely independent and have no chance of causing conflicts.

To disable queue for a pipeline, follow these steps:

  1. Open the pipeline file
  2. Add queue, processing: parallel at the root level of the YAML
  3. Save the file and push it to the repository

The following example shows a pipeline that always runs in parallel. So, the pipeline starts as soon as an agent is available to take the jobs.

Pipeline that always runs in parallel
version: v1.0
name: Tests
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu2004
queue:
  processing: parallel
blocks:
  - name: Unit Tests
    task:
      jobs:
      - commands:
          - make test

Conditional queues​

You can use conditional statements to assign pipelines based on parameters like branch name or tag name.

The following example uses three rules:

  • On "master" branch: assign pipeline to organization queue called "Production"
  • When the commit includes any Git tag: assign pipeline to project-level queue called "Image build queue"
  • If none of the other rule match: default to running pipeline in parallel (no queue)
Conditional queue example
version: v1.0
name: Example project
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu2004
queue:
  - when: "branch = 'master'"
    name: Production
    scope: organization
  - when: "tag =~ '.*'"
    name: Image build queue
  - when: true
    processing: parallel
blocks:
  - name: Unit Tests
    task:
      jobs:
      - commands:
          - make test
  - name: Build and push image
    run:
      when: "branch = 'master' or tag =~ '.*'"
    task:
      jobs:
      - commands:
          - make image.build
          - make image.push
  - name: Deploy to production
    run:
      when: "branch = 'master'"
    task:
      jobs:
      - commands:
          - make deploy

See conditional queue reference for more details.

Pipeline limits​

Semaphore enforces a few limits to prevent misconfigured jobs and runaway processes from using all the resources in your organization.

This section describes the limits that Semaphore applies to pipelines. See job limits to see limits that apply to jobs.

Global job duration​

All jobs in a pipeline have a 1 hour limit. Jobs exceeding this limit are terminated.

You can change the limit up to a maximum value of 24 hours.

To change the global time limit for all jobs in a pipeline, follow these steps:

  1. Open the workflow editor
  2. Select the pipeline
  3. Scroll down to Execution time limit on the right-side menu
  4. Select a new value in Hours or Minutes
  5. Press Run the workflow, then Start

Changing global job limit duration

note

See job time limit to change the maximum duration for a single job.

Max pipelines per queue​

Pipeline queues have a limit of 30 pipelines. This limit is not configurable.

You can workaround the queue limit by assigning pipelines to named queues.

If you have a use case in which this limit is too constraining, please contact us at support@semaphoreci.com and we will try to work out a solution.

Max blocks per pipeline​

There is a hard limit of a 100 blocks per pipeline.

This limit is not adjustable. If you have a use case in which this limit is too constraining, please contact us at support@semaphoreci.com and we will try to work out a solution.

See also​