Monorepo Workflows#

Note: The monorepo support is currently in beta stage of development.

This guide shows you how to optimize your Semaphore workflow for monorepo projects.

A monorepo (short for monolithic repository) is a software development strategy where code for many applications that may or may not be mutually dependent is stored in the same version-controlled repository.

Some advantages of a monorepo approach are:

  • Ease of code reuse - it is easy to abstract shared behavior into shared libraries
  • Simplified dependency management - third-party dependencies can also be shared
  • Atomic commits across multiple applications

Semaphore comes with out-of-box support for monorepos.

An example monorepo project setup#

Let's say you have a fairly simple monorepo project that consists of:

  • A WEB server application located inside /web-app/ directory
  • An iOS client application located inside /ios/ directory
  • A separate docs web page located inside /docs/ directory

A Semaphore pipeline for this monorepo project is the following:

version: "v1.0"
name: Monorepo project
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804

blocks:
  - name: Test WEB server
    dependencies: []
    run:
      when: "change_in('/web-app/')"
    task:
      jobs:
        - commands:
            - checkout
            - cd web-app
            - make test

  - name: Test iOS client
    dependencies: []
    run:
      when: "change_in('/ios/')"
    task:
      agent:
        machine:
          type: a1-standard-4
          os_image: macos-xcode12
      jobs:
        - commands:
            - checkout
            - cd ios
            - make test

  - name: Test docs page
    dependencies: []
    run:
      when: "change_in('/docs/')"
    task:
      jobs:
        - commands:
            - checkout
            - cd docs
            - make test

  - name: Integration tests
    dependencies: ["Test WEB server", "Test iOS client"]
    run:
      when: "change_in(['/web-app/', '/ios/'])"
    task:
      jobs:
        - commands:
            - checkout
            - make integration-tests

With this setup, we run separate tests for each part of the system and integration tests once both WEB server and iOS client tests pass.

The run field is used to conditionally run blocks only if the condition in its when sub-field is satisfied.

This, in combination with change_in function which checks whether there were any changes on given location(s) for a particular workflow, allows us to only run tests for those parts of the project that are currently being worked on.

In the example above this means that if we are working, for example, only on the iOS client app within the /ios/ directory of the repository, the only blocks that will be executed are the Test iOS client and Integration tests. Everything else will be skipped.

This can significantly reduce the time and cost and still provide you with required feedback about the changes that are being introduced into your monorepo.

The change_in function, by default, calculates the changeset (all changed files in the commits from commit range of interest for given workflow) in a slightly different way for the master and the other branches or pull requests. For more details and ways in which this can be modified please check the reference.

Set up the automatic deployments for a monorepo project#

Here we will assume that you already have a web-prod.yml, ios-prod.yml and docs-prod.yml deployments defined in your .semaphore/ directory (you can find examples for various kinds of deployments in the Use cases section of our docs) and we will focus on auto-promoting the right ones for a given workflow.

To achieve this we need to introduce the following promotions block to our configuration from above.

promotions:
  - name: Deploy Web Server
    pipeline_file: web-prod.yml
    auto_promote:
      when: "branch = 'master' and result = 'passed' and change_in('/web-app/')"
  - name: Deploy iOS client
    pipeline_file: ios-prod.yml
    auto_promote:
      when: "branch = 'master' and result = 'passed' and change_in('/ios/')"
  - name: Deploy docs page
    pipeline_file: docs-prod.yml
    auto_promote:
      when: "branch = 'master' and result = 'passed' and change_in('/docs/')"

With this, each part of the system will be automatically deployed when the tests pass on the master branch only if the push that initiated workflow contains changes in the location(s) given to the change_in function.

The complete example#

Here is the complete example for .semaphore/semaphore.yml file with the promotions block from above included:

version: "v1.0"
name: Monorepo project
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804

blocks:
  - name: Test WEB server
    dependencies: []
    run:
      when: "change_in('/web-app/')"
    task:
      jobs:
        - commands:
            - checkout
            - cd web-app
            - make test

  - name: Test iOS client
    dependencies: []
    run:
      when: "change_in('/ios/')"
    task:
      agent:
        machine:
          type: a1-standard-4
          os_image: macos-xcode12
      jobs:
        - commands:
            - checkout
            - cd ios
            - make test

  - name: Test docs page
    dependencies: []
    run:
      when: "change_in('/docs/')"
    task:
      jobs:
        - commands:
            - checkout
            - cd docs
            - make test

  - name: Integration tests
    dependencies: ["Test WEB server", "Test iOS client"]
    run:
      when: "change_in(['/web-app/', '/ios/'])"
    task:
      jobs:
        - commands:
            - checkout
            - make integration-tests

promotions:
  - name: Deploy Web Server
    pipeline_file: web-prod.yml
    auto_promote:
      when: "branch = 'master' and result = 'passed' and change_in('/web-app/')"

  - name: Deploy iOS client
    pipeline_file: ios-prod.yml
    auto_promote:
      when: "branch = 'master' and result = 'passed' and change_in('/ios/')"

  - name: Deploy docs page
    pipeline_file: docs-prod.yml
    auto_promote:
      when: "branch = 'master' and result = 'passed' and change_in('/docs/')"

Additional examples of monorepo configuration#

When a directory changes#

blocks:
  - name: Test WEB server
    run:
      when: "change_in('/web-app/')"

When a file changes#

blocks:
  - name: Unit tests
    run:
      when: "change_in('../Gemfile.lock')"

Changing the default branch from master to main#

blocks:
  - name: Test WEB server
    run:
      when: "change_in('/web-app/', {default_branch: 'main'})"

Exclude changes in the pipeline file#

Note: If you change the pipeline file, Semaphore will consider change_in as true. The following illustrates how to disable this behaviour.

blocks:
  - name: Test WEB server
    run:
      when: "change_in('/web-app/', {pipeline_file: 'ignore'})"

See also#