Changed-based execution for monorepos#

This guide shows you how to use change detection to optimize monorepo continuous integration (CI).

Demo project#

A monorepo is a container repository with multiple separate applications or projects. Semaphore maintains a monorepo reference demo on GitHub that you can use to learn:

You’ll find a Semaphore configuration file with a sample CI pipeline at .semaphore/semaphore.yml.

The repository contains three individually-deployable applications. They can be found under the /services folder.

  • Billing: a billing system written in Go. It uses mux to provide an HTTP endpoint on port 8000.
  • User: a user account management application. Written in Ruby, it employs an in-memory database and uses Sinatra to expose an HTTP endpoint.
  • UI: an Elixir-based Web application component.

These applications are meant to work together. After forking and cloning the repository, you can start the suite with the following command:

$ bash start.sh

Overview of the pipeline#

The pipeline performs the following tasks for the three applications in the monorepo:

  • Lint: uses a linting tool to detect potential errors in the source code.
  • Test: runs the application's unit tests.

To avoid re-running tasks for unchanged code, the pipeline uses change detection. This means that when an application has been modified by recent commits, the related block runs; but unchanged blocks are skipped.

The following screenshot shows the workflow resulting from changing a file inside the /service/ui folder:

CI pipeline for monorepo

Sample configuration#

The demo uses the following configuration. If you're new to Semaphore, we recommend going through the guided tour and related documentation for more information.

# Use the latest stable version of Semaphore 2.0 YML syntax:
version: v1.0

# Name your pipeline. In case you connect multiple pipelines with promotions,
# the name will help you differentiate between, for example, a CI build phase
# and delivery phases.
name: Monorepo Demo

# An agent defines the environment in which your code runs.
# It is a combination of one of available machine types and operating
# system images.
# See https://docs.semaphoreci.com/article/20-machine-types
# and https://docs.semaphoreci.com/article/32-ubuntu-1804-image
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804

# Blocks are the heart of a pipeline and are executed sequentially.
# Each block has a task that defines one or more jobs. Jobs define the
# commands to execute.
# See https://docs.semaphoreci.com/essentials/concepts/
blocks:

  # This block tests the UI service application
  - name: "UI Service"
    dependencies: []

    # Run block only when a file changes in the /services/ui folder
    # except when is markdown
    run:
      when: "change_in('/services/ui', {exclude: '/services/ui/**/*.md'})"

    task:
      # The prologue runs *before* each job in the block
      # The prologues clones the repository, installs Elixir dependencies
      # and caches them
      # See https://docs.semaphoreci.com/article/50-pipeline-yaml#prologue
      prologue:
        commands:
          - checkout
          - cd services/ui
          - sem-version elixir 1.9
          - cache restore
          - mix local.hex --force
          - mix local.rebar --force
          - mix deps.get
          - mix deps.compile
          - cache store

      # This block runs two parallel jobs: a linter to check the code for
      # deficiencies and the application unit tests
      jobs:
        - name: Lint
          commands:
            - mix credo
        - name: Test
          commands:
            - mix test

  # This block tests the user service application
  - name: "User Service"
    dependencies: []

    # Run block only when a file changes in the /services/user folder
    # except when is markdown
    run:
      when: "change_in('/services/users', {exclude: '/services/users/**/*.md'})"

    task:
      # The prologue clones the repo, installs and caches Ruby dependencies
      prologue:
        commands:
          - checkout
          - cd services/users
          - sem-version ruby 2.5
          - cache restore
          - bundle install
          - cache store

      # This block runs two parallel jobs: a linter to check the code for
      # deficiencies and the application unit tests
      jobs:
        - name: Lint
          commands:
            - bundle exec rubocop

        - name: Test
          commands:
            - bundle exec ruby test.rb

  # This block tests the billing application
  - name: "Billing Service"
    dependencies: []

    # Run block only when a file changes in the /services/ui folder
    # except when is markdown
    run:
      when: "change_in('/services/billing', {exclude: '/services/billing/**/*.md'})"

    task:
      # The prologue installs Go modules and caches them after cloning the repo
      prologue:
        commands:
          - checkout
          - cd services/billing
          - sem-version go 1.14
          - cache restore
          - go get ./...
          - cache store

      # This block runs two parallel jobs: a linter to check the code for
      # deficiencies and the application unit tests
      jobs:
        - name: Lint
          commands:
            - gofmt -l .

        - name: Test
          commands:
            - go test ./...

Run the demo yourself#

A good way to start using Semaphore is to take a demo project and run it yourself. Here’s how to build the demo project with your own account:

  1. Fork the project on GitHub.
  2. Clone the repository to your local machine.
  3. In Semaphore, follow the link on the upper right side to create a new project.
  4. Select the forked repository.
  5. Modify or create a file inside an application folder and commit the changes to master.

Next steps#

You can use change_in to build smarter pipelines. The change function not only allows you to skip or activate blocks, it can also be used within promotions to start additional pipelines.

To learn more about change_in, you can also check out these resources: - Monorepo workflows - Change_in reference

See also#