Django Continuous Integration

This guide shows you how to use Semaphore to set up a continuous integration (CI) pipeline for a Python Django web application. First, create a new Semaphore project.

Demo project

Semaphore maintains an example Django project:

The Semaphore annotated configuration file can be found in the repository file .semaphore/semaphore.yml.

The demo application is a simple task manager. We can create, edit and delete tasks. It also features an admin site to manage users, groups and permissions. The project uses the Django framework, unittest and nose for unit testing, pylint for code analysis, selenium for browser tests and MySQL as a database.

Overview of the CI pipeline

Our Django CI pipeline contains the following tasks:

  • Install dependencies.
    • Install python packages with pip.
  • Code Analysis.
    • Run static code analysis with pylint.
  • Unit tests.
    • Run unit tests for views and models.
  • Browser tests.
    • Run browser tests with python selenium webdriver.
  • Security tests.
    • Run security tests with Django deployment checklist.

The example pipeline is comprised of 5 blocks:

Project pipeline

We want fast feedback through the test pipeline, so we'll fail fast if the linting fails. This prevents running time consuming tests for semantically incorrect code.

Sample configuration

# Use the latest stable version of Semaphore 2.0 YML syntax:
version: v1.0

# Name your pipeline. In case you connect multiple pipelines with promotions,
# the name will help you differentiate between, for example, a CI build phase
# and delivery phases.
name: Semaphore Python / Django Example Pipeline

# An agent defines the environment in which your code runs.
# It is a combination of one of available machine types and operating
# system images.
# See
# and
    type: e1-standard-2
    os_image: ubuntu1804

# Blocks are the heart of a pipeline and are executed sequentially.
# Each block has a task that defines one or more jobs. Jobs define the
# commands to execute.
# See
  - name: "Install Dependencies"
      # This block installs all the python dependencies,
      # as well as all the required Linux packages.
      # The prologue section is always executed before each job on
      # the block.
      # See
          # Set the python version.
          # See
          - sem-version python 3.7
          # Install Linux dependencies.
          - sudo apt-get update && sudo apt-get install -y python3-dev && sudo apt-get install default-libmysqlclient-dev
        - name: pip
            # Get the latest version of our source code from GitHub:
            # See
            - checkout
            # Restore dependencies from cache. This command will not fail in
            # case of a cache miss. In case of a cache hit, pip can use it
            # to speed up the installation.
            # For more info on caching, see
            - cache restore requirements-$SEMAPHORE_GIT_BRANCH-$(checksum requirements.txt),requirements-$SEMAPHORE_GIT_BRANCH-,requirements-master-
            # Install python dependencies.
            # If not found in the cache, pip will download them.
            - pip download --cache-dir .pip_cache -r requirements.txt
            # Persist downloaded packages for future jobs.
            - cache store requirements-$SEMAPHORE_GIT_BRANCH-$(checksum requirements.txt) .pip_cache

  - name: "Run Code Analysis"
      # This block executes code analysis tests with pylint.
          - sem-version python 3.7
          - checkout
          # At this point, the cache contains the downloaded packages ...
          - cache restore requirements-$SEMAPHORE_GIT_BRANCH-$(checksum requirements.txt)
          # ... so pip does the installation much faster.
          - pip install -r requirements.txt --cache-dir .pip_cache
        - name: Pylint
            # list out files that are in directory and working tree
            # grep -v will exclude the files being considered for pylint
            # grep -E will matches files having .py extension
            # This command will help to pass required python files to pylint along with pylint_djanog plugin
            # Pylint with -E option will display only if there is any error
            - git ls-files | grep -v 'migrations' | grep -v '' | grep -v '' | grep -E '.py$' |
              xargs pylint -E --load-plugins=pylint_django

  - name: "Run Unit Tests"
      # This block runs the unit tests.
      # Since the test require a database, we start the database here.
      # Django automatically creates a test database schema.
          - sem-version python 3.7
          # Start a MySQL database. On Semaphore, databases run in the same
          # environment as your code.
          # See
          # Also
          - sem-service start mysql
          - checkout
          - cache restore requirements-$SEMAPHORE_GIT_BRANCH-$(checksum requirements.txt)
          - pip install -r requirements.txt --cache-dir .pip_cache
      # Two parallel test jobs are executed.
        - name: Model Test
            # Test the application's database models.
            - python test tasks.tests.test_models
        - name: View Test
            # Test the application's views.
            - python test tasks.tests.test_views

  - name: "Run Browser Tests"
      # This block runs browser-based tests.
      # We need to set environment variables.
      # See
        - name: DB_NAME
          value: 'pydjango'
      # This test requires the application to be running.
      # We start the application server here.
          - sem-version python 3.7
          # Start MySQL database.
          - sem-service start mysql
          # Install mysql client
          - sudo apt-get install -y -qq mysql-client
          # Create an empty database.
          # We can connect to the db with root and a blank password.
          - mysql --host= -uroot -e "create database $DB_NAME"
          - checkout
          - cache restore requirements-$SEMAPHORE_GIT_BRANCH-$(checksum requirements.txt)
          - pip install -r requirements.txt --cache-dir .pip_cache
          # Application is started.
          - nohup python runserver &
        - name: Browser Test
            # Run browser tests on Google Chrome.
            # On Semaphore, browsers are already installed.
            - python test tasks.tests.test_browser

  - name: "Run Security Tests"
      # This block runs through the security checklist for the project.
        - name: Deployment Checklist
           - checkout
           - sem-version python 3.7
           - cache restore requirements-$SEMAPHORE_GIT_BRANCH-$(checksum requirements.txt)
           - pip install -r requirements.txt --cache-dir .pip_cache
           # Test if project can be deployed securely.
           - python check --deploy --fail-level ERROR


Semaphore provides python 2 & 3 stable versions, as well as pip, pypy and virtualvenv.

Database access

In Semaphore, databases run in the same environment as the jobs, and can be accessed with a blank password. For more information on using databases see databases and services and sem-service.

Browser testing

Semaphore provides Chrome preinstalled so no installation steps are required for doing browser tests.

Run the demo yourself

A great way of getting started with Semaphore is to take a demo project and run it by yourself. Here's how to run the demo project with your own account:

  1. Fork the project on GitHub with your account.
  2. Clone the repository on your local machine.
  3. In Semaphore, follow the link on the sidebar to create a new project.
  4. Edit any file and do a push to GitHub, Semaphore will automatically start the CI pipeline.

Next steps

Congratulations! You have set up your first Django integration project on Semaphore. The next step is to configure deployment. For further information, please check the following tutorials:

See also

Still need help? Contact Us Contact Us