Working with Docker#
Semaphore CI/CD jobs can run and build Docker images, as well as push images to Docker repositories or other remote storage.
Docker CLI comes preinstalled on Semaphore VMs, so you don't have to install it yourself.
ℹ️ This article describes the process of building, publishing, and testing Docker containers on Semaphore. If you want to run jobs inside of a Docker image, refer to the custom CI/CD environment with Docker documentation.
version: v1.0 name: Hello Docker agent: machine: type: e1-standard-2 os_image: ubuntu1804 blocks: - name: "Build" task: jobs: - name: Docker build commands: - checkout - docker build -t awesome-image . - docker images
Semaphore provides tutorials and demo applications with working CI/CD pipelines that you can use to get started quickly:
- CI/CD for Microservices on Kubernetes and demo project on GitHub
- Java Spring CI/CD and demo project on GitHub
Using a public Docker image in CI/CD jobs#
In order to use an existing Docker image from a public Docker Hub repository,
you need to execute
docker run as follows:
docker run -d -p 1234:80 nginx:alpine
The previous command will download the public
nginx:alpine image and run it in your
Docker Hub rate limits
Please note that due to the introduction of the rate limits on Docker Hub, all pulls have to be authenticated. If you are pulling any images from the Docker Hub public repository, please make sure you are logged in to avoid any failiures. You can find more information on how to authenticate in our Docker authentication documentation.
Here is an example Semaphore pipeline file:
# .semaphore/semaphore.yml version: v1.0 name: Using a public Docker image agent: machine: type: e1-standard-2 os_image: ubuntu1804 blocks: - name: Pull Nginx image task: jobs: - name: Docker Hub commands: - checkout - echo $DOCKERHUB_PASSWORD | docker login --username "$DOCKERHUB_USERNAME" --password-stdin - docker run -d -p 1234:80 nginx:alpine - wget http://localhost:1234 secrets: - name: docker-hub
wget http://localhost:1234 command is an example to verify that the
Docker image with the Nginx web server is working and listening to TCP port
number 1234, as specified in the
docker run command.
For more information on using Docker, refer to the Docker user documentation.
Using a Docker image from a private registry#
In order to use a Docker image from a private Docker registry, you will first need to log in to that registry. The commands that you need to run for this are shown below:
echo $DOCKER_PASSWORD | docker login --username "$DOCKER_USERNAME" --password-stdin registry.example.com docker pull registry-owner/image-name
We also need a secure way to store and use account credentials, without storing them in version control. We can do this in Semaphore by using secrets.
In this case, the Semaphore configuration file would appear as follows:
# .semaphore/semaphore.yml version: v1.0 name: Using a private Docker image agent: machine: type: e1-standard-2 os_image: ubuntu1804 blocks: - name: Using private Docker image task: jobs: - name: Run container from Docker Hub commands: - checkout - echo $DOCKER_PASSWORD | docker login --username "$DOCKER_USERNAME" --password-stdin - docker pull "$DOCKER_USERNAME"/myimage - docker images - docker run "$DOCKER_USERNAME"/myimage secrets: - name: docker-hub
docker-hub secret referenced in the example using
sem CLI, as shown below:
$ sem get secrets docker-hub apiVersion: v1beta kind: Secret metadata: name: docker-hub id: a2aaefdb-a4ff-4bc2-afd9-2afa9c7f3e51 create_time: "1538456457" update_time: "1538456537" data: env_vars: - name: DOCKER_USERNAME value: docker-username - name: DOCKER_PASSWORD value: docker-password files: 
Note that the names of the two environment variables used can be anything you want. We recommend always using descriptive names.
You can learn more about working with secrets in Semaphore 2.0 in the secrets documentation.
Building a Docker image from a Dockerfile#
You can use Semaphore to build Docker images directly from a
in your source code repository.
For example, let's say that you have the following
FROM golang:alpine RUN mkdir /files COPY hello.go /files WORKDIR /files RUN go build -o /files/hello hello.go ENTRYPOINT ["/files/hello"]
This example assumes that you have a file named
hello.go in your Git repository. The
Dockerfile creates a new directory in the Docker image and puts
it. Then, it compiles the Go file and the executable file is stored as
ENTRYPOINT Docker command will automatically execute
files/hello when the Docker image is run.
Please note that the
Dockerfile should be committed to Git as it will be
used by Semaphore 2.0.
After that, the contents of your Semaphore 2.0 pipeline file should appear as follows:
# .semaphore/semaphore.yml version: v1.0 name: Building Docker images agent: machine: type: e1-standard-2 os_image: ubuntu1804 blocks: - name: Build Go executable task: jobs: - name: Docker Hub commands: - checkout - docker build -t hello:v1 . - docker run hello:v1
The default name of the image, in this case, will be
hello:v1 – you can, however, rename it as you see fit.
Pushing a Docker image to a registry#
Once you create a container image, you need to push it to a registry in most cases.
For this purpose you will first need to authenticate via
Here's an example Semaphore configuration file, in which we push to a private registry on Docker Hub. You can use this for any other container registry as well:
# .semaphore/semaphore.yml version: v1.0 name: Pushing a Docker image agent: machine: type: e1-standard-2 os_image: ubuntu1804 blocks: - name: Push Docker image to registry task: jobs: - name: Docker Hub commands: - checkout - echo $DOCKER_PASSWORD | docker login --username "$DOCKER_USERNAME" --password-stdin - docker build -t hello:v1 . - docker tag hello:v1 "$DOCKER_USERNAME"/hello:v1 - docker push "$DOCKER_USERNAME"/hello:v1 - docker pull "$DOCKER_USERNAME"/hello:v1 - docker images secrets: - name: docker-hub
The name of the image will be
"$DOCKER_USERNAME"/hello and its tag will be
v1. Therefore, in order to
docker pull this image, you will have to use its
docker images command executed at the end of the job is an example to
verify that the desired image has been downloaded and is available for further
In this example, we are using a
docker-hub secret as defined in a
here, dealing with pulling
from a private registry.
Note that you can only use promotions to build images in certain branches. Refer to the promotions documentation and the pipeline reference for more information on orchestrating workflows.
More examples of pushing to Docker registries#
- Pushing Docker images to AWS Elastic Container Registry (ECR)
- Pushing Docker images to Google Container Registry (GCR)
Using Docker Compose#
You can use Docker Compose in your Semaphore jobs as you would on any Linux machine. For a detailed example, see Using Docker Compose in CI.
Using databases and background services#
For example, let's say that your CI build needs Redis and PostgreSQL:
# .semaphore/semaphore.yml version: v1.0 name: Docker Based Builds agent: machine: type: e1-standard-2 containers: - name: main image: 'registry.semaphoreci.com/ruby:2.6' - name: db image: 'registry.semaphoreci.com/postgres:9.6' env_vars: - name: POSTGRES_PASSWORD value: keyboard-cat - name: cache image: 'registry.semaphoreci.com/redis:5.0' blocks: - name: "Hello" task: jobs: - name: Hello commands: # install postgres and redis clients - apt-get -y update && apt-get install postgresql-client redis-tools # create a database by connecting to 'db' container - PGPASSWORD="keyboard-cat" createdb -U postgres -h db -p 5432 -e hello # list key in redis container by connecting to the cache container - redis-cli -h cache KEYS *
In this example, we used the Semaphore-hosted Postgres and Redis images to start the services.
Using services and test data across blocks#
Note that the services that you start in one job are not automatically available in other jobs, since all jobs run in isolated environments. The isolation of jobs from each other within their blocks also means that services are not shared across blocks or pipelines.
To use a service or populate test data in all parallel jobs within a block, you have to specify it in the task prologue. Repeat the same steps in the definition of each block as needed.
Installing a newer Docker version#
A recent version of Docker toolchain is preinstalled by default. In the event that there's a newer version which hasn't yet been added to Semaphore, you can use the example below to set it up:
# .semaphore/semaphore.yml version: v1.0 name: Update Docker agent: machine: type: e1-standard-2 os_image: ubuntu1804 blocks: - name: Update docker-ce task: jobs: - name: Update docker commands: - checkout - docker --version - sudo apt-get update - sudo apt-get -y -o Dpkg::Options::="--force-confnew" install docker-ce - docker --version