Docker Hub authentication#

As announced in the Docker blog post, on November 1st 2020, Docker Hub introduced rate limits on image pulls.

Exceeding the explained rate limits will disrupt your Semaphore workflows. You can find the recommended steps to avoid it below.

Does this affect you#

Semaphore runs jobs from a shared pool of IPs and anonymous public image pulls are counted based on the IP address. This means that if you are pulling images from a public Docker Hub repository as an anonymous user, your Semaphore jobs will be affected by the DockerHub rate limit.

How Semaphore is helping#

Semaphore provides the Semaphore Container Registry which hosts some of the most frequently used Docker images. You can pull these images in your Semaphore environment without any restrictions or limitations.

If you are using a Docker-based CI/CD environment in combination with convenience images Semaphore will automatically redirect any pulls from the semaphoreci Docker Hub repository to the Semaphore Container Registry.

What should you do to minimize the effect of the rate limit#

  • Switch to Semaphore Container Registry - if the image you need is available in our Container Registry you can update your configuration to pull images from
  • Authenticate your pulls - If you have a Docker Hub account start authenticating your pulls in your Semaphore configuration.

How to authenticate Docker pulls#

Step 1: Create a Semaphore secret#

The first step is to store your Docker Hub credentials. You can use Semaphore secrets to safely store any credentials and make them available in your projects.

Creating a secret from the UI

  1. Click on the organization icon in the top right corner
  2. From the menu select Settings
  3. On the left side pick Secrets
  4. Click on New Secret
  5. Fill in a unique name for your secret
  6. Add the first environment variable: Variable name: "DOCKER_CREDENTIAL_TYPE", Value: "DockerHub"
  7. Click on + Add another and add new variable: Variable name: DOCKERHUB_USERNAME, Value:<your-dockerhub-username>
  8. Add the third environment variable: Variable name: DOCKERHUB_PASSWORD, Value:<your-dockerhub-password>
  9. Click on Save Secret

Creating a secret through CLI
Before you begin, you'll need to install the Semaphore CLI.

After connecting to your Semaphore organization update the details in the example command below and run it:

sem create secret <name-of-your-secret> \
  -e DOCKERHUB_USERNAME=<your-dockerhub-username> \
  -e DOCKERHUB_PASSWORD=<your-dockerhub-password>

Step 2: Add a secret to your pipeline YAML#

To use the newly created secret in your jobs, you first need to attach it. You can attach a secret to individual blocks in your workflow or the whole pipeline. We suggest doing the latter so that it's available to all jobs in the workflow.

You can achieve it by using global_job_config like this:

version: v1.0
name: My project
    type: e1-standard-2
    os_image: ubuntu1804

  # Connect secret to all jobs in the pipeline
    - name: <your-docker-hub-secret>


Step 3-a: Use the secret to authenticate Docker images pulls#

For your docker pulls to be authenticated you have to log into Docker Hub:

echo $DOCKERHUB_PASSWORD | docker login --username "$DOCKERHUB_USERNAME" --password-stdin

You should run this command before pulling any Docker images.

Same as with secrets, to avoid having to add this to every job or block, we suggest that you include it in the global_job_config prologue of your pipeline:

  # Execute at the start of every job in the pipeline
      - echo $DOCKERHUB_PASSWORD | docker login --username "$DOCKERHUB_USERNAME" --password-stdin

This way the docker login command will be run at the start of each job in the pipeline.

If however, you prefer to log in to Docker Hub in individual jobs only, you can do it like in this example:

# .semaphore/semaphore.yml
version: v1.0
name: Using a Docker image
    type: e1-standard-2
    os_image: ubuntu1804

  - name: Run container from Docker Hub
      - name: Authenticate docker pull
          - checkout
          - echo $DOCKERHUB_PASSWORD | docker login --username "$DOCKERHUB_USERNAME" --password-stdin
          - docker pull <repository>/<image>
          - docker images
          - docker run <repository>/<image>
      - name: docker-hub

Step 3-b: Running jobs inside a Docker image#

If you're using Docker image as your pipeline CI/CD environment, you only need to attach the docker-hub secret to your agent's properties:

    type: e1-standard-2

    - name: main
      image: <repository>/<image>

    - name: <your-docker-hub-secret>

How to check if you are logged in#

The docker login command will display a Login Succeeded message as an output if authentication was successful.

Another way to check is to open ~/.docker/config.json and check the auths field.

If you have been successfully logged in, then the auths field will be updated accordingly:

    "auths": {
        "": {....

If you are not logged in, the auths field will be empty:

    "auths": {},

How can you know if you are hitting the limit#

If you have exceeded the rate limit, Docker will throw the Too Many Requests error. Check the output of your docker pull command in the job log. If you have exceeded the rate limit, the output will be the following:

You have reached your pull rate limit. You may increase the limit by authenticating and upgrading:`

Docker published this guide that can also help in determining how close you are to reaching the rate limit.