Install Agents

Available On

PlansStartup (Hybrid)Scaleup (Hybrid)

Overview

Before you can run jobs in your hardware, you need to install and register the self-hosted agent stack. This page explains how to install the stack on several platforms.

How to register agent type

The agent type is the name assigned to agents running on the same hardware or platform. Semaphore expects all self-hosted agents to belong to one agent type.

To register a self-hosted agent type, follow these steps:

1. Open the organization menu on the top-right corner

2. Select Self-hosted agents

3. Press Add self-hosted agent type

4. Type the name of the agent type. Self hosted agents all begin with s1-, e.g. s1-gpu-2

5. Select the how the agent name is assigned

6. Select when the agent name is released

7. Press Register

   

The next page shows detailed instructions to install and register the self-hosted. Select the platform you're using and press Reveal to view the registration token. Save it in a safe place for the next step.

Follow the on-screen instructions. See how to install agent stack for more options.

When the agent connects you should be able to see it on the self-hosted agents page.

Agent name assignment

Every agent must have a unique name. There are two ways in which the name can be assigned:

• Name assigned by agent: the agent provides its own name
• Name assigned by AWS STS: agents running on AWS can get an additional level of security by validating their name with AWS Security Token Service

Name assigned by the agent

In this mode, the agent selects its own name and sends it during registration.

The agent picks a random name by default. You can override the name by specifying it during service start. For example:

Agent start with name

agent start --name my-agent-name

You may need to edit the configuration file of the service manager to change the name. For example, to change the agent name when using systemd, follow these steps:

1. Edit the service file

   sudo systemctl edit semaphore-agent.service

2. Override ExecStart to add the --name argument and save the file

   Add an override to the systemd service file

   [Service]
   ExecStart=
   ExecStart=/opt/semaphore/agent/agent start --config-file /opt/semaphore/agent/config.yaml --name my-agent-name

3. Restart the service and confirm it has successfully registered

   sudo systemctl restart semaphore-agent

note

Agent names should have between 8 and 80 characters.

Name assigned by AWS STS

AWS Security Token Service provides a second layer of security that ensures only allowed agents can connect with Semaphore.

To use this option, you must run your agent in AWS EC2 instances. The Autoscaling AWS Stack uses this feature by default.

When AWS STS is enabled, the agent sends a name request to Semaphore during registration, which in turn validates the access with the AWS-secured endpoint. This mechanism thwarts attempts to register rogue agents even if the attacked has secured access to a valid registration token.

To use AWS STS name assignments, follow these steps:

1. Create an IAM user on AWS with permissions to create and delete EC2 instances

2. Select the option Agent name is assigned from a pre-signed AWS STS GetCallerIdentity URL during agent registration

3. Type your AWS account ID

4. Type the list of roles the IAM user is allowed to assume

5. Press Save

   

Name release

You can select what happens when the agent name when it disconnects. The default behavior is to release for reuse the name immediately after disconnection.

On some cases, however, you may want to keep the name reserved, for example:

• If you want to connect to self-hosted agent machine to troubleshoot some issue
• If you're using AWS STS endpoints, you may want to reserve the name for the duration of the EC2 instance

You can select how long to reserve the agent name during agent registration.

How to install agent stack

The Semaphore self-hosted agent is open source. The prerequisites to install the self-hosted agent stack are:

• Git
• Bash (Linux and macOS) or PowerShell (Windows)
• A local user with sudo powers and Docker management permissions (Linux or macOS)
• A user with sudo powers (Linux and macOS)
• Docker running as non-root (Linux and macOS)
• Docker Compose (Linux and macOS)

Scroll down to learn how to install the stack in your hardware.

Kubernetes

Semaphore provides Helm charts to run agents in a Kubernetes cluster.

To install the Semaphore custom controller, follow these steps:

1. Install Helm

2. Add the Semaphore Helm chart

   Add Helm repository

   helm repo add renderedtext https://renderedtext.github.io/helm-charts

3. Install the agent-k8s-controller with Helm

   Install agent-k8s-controller

   helm upgrade --install semaphore-controller renderedtext/controller \
   --namespace semaphore \
   --create-namespace \
   --set endpoint=my-org.semaphoreci.com \
   --set apiToken=<token>

   Replace:

   • <my-org.semaphoreci.com> with your organization URL
   • <token> with a valid Semaphore API token

4. Create a secret to register the agent type in the Kubernetes cluster. Create a new YAML resource file.

   semaphore-secret.yml

   apiVersion: v1
   kind: Secret
   metadata:
       name: my-semaphore-agent-type
       namespace: semaphore
       labels:
           semaphoreci.com/resource-type: agent-type-configuration
   stringData:
       agentTypeName: <AGENT_TYPE>
       registrationToken: <AGENT_TYPE_REGISTRATION_TOKEN>

   Replace:

   • <AGENT_TYPE> with the type you created during registration
   • <AGENT_TYPE_REGISTRATION_TOKEN> with the token you received during registration
   • The custom controllers looks for the label shown below to know what secret is relevant to this connection

5. Create the secret in Kubernetes

   kubectl apply -f semaphore-secret.yml

6. Create an S3 bucket as documented in the self-host cache page

7. Create a secret with the following parameters:

   • Environment variable SEMAPHORE_CACHE_BACKEND=s3
   • Environment variable SEMAPHORE_CACHE_S3_BUCKET=<your-S3-bucket-name>
   • New file /root/.aws/credentials upload the following file from your local machine: $HOME/.aws/credentials
   • New file /root/.aws/config upload the following file from your local machine: $HOME/.aws/config

8. Enable the secret in all blocks where you need to use the cache

The Helm chart provides a few additional configuration options so you can tweak your installation to best suit your needs. Run helm show values renderedtext/controller to view all available settings.

See the Helm chart repo to learn more.

Ubuntu/Debian

Follow these steps to install self-hosted agent in Ubuntu or Debian:

1. Create a user with sudo permissions to run the agent service, e.g. semaphore

   sudo adduser semaphore
   sudo adduser semaphore sudo

2. Log in or switch to the agent service user

   su - semaphore

3. Prepare the machine

   Prepare machine

   sudo mkdir -p /opt/semaphore/agent
   sudo chown $USER:$USER /opt/semaphore/agent/
   cd /opt/semaphore/agent

4. Download the agent package. Find the latest release for your platform and architecture

   Download agent package

   curl -L https://github.com/semaphoreci/agent/releases/download/v2.2.23/agent_Linux_x86_64.tar.gz -o agent.tar.gz
   tar -xf agent.tar.gz

5. Install the agent and follow the prompts. Type the organization name, the registration token and the name of the local service user, e.g. semaphore . The registration token is the one revealed during agent registration

   Install agent

   $ sudo ./install.sh
   Enter organization: my-org
   Enter registration token: <access token>
   Enter user [root]: <local-service-user>
   Downloading toolbox from https://github.com/semaphoreci/toolbox/releases/latest/download/self-hosted-linux.tar...
   [sudo] password for semaphore: 
   Creating agent config file at /opt/semaphore/agent/config.yaml...
   Creating /etc/systemd/system/semaphore-agent.service...
   Starting semaphore-agent service...

6. Add GitHub and BitBucket SSH fingerprints

   sudo mkdir -p /home/$USER/.ssh
   sudo chown -R $USER:$USER /home/$USER/.ssh
   
   curl -sL https://api.github.com/meta | jq -r ".ssh_keys[]" | sed 's/^/github.com /' | tee -a /home/$USER/.ssh/known_hosts
   curl -sL https://bitbucket.org/site/ssh | tee -a /home/$USER/.ssh/known_hosts
   
   chmod 700 /home/$USER/.ssh
   chmod 600 /home/$USER/.ssh/known_hosts

7. Add your SSH private keys into the ~/.ssh/ folder

8. Test SSH connection to GitHub or BitBucket

   Testing SSH connection

   ssh -T git@bitbucket.org
   ssh -T git@github.com

9. Restart the agent service

   Restart agent service

   sudo systemctl restart semaphore-agent

10. Check that the agent is working and is connected

    Checking self-hosted agent status

    $ sudo systemctl status semaphore-agent
    ● semaphore-agent.service - Semaphore agent
        Loaded: loaded (/etc/systemd/system/semaphore-agent.service; disabled; preset: enabled)
        Active: active (running) since Fri 2024-07-12 14:09:28 UTC; 10s ago
    Main PID: 5154 (agent)
        Tasks: 11 (limit: 509)
        Memory: 13.0M (peak: 13.8M)
            CPU: 77ms
        CGroup: /system.slice/semaphore-agent.service
                ├─5154 /opt/semaphore/agent/agent start --config-file /opt/semaphore/agent/config.yaml
                └─5157 /opt/semaphore/agent/agent start --config-file /opt/semaphore/agent/config.yaml
    
    Jul 12 14:09:28 selfhosted agent[5157]: Jul 12 14:09:28.345 sywinVS8IgIkZzgIgk2D : Starting to poll for jobs
    Jul 12 14:09:28 selfhosted agent[5157]: Jul 12 14:09:28.345 sywinVS8IgIkZzgIgk2D : SYNC request (state: waiting-for-jobs)
    Jul 12 14:09:28 selfhosted agent[5157]: Jul 12 14:09:28.442 sywinVS8IgIkZzgIgk2D : SYNC response (action: continue)
    Jul 12 14:09:28 selfhosted agent[5157]: Jul 12 14:09:28.442 sywinVS8IgIkZzgIgk2D : Waiting 4.888s for next sync...
    Jul 12 14:09:33 selfhosted agent[5157]: Jul 12 14:09:33.331 sywinVS8IgIkZzgIgk2D : SYNC request (state: waiting-for-jobs)

See self-hosted agent configuration to see the next steps in the setup. If the installation fails, try the Generic Linux installation or installing from source.

Generic Linux

Follow these steps to install self-hosted agent in any Linux distribution:

1. Create a user to run the agent service with sudo permissions, e.g. semaphore

2. Log in or switch to the agent service user

   su - semaphore

3. Prepare the machine

   Prepare machine

   sudo mkdir -p /opt/semaphore/agent
   sudo chown $USER:$USER /opt/semaphore/agent/
   cd /opt/semaphore/agent

4. Create the configuration file for the agent. Replace the endpoint with your organization URL (without HTTPS), and the registration token revealed during agent registration

   Create config file

   cat > config.yaml <<EOF
   endpoint: "my-org.semaphoreci.com"
   token: "[token]"
   EOF

5. Download and install the Semaphore toolbox. Select the correct platform and architecture

   Install Semaphore toolbox

   curl -L "https://github.com/semaphoreci/toolbox/releases/latest/download/self-hosted-linux.tar" -o toolbox.tar
   tar -xf toolbox.tar
   mv toolbox ~/.toolbox
   bash ~/.toolbox/install-toolbox
   source ~/.toolbox/toolbox
   echo "source ~/.toolbox/toolbox" >> ~/.bash_profile

6. Download the agent package. Find the latest release for your platform and architecture

   Download agent package

   curl -L https://github.com/semaphoreci/agent/releases/download/v2.2.23/agent_Linux_x86_64.tar.gz -o agent.tar.gz
   tar -xf agent.tar.gz

7. Add GitHub and BitBucket SSH fingerprints

   sudo mkdir -p /home/$USER/.ssh
   sudo chown -R $USER:$USER /home/$USER/.ssh
   
   curl -sL https://api.github.com/meta | jq -r ".ssh_keys[]" | sed 's/^/github.com /' | tee -a /home/$USER/.ssh/known_hosts
   curl -sL https://bitbucket.org/site/ssh | tee -a /home/$USER/.ssh/known_hosts
   
   chmod 700 /home/$USER/.ssh
   chmod 600 /home/$USER/.ssh/known_hosts

8. Add your SSH private keys into the ~/.ssh/ folder

9. Test SSH connection to GitHub or BitBucket

   Testing SSH connection

   ssh -T git@bitbucket.org
   ssh -T git@github.com

10. Start the agent

    Start the agent

    agent start --config-file config.yaml

See self-hosted agent configuration to see the next steps in the setup.

Red Hat Enterprise Linux (RHEL)

In order to run the self-hosted agent in FIPS-enabled Red Hat Enterprise Linux (RHEL), you must compile the binary from source.

To compile the agent, follow these steps:

1. Verify the host has FIPS mode enabled

   sudo fips-mode-setup --check

2. Install the go-toolset

   sudo yum install go-toolset

3. Create a user for the Semaphore service with sudo permissions

   adduser semaphore
   passwd semaphore
   usermod -aG wheel semaphore
   su - semaphore

4. Download the source and compile it. Find the latest release and download the source package

   curl -L https://github.com/semaphoreci/agent/archive/refs/tags/v2.2.23.tar.gz -o agent.tar.gz
   tar -xf agent.tar.gz
   cd agent-2.2.23
   make build

5. Verify that the binary is FIPS compatible

   go tool nm ./build/agent | grep FIPS

6. Install and follow the prompts

   make install
   $ sudo ./install
   Enter organization: my-org
   Enter registration token: <access token>
   Enter user [root]: <local-service-user>
   Downloading toolbox from https://github.com/semaphoreci/toolbox/releases/latest/download/self-hosted-linux.tar...
   [sudo] password for semaphore: 
   Creating agent config file at /opt/semaphore/agent/config.yaml...
   Creating /etc/systemd/system/semaphore-agent.service...
   Starting semaphore-agent service...

7. Add GitHub and BitBucket SSH fingerprints

   sudo mkdir -p /home/$USER/.ssh
   sudo chown -R $USER:$USER /home/$USER/.ssh
   
   curl -sL https://api.github.com/meta | jq -r ".ssh_keys[]" | sed 's/^/github.com /' | tee -a /home/$USER/.ssh/known_hosts
   curl -sL https://bitbucket.org/site/ssh | tee -a /home/$USER/.ssh/known_hosts
   
   chmod 700 /home/$USER/.ssh
   chmod 600 /home/$USER/.ssh/known_hosts

8. Add your SSH private keys into the ~/.ssh/ folder

9. Test SSH connection to GitHub or BitBucket

   Testing SSH connection

   ssh -T git@bitbucket.org
   ssh -T git@github.com

macOS

To install the Semaphore self-hosted agent in macOS

1. Create a user to run the agent service with sudo permissions, e.g. semaphore

2. Log in or switch to the agent service user

   su - semaphore

3. Prepare the machine

   Prepare machine

   sudo mkdir -p /opt/semaphore/agent
   sudo chown $USER:$USER /opt/semaphore/agent/
   cd /opt/semaphore/agent

4. Download the agent package. Find the latest release for your platform and architecture (x86_64 or arm64)

   Download agent package

   curl -L https://github.com/semaphoreci/agent/releases/download/v2.2.23/agent_Darwin_arm64.tar.gz -o agent.tar.gz
   tar -xf agent.tar.gz

5. Install the agent and follow the prompts. Type the organization URL, the registration token and the name of the local service user. The registration token is the one revealed during agent registration

   Install agent

   $ sudo ./install.sh
   Enter organization: my-org.semaphoreci.com
   Enter registration token: <access token>
   Enter user [root]: <local-service-user>
   Downloading toolbox from https://github.com/semaphoreci/toolbox/releases/latest/download/darwin.tar...
   [sudo] password for semaphore: 
   Creating agent config file at /opt/semaphore/agent/config.yaml...
   Creating /etc/systemd/system/semaphore-agent.service...
   Starting semaphore-agent service...

6. Add GitHub and BitBucket SSH fingerprints

   sudo mkdir -p /home/$USER/.ssh
   sudo chown -R $USER:$USER /home/$USER/.ssh
   
   curl -sL https://api.github.com/meta | jq -r ".ssh_keys[]" | sed 's/^/github.com /' | tee -a /home/$USER/.ssh/known_hosts
   curl -sL https://bitbucket.org/site/ssh | tee -a /home/$USER/.ssh/known_hosts
   
   chmod 700 /home/$USER/.ssh
   chmod 600 /home/$USER/.ssh/known_hosts

7. Add your SSH private keys into the ~/.ssh/ folder

8. Test SSH connection to GitHub or BitBucket

   Testing SSH connection

   ssh -T git@bitbucket.org
   ssh -T git@github.com

macOS (Homebrew)

You can install the macOS agent using Homebrew. Follow these steps:

1. Install Homebrew

2. Install the Semaphore self-hosted agent

   brew install semaphoreci/tap/agent

3. Download and install the Semaphore toolbox. Select the correct platform and architecture

   Install Semaphore toolbox

   curl -L "https://github.com/semaphoreci/toolbox/releases/download/v1.21.19/darwin-arm.tar" -o toolbox.tar
   tar -xf toolbox.tar
   mv toolbox ~/.toolbox
   bash ~/.toolbox/install-toolbox
   source ~/.toolbox/toolbox
   echo "source ~/.toolbox/toolbox" >> ~/.bash_profile

4. Add GitHub and BitBucket SSH fingerprints

   sudo mkdir -p /home/$USER/.ssh
   sudo chown -R $USER:$USER /home/$USER/.ssh
   
   curl -sL https://api.github.com/meta | jq -r ".ssh_keys[]" | sed 's/^/github.com /' | tee -a /home/$USER/.ssh/known_hosts
   curl -sL https://bitbucket.org/site/ssh | tee -a /home/$USER/.ssh/known_hosts
   
   chmod 700 /home/$USER/.ssh
   chmod 600 /home/$USER/.ssh/known_hosts

5. Add your SSH private keys into the ~/.ssh/ folder

6. Test SSH connection to GitHub or BitBucket

   Testing SSH connection

   ssh -T git@bitbucket.org
   ssh -T git@github.com

7. Start the agent. Replace the endpoint with your organization URL. Type the registration token you received earlier

   Start the agent

   agent start --endpoint my-org.semaphoreci.com --token <TOKEN>

Windows

To install the self-hosted agent in Windows, follow these steps:

1. Prepare your machine

   New-Item -ItemType Directory -Path C:\semaphore-agent
   Set-Location C:\semaphore-agent

2. Download the agent. Find the latest release for your platform and architecture

   Invoke-WebRequest "https://github.com/semaphoreci/agent/releases/download/v2.2.23/agent_Windows_x86_64.tar.gz" -OutFile agent.tar.gz
   tar.exe xvf agent.tar.gz

3. Install the agent and follow the prompts

   $env:SemaphoreEndpoint = "<your-organization>.semaphoreci.com"
   $env:SemaphoreRegistrationToken = "<your-agent-type-registration-token>"
   .\install.ps1

AWS Autoscaler

With AWS (or any other cloud), you can spin up an EC2 instance and install the Ubuntu, Linux, macOS, or Windows agents.

Semaphore, however, also provides an AWS stack to run an auto-scaling fleet of agents. To learn how this feature works, see the Autoscaling agents in AWS page.

See also

• How to configure self-hosted agents
• How to run an autoscaling fleet of agents in AWS
• Self-hosted agents configuration reference