Job environment isolation

Self-hosted agents - closed beta

Self-hosted agents are in closed beta. If you would like to run Semaphore agents on your infrastructure, please contact us and share your use case. Our team will get back to you as soon as possible.

Even though some use cases might benefit from jobs sharing the same environment, a clean environment for each and every job might be needed for others.

Using docker containers#

Using docker containers is the fastest approach available. Creating, starting, stopping, and destroying docker containers is a very fast operation, especially if you cache your docker images in the machine running the agent.

There are two different ways that Docker containers can be used by an agent:

  • the agent itself can run inside a docker container. disconnect-after-job can be used to instruct the agent to shutdown after a job is done.
  • You can configure jobs to use docker images and the self-hosted agent type of your choice. In this scenario, the agent won't run inside a Docker container, rather it will execute jobs inside the container. This approach doesn't require shutdown-hook-path or disconnect-after-job.

If you need a clean environment for every job, the recommended approach is to use Docker containers.

Using cloud instances#

Sometimes, something other than a Docker container might be required. For instance, you may need to run your agents in AWS EC2 instances. In this case, you might need a combination of shutdown-hook-path and disconnect-after-job to properly instruct the EC2 instance to terminate after a job is done.

A part of the shutdown-hook-path script used by agent-aws-stack and executed when an agent shuts down after a job is finished is shown below:

if [[ $SEMAPHORE_AGENT_SHUTDOWN_REASON == "IDLE" ]]; then
  aws autoscaling terminate-instance-in-auto-scaling-group \
    --region "$region" \
    --instance-id "$instance_id" \
    --should-decrement-desired-capacity
else
  aws autoscaling terminate-instance-in-auto-scaling-group \
    --region "$region" \
    --instance-id "$instance_id" \
    --no-should-decrement-desired-capacity
fi

In order to rotate instances and guarantee a clean environment for every job, you can use the AWS CLI to instruct the auto scaling group to terminate the EC2 instance once the agent shuts down. AWS will then replace the EC2 instance with a new, clean one and a new agent will start up.

EC2 instance rotation

Keep in mind that rotating AWS EC2 instances is not as fast as rotating Docker containers.