Tests#

Semaphore collects XML test reports and uses them to provide insight into your pipelines.

With test reports, you enable your team to get an effective and consistent view of your CI/CD test suite across different test frameworks and stages in a CI/CD workflow. You get a clear failure report for each executed pipeline. Failures are extracted and highlighted, while the rest of the suite is available for analysis.

Published test results are available under the Tests tab.

Tests Tab on Workflow Page

Setting up test reports#

Semaphore supports any test runner that generates JUnit XML reports.

To collect test results in your pipelines, follow these three steps.

Step 1 — Export XML results from your test suite#

Most test runners can export JUnit XML test results. In the following list, you can find popular test runners and their formatters.

Language Test Runner Formatter Example
JavaScript Mocha mocha-junit-reporter
JavaScript Karma karma-junit-reporter
JavaScript ESLint built-in
JavaScript Jest jest-junit
Ruby RSpec rspec_junit_formatter Exporting XML reports with RSpec
Ruby Cucumber built-in
Elixir ExUnit junit-formatter Exporting XML reports from ExUnit
Go GoTestSum built-in Exporting XML reports with GoTestSum
PHP PHPUnit built-in
Python PyTest built-in
Bash Bats built-in
Rust Cargo Test junit-report
Java Maven maven-surefire

If your test runner is not the above list, you can still collect Tests. Find a test formatter for your test runner that can export JUnit XML results.

Step 2 — Publish XML results from your jobs#

Given that your test suite is exporting an XML test result named report.xml, publish it from your jobs with the test-results CLI tool.

Add the following snippet to your pipeline YAML file:

global_job_config:
  epilogue:
    always:
      commands:
         - '[[ -f report.xml ]] && test-results publish report.xml'

The test-results tool is part of Semaphore's Toolbox and it is available in every Semaphore job.

If you are running tests inside of a Docker container, see how to use test-results CLI with Docker to extract and publish the result file.

Step 3 — Collect and Merge all XML results in a pipeline#

Finally, to collect and merge XML reports from all jobs in a pipeline, add the following snippet to your pipeline YAML file:

after_pipeline:
  task:
    jobs:
      - name: Publish Results
        commands:
          - test-results gen-pipeline-report

Inspecting executed tests on the Tests dashboard#

After each executed pipeline run, your team will get access to a report under the Tests tab on the Workflow Page. What follows is a list of most common use cases.

Filter failed tests on the Tests dashboard#

By default, the Tests dashboards display only the failed tests, as they are the most useful during a typical red-green-refactor cycle.

Displaying only failed tests

Find the slowest test in your test suite#

While developing a new feature, you can introduce unwanted performance degradation. If you notice that your test suite has performance problems, finding the slowest tests can help you narrow down source of the problem.

On the Tests dashboard, select the Slowest First option to get a list of your slowest tests.

Displaying the slowest tests first

Find skipped tests#

Skipping a test is a common strategy to isolate problematic tests in your test suite. This is a good short-term strategy. In the long term however, if the number of skipped tests accumulates, your team is risking by shipping untested features.

Visiting the Tests dashboard can give you a good overview of how many tests are skipped in your test suite. Select View, and uncheck Hide skipped tests.

Displaying skipped tests

See also#