Tests#

Semaphore collects XML test reports and uses them to provide insight into your pipelines.

With test reports, you and your team get an effective and consistent view of your CI/CD test suite across different test frameworks and stages in CI/CD workflows. You also get a clear failure report for each executed pipeline. Failures are extracted and highlighted, and the rest of the suite is available for analysis.

Published test results are available under the Tests tab.

Tests Tab on Workflow Page

Setting up test reports#

Semaphore supports any test runner that generates JUnit XML reports.

To collect test results for your pipelines, follow these three steps.

Step 1 — Export XML results from your test suite#

Most test runners can export JUnit XML test results. In the following list, you can find popular test runners and their formatters.

Language Test Runner Formatter Example
JavaScript Mocha mocha-junit-reporter
JavaScript Karma karma-junit-reporter
JavaScript ESLint built-in
JavaScript Jest jest-junit
Ruby RSpec rspec_junit_formatter Exporting XML reports with RSpec
Ruby Cucumber built-in
Elixir ExUnit junit-formatter Exporting XML reports from ExUnit
Go GoTestSum built-in Exporting XML reports with GoTestSum
PHP PHPUnit built-in
Python PyTest built-in
Bash Bats built-in
Rust Cargo Test junit-report
Java Maven maven-surefire

If your test runner is not the previous list, you can still collect Tests. All you need to do is find a test formatter for your test runner that can export JUnit XML results.

Step 2 — Publish XML results from your jobs#

Given that your test suite exports an XML test result named report.xml, you can publish it from your jobs with the test-results CLI tool.

To do so, add the following snippet to your pipeline YAML file:

global_job_config:
  epilogue:
    always:
      commands:
         - '[[ -f report.xml ]] && test-results publish report.xml'

The test-results tool is part of Semaphore's Toolbox and it is available for every Semaphore job.

If you are running tests inside of a Docker container, see how to use test-results CLI with Docker to extract and publish the result file.

Step 3 — Collect and Merge all XML results for a pipeline#

Finally, to collect and merge XML reports from all jobs in a pipeline, add the following snippet to your pipeline YAML file:

after_pipeline:
  task:
    jobs:
      - name: Publish Results
        commands:
          - test-results gen-pipeline-report

Inspecting executed tests in the Tests dashboard#

After each executed pipeline run, your team will get access to a report under the Tests tab on the Workflow Page. The following is a list of the most common use cases.

Filtering failed tests in the Tests dashboard#

By default, the Tests dashboard only displays failed tests, as they are the most useful during a typical red-green-refactor cycle.

Displaying only failed tests

Find the slowest test in your test suite#

While developing a new feature, you can introduce unwanted performance degradation. If you notice that your test suite has performance problems, finding the slowest tests can help you narrow down the source.

In the Tests dashboard, select the Slowest First option to get a list of your tests ordered from slowest to fastest.

Displaying the slowest tests first

Find skipped tests#

Skipping a test is a common short-term strategy to isolate problematic tests in your test suite. In the long term, however, your team risks shipping features that have not been thoroughly tested if the number of skipped tests grows too large.

Visiting the Tests dashboard can give you a good overview of how many tests are being skipped in your test suite. To view skipped tests, select View and uncheck Hide skipped tests.

Displaying skipped tests

See also#