Semaphore collects XML test reports and uses them to provide insight into your pipelines.
With test reports, you and your team get an effective and consistent view of your CI/CD test suite across different test frameworks and stages in CI/CD workflows. You also get a clear failure report for each executed pipeline. Failures are extracted and highlighted, and the rest of the suite is available for analysis.
Published test results are available under the Tests tab.
Setting up test reports#
Semaphore supports any test runner that generates JUnit XML reports.
To collect test results for your pipelines, follow these three steps.
Step 1 — Export XML results from your test suite#
Most test runners can export JUnit XML test results. In the following list, you can find popular test runners and their formatters.
|Ruby||RSpec||rspec_junit_formatter||Exporting XML reports with RSpec|
|Elixir||ExUnit||junit-formatter||Exporting XML reports from ExUnit|
|Go||GoTestSum||built-in||Exporting XML reports with GoTestSum|
If your test runner is not the previous list, you can still collect Tests. All you need to do is find a test formatter for your test runner that can export JUnit XML results.
Step 2 — Publish XML results from your jobs#
Given that your test suite exports an XML test result named
you can publish it from your jobs with the
test-results CLI tool.
To do so, add the following snippet to your pipeline YAML file:
global_job_config: epilogue: always: commands: - '[[ -f report.xml ]] && test-results publish report.xml'
test-results tool is part of Semaphore's Toolbox
and it is available for every Semaphore job.
If you are running tests inside of a Docker container, see how to use test-results CLI with Docker to extract and publish the result file.
Step 3 — Collect and Merge all XML results for a pipeline#
Finally, to collect and merge XML reports from all jobs in a pipeline, add the following snippet to your pipeline YAML file:
after_pipeline: task: jobs: - name: Publish Results commands: - test-results gen-pipeline-report
Inspecting executed tests in the Tests dashboard#
After each executed pipeline run, your team will get access to a report under the Tests tab on the Workflow Page. The following is a list of the most common use cases.
Filtering failed tests in the Tests dashboard#
By default, the Tests dashboard only displays failed tests, as they are the most useful during a typical red-green-refactor cycle.
Find the slowest test in your test suite#
While developing a new feature, you can introduce unwanted performance degradation. If you notice that your test suite has performance problems, finding the slowest tests can help you narrow down the source.
In the Tests dashboard, select the Slowest First option to get a list of your tests ordered from slowest to fastest.
Find skipped tests#
Skipping a test is a common short-term strategy to isolate problematic tests in your test suite. In the long term, however, your team risks shipping features that have not been thoroughly tested if the number of skipped tests grows too large.
Visiting the Tests dashboard can give you a good overview of how many tests are being skipped in your test suite. To view skipped tests, select View and uncheck Hide skipped tests.