Semaphore collects XML test reports and uses them to provide insight into your pipelines.
With test reports, you enable your team to get an effective and consistent view of your CI/CD test suite across different test frameworks and stages in a CI/CD workflow. You get a clear failure report for each executed pipeline. Failures are extracted and highlighted, while the rest of the suite is available for analysis.
Published test results are available under the Tests tab.
Setting up test reports#
Semaphore supports any test runner that generates JUnit XML reports.
To collect test results in your pipelines, follow these three steps.
Step 1 — Export XML results from your test suite#
Most test runners can export JUnit XML test results. In the following list, you can find popular test runners and their formatters.
|Ruby||RSpec||rspec_junit_formatter||Exporting XML reports with RSpec|
|Elixir||ExUnit||junit-formatter||Exporting XML reports from ExUnit|
|Go||GoTestSum||built-in||Exporting XML reports with GoTestSum|
If your test runner is not the above list, you can still collect Tests. Find a test formatter for your test runner that can export JUnit XML results.
Step 2 — Publish XML results from your jobs#
Given that your test suite is exporting an XML test result named
publish it from your jobs with the
test-results CLI tool.
Add the following snippet to your pipeline YAML file:
global_job_config: epilogue: always: commands: - '[[ -f report.xml ]] && test-results publish report.xml'
test-results tool is part of Semaphore's Toolbox
and it is available in every Semaphore job.
If you are running tests inside of a Docker container, see how to use test-results CLI with Docker to extract and publish the result file.
Step 3 — Collect and Merge all XML results in a pipeline#
Finally, to collect and merge XML reports from all jobs in a pipeline, add the following snippet to your pipeline YAML file:
after_pipeline: task: jobs: - name: Publish Results commands: - test-results gen-pipeline-report
Inspecting executed tests on the Tests dashboard#
After each executed pipeline run, your team will get access to a report under the Tests tab on the Workflow Page. What follows is a list of most common use cases.
Filter failed tests on the Tests dashboard#
By default, the Tests dashboards display only the failed tests, as they are the most useful during a typical red-green-refactor cycle.
Find the slowest test in your test suite#
While developing a new feature, you can introduce unwanted performance degradation. If you notice that your test suite has performance problems, finding the slowest tests can help you narrow down source of the problem.
On the Tests dashboard, select the Slowest First option to get a list of your slowest tests.
Find skipped tests#
Skipping a test is a common strategy to isolate problematic tests in your test suite. This is a good short-term strategy. In the long term however, if the number of skipped tests accumulates, your team is risking by shipping untested features.
Visiting the Tests dashboard can give you a good overview of how many tests are skipped in your test suite. Select View, and uncheck Hide skipped tests.