This section describes commonly used terms relating to running, inspecting, and reporting about tests.
A single test within a test suite. Contains assertions that the system under test can either pass or fail.
A file that contains several test cases. In dynamic languages, like Ruby, filtering and grouping can be done by test files.
A tag or a label that is added to the test case, for example the "wip" tag, usually used for filtering tests.
A collection of test cases.
A single execution of the test suite or test case.
The tool used for running tests. Example: RSpec, GoTestSum, ExUnit.
Usually, tests are written for a specific test runner in mind. For example, RSpec tests are written for RSpec test runners, and Cucumber tests are written for Cucumber test runners.
In some cases, multiple runners exist for the same test format. For example,
in the Golang ecosystem you can run tests with
go test or with
A mechanism in test runners for filtering which test cases are executed. Filtering can be based on file paths, tags, or other properties associated with test cases.
The results of test execution.
A single test case can have the following result:
|Passed||The test passed the assertions in the test case.|
|Failed||The test failed the assertions in the test case.|
|Error||The test case failed to execute. Assertions were not tested.|
|Timeout||The test case failed to execute in a preset interval. Assertions were not tested.|
|Skipped||The test case was skipped and has no result.|
Test results (plural) refer to the aggregate result of all executed test cases in a test suite. It can be one of the following:
|Failed||At least one test case has the result failed, error, or timeout.|
|Passed||No test case has is the result failed, error, or timeout. Acceptable results are passed and skipped.|
Test output formatter#
The formatter used for formatting test results. Common formatters are:
- "dot" format
- "documentation" format
- "junit" format
Test runners come with several built-in formatters, and usually, they can be extended with additional formatters via plugging.
Test results CLI#
A CLI tool included in the Semaphore Toolbox used to generate and publish test reports.
Learn more about this CLI tool on the Test Results CLI reference documentation page.
A report that was generated and published with the test-results CLI tool. Published reports are available on the workflow page and job page under the Tests tab.
The Tests tab contains multiple test reports, usually one for each test suite.