Requirement coverage analysis is a key aspect in Xray
A requirement may be either covered by one or multiple Tests. In fact, the status of a given requirement goes way further than the basic covered/not covered information: it will take into account your test results.
As soon as you start running your Tests, the individual test execution result may be one of many and be very specific to your use case.
To make your analysis even more complex, you may be using sub-requirements and executing related Tests.
Thus, how do all these factors contribute to the calculation of a requirement status?
Let’s start by detailing the different possible values for the requirement, Test and Test step statuses. In the end, we’ll see how they’ll impact on the calculation of the coverage status of a requirement in some specific version.
In Xray, for a given requirement, its coverage status may be:
- OK – All the Tests associated with the Requirement are PASSED
- NOK – At least one Test associated with the Requirement is FAILED
- NOT RUN – At least one Test associated with the Requirement is TODO or ABORTED and there are no Tests with status FAILED
- UNKNOWN – At least one Test associated with the Requirement is UNKNOWN and there are no Tests with status FAILED
- UNCOVERED – The Requirement has no Tests associated
It’s not possible to create custom requirement statuses, as this value is computed as mentioned above.
You can see that in order to calculate a requirement’s coverage status, for some specific system version, we “just” need to take into account the status of the related Tests for that same version. We’ll come back to this later on.
Xray provides some built-in Test Run statuses (which can’t be modified nor deleted):
- TODO – Test is pending execution; this is a non-final status;
- EXECUTING – Test is being executed; this is a non-final status;
- FAIL – Test failed
- ABORTED – Test was aborted
- PASS – Test passed successfully
Each of this status maps to a requirement status, accordingly with the following table.
The status (i.e. result) of a Test Run is an attribute of the Test Run (a “Test Run” is an instance of a Test and is not a JIRA issue) and is the one taken into account to assess the status of the requirement.
On the other hand, a completely different thing is the TestRunStatus custom field, which is a calculated field that belongs to the Test issue and that takes into account several Test Runs; the TestRunStatus does not affect the calculation of the status of requirements.
Managing Test Statuses
Creating new Test Run statuses may be done in the Manage Test Statuses configuration section of Xray.
Whenever creating/editing a Test status, we have to identify the Requirement status we want this Test status to map to.
One important attribute of a Test status is the “final” attribute. If “Final Statuses have precedence over non-final” flag is enabled, then only Test Runs in final statuses are considered for the calculation of the requirement status.
Only final statuses will appear in the TestRunStatus custom field as well.
Test Step statuses
Creating new Test Step statuses may be done in the Manage Test Step Statuses configuration section of Xray.
Whenever creating/editing a Test Step status, we have to identify the Test status we want this step status to map to.
Note that native Test Step statuses can’t be modified nor deleted.
Calculation of the status of a given Requirement
It is possible to calculate the status of a Requirement either by Version or Test Plan.
For a given requirement X, in order to calculate the coverage status for version V, we need to evaluate the related Tests statuses that were executed on that same version V.
By Test Plan:
For a given requirement X, in order to calculate the coverage status for Test Plan TP, we need to evaluate the related Tests statuses that were executed on Test Executions associated with Test Plan TP.
The algorithm is:
- Obtain the list of Tests that directly or indirectly through Sub-Requirements (info here) cover the requirement
- This depends on the Requirement Coverage configuration
- Calculate the Test status for all the Tests, in version V or Test Plan TP
- This takes into account Test Runs in version V (as a result of Test Executions in version V) or Test Runs in Test Plan TP (within Test Executions associated with Test Plan TP)
- If a specific Environment is also chosen, then only Test Runs from Test Executions with this Environment will be considered. In case no Environment is specified then all Test Executions are considered. (info here).
- Calculate the overall requirement status accordingly with the rules explained initially in the “Requirement statuses”, aggregating all Test Results.
Note: do not mix up the status of a given requirement with the Requirement Status custom field, which shows the status of a given requirement for a specific version, depending on the configuration under Custom Field Preferences.
The status of a requirement in some Version or Test Plan depends on the status of the related Tests. However, this is affected by several factors such as Test Environments and the existence or not of sub-requirements.
Xray flexibility allows customization of the allowed Test and Test Step statuses which in turn will affect the status of requirements.