Requirement coverage analysis is a key aspect in Xray

A requirement may be either covered by one or multiple Tests. In fact, the status of a given requirement goes way further than the basic covered/not covered information: it will take into account your test results.

As soon as you start running your Tests, the individual test execution result may be one of many and be very specific to your use case.

To make your analysis even more complex, you may be using sub-requirements and executing related Tests.

Thus, how do all these factors contribute to the calculation of a requirement status?

Let’s start by detailing the different possible values for the requirement, Test and Test step statuses. In the end, we’ll see how they’ll impact on the calculation of the coverage status of a requirement in some specific version.

 

Requirement statuses

In Xray, for a given requirement, its coverage status may be:

  • OK – All the Tests associated with the Requirement are PASSED
  • NOK – At least one Test associated with the Requirement is FAILED
  • NOT RUN – At least one Test associated with the Requirement is TODO or ABORTED and there are no Tests with status FAILED
  • UNKNOWN – At least one Test associated with the Requirement is UNKNOWN and there are no Tests with status FAILED
  • UNCOVERED – The Requirement has no Tests associated

It’s not possible to create custom requirement statuses, as this value is computed as mentioned above.

You can see that in order to calculate a requirement’s coverage status, for some specific system version, we “just” need to take into account the status of the related Tests for that same version. We’ll come back to this later on.

 

Test statuses

Xray provides some built-in Test Run statuses (which can’t be modified nor deleted):

  • TODO – Test is pending execution; this is a non-final status;
  • EXECUTING – Test is being executed; this is a non-final status;
  • FAIL – Test failed
  • ABORTED – Test was aborted
  • PASS – Test passed successfully

Each of this status maps to a requirement status, accordingly with the following table.

test status

 

The status (i.e. result) of a Test Run is an attribute of the Test Run (a “Test Run” is an instance of a Test and is not a JIRA issue) and is the one taken into account to assess the status of the requirement.

On the other hand, a completely different thing is the TestRunStatus custom field, which is a calculated field that belongs to the Test issue and that takes into account several Test Runs; the TestRunStatus does not affect the calculation of the status of requirements.

 

Managing Test Statuses

Creating new Test Run statuses may be done in the Manage Test Statuses configuration section of Xray.

Whenever creating/editing a Test status, we have to identify the Requirement status we want this Test status to map to.

teststatus

 

One important attribute of a Test status is the “final” attribute. If “Final Statuses have precedence over non-final” flag is enabled, then only Test Runs in final statuses are considered for the calculation of the requirement status.

Only final statuses will appear in the TestRunStatus custom field as well.

 

Test Step statuses

Creating new Test Step statuses may be done in the Manage Test Step Statuses configuration section of Xray.

table test step statuses

 

Whenever creating/editing a Test Step status, we have to identify the Test status we want this step status to map to.

edit test step status

Note that native Test Step statuses can’t be modified nor deleted.

 

Calculation of the status of a given Requirement

It is possible to calculate the status of a Requirement either by Version or Test Plan.

By Version:

For a given requirement X, in order to calculate the coverage status for version V, we need to evaluate the related Tests statuses that were executed on that same version V.

By Test Plan:

For a given requirement X, in order to calculate the coverage status for Test Plan TP, we need to evaluate the related Tests statuses that were executed on Test Executions associated with Test Plan TP.

 

The algorithm is:

  • Obtain the list of Tests that directly or indirectly through Sub-Requirements (info here) cover the requirement
  • Calculate the Test status for all the Tests, in version V or Test Plan TP
    • This takes into account Test Runs in version V (as a result of Test Executions in version V) or Test Runs in Test Plan TP (within Test Executions associated with Test Plan TP)
    • If a specific Environment is also chosen, then only Test Runs from Test Executions with this Environment will be considered. In case no Environment is specified then all Test Executions are considered. (info here).
  • Calculate the overall requirement status accordingly with the rules explained initially in the “Requirement statuses”, aggregating all Test Results.

 

Note: do not mix up the status of a given requirement with the Requirement Status custom field, which shows the status of a given requirement for a specific version, depending on the configuration under Custom Field Preferences.

 

In sum…

The status of a requirement in some Version or Test Plan depends on the status of the related Tests. However, this is affected by several factors such as Test Environments and the existence or not of sub-requirements.

Xray flexibility allows customization of the allowed Test and Test Step statuses which in turn will affect the status of requirements.

 

Learn more…

 

FacebookTwitterLinkedIn

9 comments

  • Hello,

    I have a question.
    I have 2 requirements: ReqA and ReqB
    I create 2 subtests-executions and I link same test cases to both, because the test case is the same, but it changes the environment
    I execute first test execution, and it fails for ReqA, but automatically ReqB is marked as NOK, but actually ReqB is OK because the test works OK for this specific environment.
    From my point of view, as the test run status depends on the “test execution” should not be shared for requirements, because it does not allow re usability.
    For this specific situation, I have to duplicate the test cases to avoid falsify of the requirement status.
    Do I have another workaround to overpass this situation?
    Thanks and regards

      • Hi,

        I have the same question as Ana and I think this a general question and would appreciate an answer here for many to read.

        If you do not want to use Fixed Versions because you for example implement continuous delivery, the behaviour described by Ana stops you from re-use of Tests, one of the very central advantages using Xray.

        How to make TestRun statuses affect only related Requirements? Not like now, when the status of the *Tests* hits *all* requirements independent of TestRun. Shouldn’t each Test Execution (and its TestRuns) affect only related Requirements?

        Regards,
        Johan

        • Hi Ana and Johan,
          let me give feedback to your previous questions.

          Whenever you report a result to a given Test Run (i.e. a Test in the context of a Test Execution), by default that result will affect the calculation of all the requirements that are being validated by Test.
          However, there’s a setting named “Separation of Concerns”, available in the “Requirement Coverage” section of Xray administration settings, that you can use in order to fulfill your needs. If you uncheck it, you can report how it will affect the calculation of the status of all the linked requirements; and you can do it, independently for each requirement.

          More info about it here:
          https://confluence.xpand-addons.com/display/XRAY/Requirements+Coverage#RequirementsCoverage-SeparationofConcerns

          Also, another thing to keep in mind is that Xray supports Test Environments.. so you can also report results and track the results independently for each Test Environment. More info here: https://confluence.xpand-addons.com/display/XRAY/Working+with+Test+Environments

          If you’re using CI+CD and using either Jenkins/Bamboo/REST API in order to submit results, you can specify the version under test and also the Test Environment; however you can’t specify how you want each Test Run result to affect the related requirements. The only way to overcome this, is to have different Tests for validating properly and more specificaly each requirement.
          If you’re executing a Test linked to multiple requirements and you want to affect the related requirements status calculation, the best would be to decompose your test scenarios further so you have a more clear idea of what is being tested exactly by each Test.
          Regards,
          Sergio

          • Hi Sérgio!

            Thank you very much for a comprehensive and clarifying answer!

            We will try the separation of concern configuration. Otherwise I think we have to consider implement Fixed versions in our CD workflow.

            Test environments is great! It works very well for us in our workflow.

            Regards,
            Johan

  • This is regarding X-Ray Requirement Status. A story from DB2 DBA Team’s JIRA Project was showing Requirement Status and is set to Uncovered. Db2 DBA team doesn’t use X-Ray as they don’t have any application to test. How can I make this go away. I was told this field is for X-Ray testing status. Appreciate your support and help.

    • Hello Mana,

      This means that you have enabled on your project the “Requirement Coverage Analysis”. To disable the “Requirement Coverage analysis”, you should go to the Project Settings -> Actions (button top of the page) -> Disable Xray Requirement Coverage.

      If you have any problem, please raise a support ticket here.

      Thank you,
      Xray Team

Leave a reply