Hardware and software systems often involve numerous parameters that influence a system’s output(s). Those parameters can represent input, environment, or usage patterns and can have a finite or infinite number of options/values.
For instance, on a booking flights website, the Flying From, Flying to, Class, (number of) Adults, and (number of) Children input parameters can easily provide hundreds of different combinations for creating tests.
In the case of the example above, even with a limited number of potential variables, we would still have 162 tests that could be executed.
We might generally assume that if we were to test systems like this adequately, we would need to test them using every possible combination of these parameters' values.
But how do we perform such “adequate”, exhaustive testing in a practical and logical way?
In this article, we'll use the example given above to elaborate on how you can ensure the balance of testing efficiency and quality using Test Case Design techniques, and show you how you can use Xray Enterprise’s Test Case Designer tool to facilitate your process.
Combinatorial Testing considering 2-way (pairwise) and t-way interaction of parameters
Combinatorial testing is an approach that, while keeping the size of the test suite manageable, delivers exceptional outcomes when it comes to fault discovery and defect slippage avoidance. By definition, it is a "black-box test technique in which test cases are designed to exercise specific combinations of values of several parameters" (ISTQB).
How is balancing the test suite size and the defect prevention possible? Based on the practical research, there are (almost) no defects involving all variables in your system at the same time. Multiple studies from the National Institute of Standards and Technology indicate that the vast majority of defects are related to only 1 or 2 inputs.
Why use Combinatorial Test Design?
Here you can check the average results from 10 published pilot projects across a diverse range of software testing projects:
Source: "Combinatorial Testing" IEEE Computer, August 2009. Dr. Rick Kuhn, Dr. Raghu Kacker, Dr. Jeff Lei, Justin Hunter
There are several methods to create t-way (such as pairs and triplets) parameter interactions. Each tool may take a different amount of time and may generate a different number of scenarios to get the same level of interaction coverage.
The following tool features should be considered:
Ability to reduce the number of tests and the time to create them
Combinatorial tool’s primary goal is to locate the smallest mathematically attainable collection of test cases that satisfy the t-way condition as quickly as possible. Using the example from before, a good tool would only generate around 13 test cases using the pairwise technique (which is a significant efficiency increase from 162), and it would only take seconds
Support for flexible coverage goals
In some situations, we might need to test more thoroughly than pairwise (e.g. to guarantee 90%+ defect prevention on a highly critical project). To enable that, a tool should be able to provide a wide selection of coverage goals. For instance, we could cover every combination of three values across all parameters by selecting “3-way interactions”, which would still require just 35 scenarios. Even risk-based use cases may exist where specific parameter combinations are prioritized higher than others.
Furthermore, the generated test suite should be structured so that the incremental interaction coverage is higher with the first tests and lower with the final ones. In this manner, if we stop testing at a given point, we can be sure that we would have executed the greatest number of practical variations.
Ability to combine business insight with statistical techniques
Even if we dramatically reduce the number of test cases using t-way testing, not all of these combinations may make sense for a variety of reasons. The tool's statistical component would not automatically consider subject matter expertise.
For instance, the Departure and Destination values must be different in our flight booking scenario. Additionally, there may be rules that forbid children from traveling First Class.
The tool should therefore be able to handle rules that limit particular interactions in the generated data set.
One of such tools is the Test Case Designer (TCD) inside Xray Enterprise, let’s see why combinatorial testing with it can be beneficial.
Combinatorial testing using Xray's Test Case Designer
Defining the model
First, we need to define the parameters and values for our testing scope, either in the table/plain text format or the mind map one. Test Case Designer also offers support for ranges and equivalence classes.
Applying business rules
Next, we can set up "constraints" in Test Case Designer that define the relationship between two parameters. We can use a variety of restrictions, as demonstrated in the example below: Class=First and Children=[1, more than 1] cannot coexist.
On the other hand, there are occasionally very significant interactions because they indicate commonly used happy paths or particularly significant prior defects. Since they are “unique,” we wouldn’t want to increase the overall coverage goal.
Instead, users are able to alter the priority of specific scenarios and intervene in the algorithm's statistically determined order.
That can be accomplished by using Forced Interactions. In the example below, we consider a hypothetical rule that offers a discount to flights leaving the USA with First Class and more than adult (i.e., 3 specific parameter values participating in 1 condition).
Generating and reviewing test cases
Once Parameters and Rules are in place, we launch the scenario generation (pairwise by default), and we can see that the resulting data set includes our Forced Interaction as the first row and doesn’t include . The creation of this suite only took several seconds, and the coverage goal dropdown provides sufficient flexibility for other thoroughness levels, including risk-based testing.
TCD also lets us analyze the precise coverage of combinations achieved after any number of “executed” tests.
Using t-way approach for scripted testing and exploratory testing
T-way generation only produces the set of parameter values; it doesn't instruct us how to actually perform testing.
Typically, an optimized dataset will be used to data-drive a scripted test case (e.g., a manual test composed of steps/expected results or an automated test script). However, we may also use this technique if we adopt an exploratory testing strategy, such as for certain hardware/software setups.
In the first situation, testers would outline the procedures to take while making references to the associated parameters (instead of hardcoding text). The test is iterated as many times as there are rows in the created dataset.
TCD Scripts provide the ability to generate execution instructions for both manual and automated testing in a data-driven fashion, which increases the flexibility of selecting the execution approach.
Improve your speed and quality with Xray Enterprise
Xray Enterprise’s Test Case Designer is a more advanced method, not a replacement for Xray's built-in capability for parameterized tests and datasets. Both are suitable for a certain project.
With the help of Test Case Designer, you can create a manageable number of test scenarios and ensure that most value combinations are covered early on in the test suite, hence minimizing risk while optimizing your testing. It is possible to:
- define parameters and values;
- enforce system-under-test rules;
- generate optimized datasets using 2-way or t-way settings, up to a certain level, within minutes;
- review achieved coverage with intuitive visualizations;
- and create execution instructions in an efficient, template-based manner.
Get Xray Enterprise and access all the features from Test Case Designer. Start improving your testing efficiency with advanced Test Case Design techniques.
