According to ISTQB, test design is “the process of transforming general test objectives into tangible test conditions and test cases.”
Effective test design is a crucial part of SDLC as it helps quickly identify defects in product implementation or gaps in requirements. Achieving those benefits, however, is no easy task, as you can see from the graph below:
As business and development strategies continue to evolve, it is important to understand the implications for the QA practices and how you can best stay up to date. In this article, we will first look at broader trends, then discuss how they affect test design specifically and what role technology plays.
Major relevant trends in test design and software development
The software development landscape is undergoing a significant transformation. There is a lot more demand for faster product life cycles, improved customer experience, better security - all at lower cost. Those goals cannot be achieved without a strong focus on quality.
Overall, the trends below seem to have the most widespread impact:
Agile/DevOps/QAOps
DevOps adoption is growing but nearly 80% of companies remain in the middle of their journey, experiencing varying degrees of success at the team level but rarely across the entire enterprise. Larger organizations tend to be more proactive with adoption – 90% of them reported Agile, 40% - DevOps.
Automation (yes, still)
Even though test automation is not new, the adoption is hard and we continue to see it as a top priority for teams for the next 6 to 12 months followed by the need to increase automation coverage and reduce time of regression testing.
When asked about the worthy investments, companies are pointing to codeless frameworks (43%), tools that can be used by their entire team regardless of coding skill.
Quantifying results
Stakeholders want more precision in tracking the status and ROI of their projects. That requires establishing and optimizing KPIs at each stage of the process, including test design. One metric in particular, “coverage”, has reappeared as a growing software testing priority for the teams.
AI effect
Artificial Intelligence will be a hot topic in the testing industry for the foreseeable future. With the evolution of AI tools like ChatGPT and existing tools like Applitools & Reportal.io, testers aim to make their tasks easier, perform them faster, and avoid manual errors as much as possible.
Read more about the software testing trends for 2023.
Impact on Test Design
Based on these broader trends, the important considerations (in order of priority) for the test design strategy are:
Quality
With more companies hitting the diminishing returns from the initial speed optimization of Agile, quality has started to receive more attention. There are several aspects to evaluate:
- The coverage of each test case - making sure each scenario maximizes incremental thoroughness but also validates relevant functionality as efficiently as possible.
- The coverage of the test suite - being able to easily establish traceability between test scenarios and requirement groups at the project level to analyze risk areas.
Speed
Of course, quality-vs-speed is still a tradeoff that should be properly balanced, and there are areas of improvement in the second metric as well. There are several aspects to evaluate:
- The speed of creation - agile-based approaches imply development in short, iterative cycles, and test design often needs to be done several times within each iteration. There is no room for redundant activities and deliverables.
There are two components in achieving this:
Faster design of scripted tests
Manually writing tests (one at a time, for each isolated acceptance criteria) does no longer work. Careful planning at the feature level and smart scripting tools have become a must.
Growth of exploratory testing
It continues to be an approach that more and more organizations are incorporating into their strategies. Not only does it help uncover unknown risks, but, with the right tools, also allows to save time when formalized tests are not as necessary.
- The speed of execution - in chasing the thoroughness, the answer cannot be “excessive size of each test suite”, both for new and regression activities.
- The speed of maintenance - to maximize this across iterations, we need to quickly perform requirements impact analysis and intelligently reuse as many assets as possible.
To facilitate all 3 aspects, automating QA processes and integrating them into the software development lifecycle are becoming the norm. That creates a long-term, sustainable approach to quality assurance.
Collaboration
More specifically, collaborative design with automation in mind. With the move towards DevOps, quality becomes the embedded function and the responsibility of all teams, which means everyone does test design and effective coordination is crucial.
First, all stakeholders need to get on the same page about the testing scope with the help of artifacts that do not require any special skills (e.g. mind maps, decision tables).
Second, tests need to be in a consistent and concise format that can be both easily automated and quickly reviewed by any non-technical person. Which is one of the reasons why tools that support behavior-driven development (BDD) are rising in popularity. Companies that practice BDD place a higher value on testing (74%, vs 69% overall).
Lastly, the tools involved in management and execution need to have features that enable quick and accurate feedback loops back to the design process. And each tool needs to provide a variety of metrics that support the status and ROI evaluation.
Versatility
To truly unlock reusability and knowledge sharing and ensure KPI consistency, we need test design approaches that work for a variety of systems, testing types and easily integrate with other toolchain elements.
In addition to the diversity of artifact formats, we should consider:
- The mix of design techniques across, e.g.,
- Boundary value analysis
- Equivalence partitioning
- Decision tables
- State transition
- Error guessing
- The scenario generation approaches, e.g. combinatorial, orthogonal array, linear expansion, random, etc.
- The range of scripting capabilities (e.g. traditional step/expected result, BDD, recording, code-based) and the ease of conversion between them.
How technology evolution plays into the test design choices
We mentioned “tools” a few times already, so just to reiterate - achieving 4 goals above manually is not possible. Different kinds of technologies emerge to facilitate (but not completely replace) human effort.
AI and machine learning have arguably been the biggest advancements in the space. By leveraging those algorithms, companies can significantly increase the speed, reduce repetitive tasks, and simplify the software development life cycle.
QA members can use AI tools to:
- Draft a rough test script based on best practices and user-defined criteria.
- Identify various possible combinations to test a product.
- Apply predictive test selection.
- Generate a code template for automation.
The output from such tools, though not perfect, can be a good starting point. The applications are, of course, not limited to test design - they include bug reports and “virtual teammates”, among other areas.
As artificial intelligence and machine learning improve and become more mainstream, companies are likely to accelerate the adoption of scripting standards that are both human- and machine-readable (e.g. BDD).
However, with the current shortcomings of AI/ML tools, we can leverage other algorithmic approaches to achieve similar results. For example, combinatorial design engines do a good job of identifying optimal test sets that achieve high coverage. That, combined with the customizable reporting features, also helps with the test selection challenges. Conversion tools, like “Automated Test Framework” inside Xray Test Case Designer, can accelerate the creation of code templates based on the existing scripts in Gherkin language.
Deliver high quality products with Xray Enterprise’s Test Case Designer
Test design is an integral part of product delivery, so you might as well invest in performing it properly.
It is hard to imagine a slowdown or a fundamental shift in the industry transformation we mentioned in the opener. That means the role of the 4 criteria above in the project success will be pretty consistent.
If you can create an effective test design strategy based on them, you will generate value for the organization, ensuring you can deliver top-quality products as fast as the market demands them.
Get onboard sooner rather than later.
- For model-based, combinatorial test design that supports equivalence classes, boundary conditions, decision tables, traditional and BDD scripting, try Xray Test Case Designer.
- For robust exploratory testing, with easy feedback notes and screenshots, try Xray Exploratory App.
