Didn’t you have the chance to listen to our podcast? Don’t worry. With this article series, you can read the highlights of each episode.
In this episode, Maaret Pyhäjärvi is our QA therapist and helps us diagnose and prescribe the symptoms of exploratory testing. Maaret has several significant awards in the testing space. She has delivered hundreds of talks and is a frequent keynote speaker. She lives testing from the inside out and has contributed to the industry for years.
Do you know what contemporary exploratory testing is? Some of us may have heard the buzz around exploratory testing, while others may have missed it. What does exploratory testing have to bring to the table in a world with a legacy around scripted test cases? To find out, keep on reading or listening as our special guest, Maaret Pyhäjärvi shares her valuable insights with us.
Listen to the episode:
Episode highlights
What is contemporary exploratory testing?
Our guest reminded us of the 80s when Cem Kaner mentioned that some companies in Silicon Valley did exploratory testing and described that as an independent, results-oriented, more professional way of doing testing. This was the best option available back then.
Looking at the testing today, Maaret realized that contemporary exploratory testing still involves the same principles Cem Kaner described before, but with the addition of everything new in the world (test automation, CI/CD pipelines, unit testing, microservices architectures, and all the ways we isolate features of quality that need testing).
She considers contemporary exploratory testing frames test automation and the so-called exploratory manual side of it as the same thing. They go hand in hand and are essential for smart and intelligent testing.
Maaret believes that a team's automation responsibility is reflected in the output of exploratory testing, which forms the foundation for future testing. She gives us an example of a tiny team that may have 1250 test cases in automation that run 20-80 times a day. However, this only accounts for about 10% of their testing, as the rest is exploratory testing that supports future exploration rounds. With the changes in the world and the products, we will have to explore not just the old functionalities but also new ones. All of that is a great foundation for doing what Maaret calls “contemporary exploratory testing”.
Therefore, contemporary exploratory testing is crucial in her perspective, as it combines new tools with traditional principles to achieve the best possible testing results.
What are the outputs of exploratory testing?
- Communicating bugs with fixes
Maaret believes in finding ways to communicate about issues beyond writing bug reports. Pairing with a developer to fix the bug together or fixing it herself is often more effective. While it's not always possible to report a bug with a fix, working in a pair or ensemble format can always help.
- Better skilled testers
In her opinion, this is an essential part of what exploratory testing produces. No matter what our role is, when we explore, we will grow in those skills. We are becoming better testers and better people the next day. That is the core of exploratory testing.
- Thoughtful artifact creation
Creating artifacts that document what we've learned during testing is essential. Maaret considers that test cases should be viewed as an output of testing rather than an input. She advises us to take a moment to reflect on what we've learned, review our artifacts, and ensure that they document our best work of that day. Those artifacts will serve our future selves.
What is serendipity in exploratory testing?
Maaret defines serendipity as a lucky accident.
She explained that in exploratory testing, we hope that the features that used to work still work and that the ones we made work (but never tested again) will keep working. We are always surprised when there is a problem.
Sometimes the issues are easily noticeable, she adds. These problems are like pizza boxes in the middle of a living room floor. We can point them out and say that they don't belong there.
But a lot more often, there are problems no human could think of how to put together to create that kind of problem. Those problems couldn’t be predicted, are more complex and require some luck to uncover. That is when serendipity hits. By exploring the application and testing different scenarios, we increase our chances of encountering unforeseen problems.
How does automation connect with exploratory testing?
Maaret gave us a couple of examples of how she frames it:
- Adding unattended testing (automation) to maximize test results
Once, she needed to test a new system that generated an hourly report for airports. However, waiting an hour for each report was too long, so she increased the frequency to every five minutes. Maaret then realized that manually keeping track of changes was difficult and required automation to ensure she got all the important details.
To test the feature, our guest created an automation script that checked the report at four, five, and six-minute intervals to ensure it reflected the expected changes. She then reviewed the logs at the end of each day to identify any issues and improve the script for the next day's testing.
For Maaret, this is the essence of exploratory testing: discovering new information. Using automation for unattended testing and reviewing logs for attended testing, she could identify areas where the system didn't work as expected. Then she adjusted her automation script to collect the right information in these areas. This helped to improve the accuracy of her testing and ensure that the system functioned as intended.
- Optimizing unit tests: using parameterization for exploration
Maaret often receives a pull request with unit tests that the developers want to keep. Her next step is to parametrize them, which allows her to test the unit with different values. This starting point of small-scale exploration is valuable as it can uncover potential issues that may not have been apparent with just one example.
She questioned whether the examples provided were enough or if she should consider testing with different data combinations to ensure the code's reliability. Although developers usually find the best example to test with, it may not be sufficient. She believes there is room to do more, as research suggests that up to 80% of issues that customers experience in production can be found during unit testing if we explore different testing scenarios.
After testing with varying data, Maaret decides if there are any tests worth keeping, in addition to the ones provided by the developers. If there are, she would add them to her testing suite. If not, she discards them.
- Introducing automation gambit to new testers
In this example, she uses complete newbies who have never done testing professionally before and introduces them to the automation gambit. In the automation gambit, the newbies receive a constraint: they have to write documentation that is also executable. This method works incredibly well for simple web applications as a teaching tool. With the automation gambit, they create test automation that finds the most bugs, which can be used later in CI pipelines. Maaret emphasizes that this approach allows them to learn how to test, be resourceful, and document all at once with automation.
Did we make you curious?
A discussion with Maaret is always rich in insights. Listen to the full episode to discover why exploratory testing is important, how you can cover all your risks with an exploratory approach, and how exploratory testing fits into her day-to-day testing process.
Available on:
