Generative AI is bringing a new era of “convenience” with ChatGPT, from OpenAI, taking center stage in our daily lives. From aiding in solving complex problems to generating content, this large language model has become a helpful companion for various testing-related tasks. As generative AI is becoming increasingly present in our daily lives, we should understand how to use it and account for its limitations.
In this blog, we will analyze how to use ChatGPT in the context of exploratory testing and how to improve its outputs.
First, what's the essence of exploratory testing?
The essence of exploratory testing lies in its dynamic and adaptive approach to uncovering the true state of a software product through experimentation. Like a skilled explorer diving into uncharted territory, an exploratory tester navigates the software landscape, seeking clues from various sources such as UI layout, wording, logs, and infrastructure.
While traditional testing may rely on predefined test scripts and expectations set during development, exploratory testing emphasizes the importance of understanding the context in which the product operates. It's about assessing relevant quality information for the right people at the right time, recognizing that quality is multi-dimensional and subject to individual interpretation.
Knowing the limitations of generative AI in exploratory testing
Now, when it comes to using AI with exploratory testing, we need to be careful. ChatGPT, while possessing impressive capabilities, has difficulties in grasping the entire context to the same extent as us (human testers). It lacks a deep understanding of the product's purpose, relevance, and its target audience, making the interaction time-consuming (contrary to its intended ease). However, this doesn't negate its potential to assist in various testing-related tasks. That is why it is important to be aware of its limitations while using it.
Generative AI (ChatGPT) relies on an extensive training dataset as its primary source and is governed by many parameters, which serve as the repository for its accumulated "knowledge." As a result, it excels at tasks like text classification, question answering, document summarization, and text generation. However, the AI and ML community has raised valid concerns regarding several limitations:
-
Hallucination and fabrication of facts and references;
-
Reliance on non-verifiable sources with limited accessibility;
-
Bias in generated content;
-
Limited context window (restriction in the number of words imputed leading to content that lacks context).
How to reliably use generative AI for exploratory testing
It's important to clarify that we cannot rely solely on generative AI to perform exploratory testing. Exploratory testing is a process that requires a human tester's expertise, evaluation, learning, and continuous iteration to extract valuable insights.
However, using generative AI in conjunction with us (human testers) can prove beneficial for testing-related tasks and act as a source of ideas and inspiration.
How can generative AI actually be helpful in exploratory testing?
1. Generate exploratory test ideas
Exploratory testing thrives on creativity, and generative AI can be a valuable tool to fuel this creativity. Due to its pre-training on large amounts of data, tools like ChatGPT are a great way to brainstorm fresh test ideas. Here are some prompts to get your creative juices flowing:
Prompt example
Given the following user story, can you enumerate some exploratory test ideas?
As a user, I can import attachments in a CSV file with Xray Test Case Importer. It should include Test and Test Step attachments.
2. Create exploratory test charters for a specific quality attribute
Test charters are like roadmaps for your exploratory testing sessions, and AI can help you brainstorm ideas to create them. As mentioned above, ChatGPT is highly dependent on your input, so experiment with different prompts until you get the answer you need.
Prompt example
Given the following user story, can you enumerate some ideas for test charters related to security?
As a user, I can import attachments in a CSV file with Xray Test Case Importer. It should include Test and Test Step attachments.
3. Summarize an exploratory testing session
After a productive exploratory testing session, summarizing your findings is essential. AI can help you do this by automatically extracting key observations, identified defects, and areas needing further exploration. Let’s see an example:
Prompt example
Given the following testing notes, can you make a brief sentence with the overall result of the testing session and the overall confidence about the quality aspect?
Session Objective: Explore the banking app to identify usability issues, functionality glitches, security concerns, and performance bottlenecks.
-
- Successfully logged in using valid credentials;
- UI elements responsive and well-organized;
- Logout option visible and functional;
- Account balances displayed accurately;
- Transaction history loads promptly;
- Icons and buttons are intuitive and self-explanatory;
- Transferred funds between accounts successfully;
- Validation messages clear and informative;
- Transaction confirmation prompt present;
- Paid utility bills without errors;
- Bill payment history updated correctly;
- App navigation smooth and intuitive;
- Back button functionality consistent across screens;
- App content readable and accessible for users with disabilities;
- Screen readers compatible with app elements;
- Color schemes and fonts consistent with the bank’s branding;
- Verified secure HTTPS connection;
- Sensitive data encrypted during transmission;
- Multi-factor authentication functioning as expected;
- Account lockout after multiple failed login attempts;
- Session timeout implemented for inactivity;
- Users logged out after prolonged idle time;
- Screens and data load within acceptable time frames. No noticeable delays in accessing account information.
4. Enumerate risks related to a story or requirement
Identifying, addressing, and prioritizing risks is key to quality success. You can use generative AI to assist you in brainstorming and analyzing risks. Remember that your input is critical to generating quality answers (garbage in, garbage out).
How you can use ChatGPT to brainstorm risks related to a requirement:
Prompt example
Given the following user story, can you enumerate some risks related to it?
As a user, I can import attachments in a CSV file with Xray Test Case Importer. It should include Test and Test Step attachments.
5. Explore scenario variation
Another way to use generative AI for your exploratory testing is by asking for variations of test scenarios to explore different paths and behaviors in the application. ChatGPT can act as a helpful assistant to ignite your creativity and give you a good starting point for your exploratory sessions. Remember, the answers will vary and may be incomplete. Take everything with a grain of salt.
Prompt example
Create performance testing scenarios for a mobile banking app to assess its responsiveness under heavy user load.
6. Generate sample data
Generating sample data is often a time-consuming task. Why not use generative AI to help populate user profiles with realistic data or create datasets? Let’s see a prompt example:
Prompt example
Generate sample financial transactions for a banking application, including deposit, withdrawal, and transfer transactions.
Tips to improve generative AI answers
As the output of generative AI can vary significantly, our goal is to improve the quality of its answers. To get the best possible answers, we should be mindful of our intent. When aiming for an answer, we want it to:
- Be highly relevant to the question asked;
- Provide accurate and factual information;
- Match the level of detail we desire, whether it's a summary, explanation, or key points;
- Align with the desired format suitable for our specific context, including layout, word size, tone style, and language.
To improve the quality of ChatGPT answers, we can take the following steps:
- Include the context
Tailoring prompts to guide ChatGPT toward our intended goal. This involves crafting proper prompts that include relevant context and intent details. - Provide examples
Offering specific examples of the desired answers (few-shot prompting) to guide ChatGPT in generating more appropriate responses. - Always review
Regularly review the answers to identify areas of improvement and ensure they meet your quality standards.
Use generative AI to assist with brainstorming, not to perform tasks
AI can undoubtedly assist in some aspects of testing, but relying solely on it for exploratory testing might leave important stones unturned. It's like having a high-tech metal detector, but you need a seasoned explorer to decide where to start scanning.
So, the key is finding the right balance between AI's capabilities and human expertise to ensure context-aware exploratory testing. When used with careful consideration, ChatGPT can help you brainstorm ideas and kickstart your sessions. But always remember to be mindful of your inputs and to consider the answers attentively.
