Test Automation is a necessary precondition for high-quality modern development. Testing and test automation must be integrated into the software delivery processes to give the team the right level of feedback, at the right time, and to ensure high-quality software.

One of the biggest challenges of building a test automation framework that works on a long-term, scalable basis, is that it requires fundamentally different conditions compared to a predominantly manual testing approach.

Switching to an automation-first approach requires multiple shifts in mindset, skill sets, and the tools and processes needed to be successful. The first shift is from a bias towards manual testing to a bias towards automated testing. The second mindset shift is changing from on-demand testing to ‘continuous testing’. The final mindset is to shift left and write the tests before development starts.

In the following article, we guide you through the different mindset shifts, and the skills and typical jobs/responsibilities to succeed in each phase.

 

3 mindset shifts to successful test automation

Creating lasting change requires changes to the tools, the processes, and, most importantly, the mindset of the stakeholders involved. Based on our experience, these 3 mindset changes must be achieved in order to succeed with test automation for the long term.

Let’s take a look at each stage in detail and some of the challenges to overcome. 

 

Starting with manual testing 

In manual testing, the ability to execute manual tests is always held back by the lack of manpower and time. Therefore, teams have to limit the number of manual tests to only the highest risk tests. They test what has the highest risk of failing, not necessarily the highest value to the customer.

Skillset required: Domain knowledge is prioritized over technical expertise.  

Typical focus: Creation, execution, tracking, and monitoring. With manual test execution, the cost of execution is high, it simply takes a time to execute each test one by one.  To avoid unnecessary rework setting up the test conditions, tests are often extended to include multiple and varied types of checks. Also, there is a lot of "on-the-fly" updating of tests and/or test data. This rework is often not tracked and hence the cost of maintaining the tests is under-estimated.

Tools: With a manual testing focus, the primary tool required is a test management tool and maybe one to help with exploratory testing.

Challenge: The challenge of manual testing is to balance validating the new features as they are implemented against the need to have sufficient regression testing of the existing features in order to be confident enough that the version is stable enough to ship/release - hence a risk-based testing approach.

Due to the constant time pressure, instead of checking every case, one case from each cluster of tests is selected, if that test passes then the risk of the cluster failing is ‘low’ and hence the other tests are skipped.

Very typically in an agile context, this scenario looks like an iterative waterfall.

The testing only starts after the development is finished or close to being finished. This also leads to a test silo mentality, and the testers are often outsourced or offshored. Unfortunately, this often results in testing being de-prioritized and/or seen as a low value add. Not exactly the result that we want.

Let's see how we can go from manual testing to automated testing. 

 

Mindset shift #1: Qualitative over quantitative

If you can automate away the many time-consuming tests that ‘need’ to be executed, then instead of concentrating on getting through the highest volume of tests, (quantitative), you can instead focus on higher-value tests (qualitative). This way you can take advantage of domain knowledge and product know-how to build a better product.

Skillset required: Domain knowledge is still necessary, but also a technical understanding of the system under test. Additionally, test automation and/or development skills are also needed.  

Typical focus: Creation and maintenance of the automated tests. Manual test cases are translated into automated tests. Automated tests also have a higher dependency on stable data, therefore a deeper understanding of the target system at a technical level is needed.  Failing tests need to be investigated in order to identify if there is a problem with the test, the data, the environment, or the application.

Improvements: The biggest advantage of automated testing is that execution is cheap.  This enables the bulk of the tests to execute automatically, enabling testers to have a much higher qualitative focus. This can also mean more time for testing in an explorative way to better understand and validate the system under test.  More time spent in the qualitative analysis means more advanced test designs, leading to a better test solution and therefore higher quality levels.

Tools required: Test Management, Exploratory Testing Tool, Test Automation tool 

Challenges: The complexity of test automation is often under-estimated and requires a much deeper technical understanding than manual testing. This means we need people with test automation skills in the team, as well as technical resources to help build up the test framework and foundations.

Having the tests is one thing, using them is another. One of the classic problems is that the ability to execute the tests ‘frequently’ is not leveraged. The automated tests become an extension of the manual test suite, a check-box, rather than something that can be used to enhance the quality of the system.

Another problem that arises is the dreaded automation test rot. This is where tests that were once successfully running start to fail. The percentages of failing tests increase and before you know it, the time spent maintaining the test automation exceeds the time spent creating new ones. This problem is exacerbated if the tests are decoupled from the build and deploy pipelines.  

This is often the point of time where test automation exercises fail, the effort in maintaining seems to outweigh the benefit that the framework brings.

So, how do we go from automated testing to continuous testing? 

 

Mindset shift #2: Test everything, all the time

Skillset required: In addition to the skillsets needed for automated testing, we also need a good understanding of DevOps tools and approaches. This includes iterative and continuous improvement based on the constant and quick feedback from the tests.

Typical focus: Tests need to be integrated into pipelines and we need to relentlessly follow up on failing tests. A zero tolerance for failing tests is the only way to stay on top.  Tests need to be redesigned to be more resilient and the dialogue with the developers ‘intensifies’.

Improvements: The tests become an active monitoring tool. They now tell us much more than if the application is functionally working. Now we can see how stable the application or application landscape is, we validate our data, we can see how the system behaves at any time. Videos, screenshots, and log files are collated on each execution and saved in our test management tool. We can create dashboards that we can offer to any level in the project management in order to see exactly how stable the system really is.

Tools required: Now the technical landscape starts to look a lot more complex:

1. Continuous Integration Tool - With a continuous testing framework, the trigger is typically the continuous integration tool, i.e. Jenkins;
2. Code Repository - The test code is stored in the code repository, it should be treated with the same respect as product code, under version control;
3. Test Automation - The tests then begin (you can use CAST, a systematic tool for test automation at scale, or ForeCAST an AI-based autonomous test generation tool) 
4. Test Management - All tests and results should be pushed to the Test Management tool (this goes without saying that Xray is the right tool here);
5. Task Management - Of course, the beauty of Xray is that it natively integrates with Jira for our task and project management;
6. Test Environment - The tests execute in a test environment;
7. Target System - The test system is our proxy to the actual system under test;
8. Log monitoring tool - For example, ElasticStack where the results are pushed to a central dashboard, where we also collect all the application logs and drill down into each failing test, see failure patterns and have one objective view on the health of our application;
9. Wiki - Confluence (if we need it).
   
   

Challenges: A typical anti-pattern that arises is that the same iterative waterfall pattern continues. The testing still begins after the development begins and only when the feature ‘stabilizes’ does the test automation begin.

As illustrated in the diagram, test automation at the end of the development process makes test automation expensive, fragile, and provides the least value to the team. 

So we have one more step to make, to go from continuous testing to test first. 

Mindset shift #3: Shift left to automate, integrate, and collaborate

Skillset required: Every member of the team is a stakeholder in quality. We need a mix of technical and business skills, domain knowledge, and of course DevOps.

Typical focus: Testing the requirements and not the code. This means creating tests upfront and making this the standard way of doing things. The ‘plumbing’ of the continuous testing framework needs to be maintained at all times. Tests can run red until they run green, then they always run green. 

Improvements: Now we have really turned the lights on. The tests are now living documentation, they extend the requirement docs to be a complete specification. The living documents are validated every time the tests execute. Now we get fast feedback on each cycle.  

This feedback loop helps us know two things:

1. For a new feature, we find out immediately if the feature is ‘done’
2. If we have broken something because each living document is automatically converted into our regression suite.

Tools required: We already have everything that we need from a tools perspective, but we need to be able to test and define all layers of the application, including:

• Front End
• Back End
• Everything in-between (msg queues, database, log files)
  
  

The tool we are going to use to automate our tests needs to be:

• Easy to Use
• We have to enable non-technical and technical members to be able to contribute fully
• Technically complete
• We have to support full-stack development
• If you have full-stack development then you need to support full-stack testing
• Fully integrated into the existing DevOps tools
• We want to leverage the existing DevOps tools, not compete with them.
• We should be leveraging the relationship between the tests and the requirements to derive full coverage

Finally, we have to build up the surrounding ecosystem to make the tests effective.

• Data
• Our approach to managing test data needs to be resilient and flexible
• Environments
• We need to have full control of the test environments
• Visible
• We need to make sure that our efforts are understood and can be viewed by all project stakeholders.

This is hard work to maintain and it needs a disciplined approach to keep it up to date and working properly. The benefits are huge, with massive improvements in speed and quality.

Challenges: As the number of tests increases, it's no longer possible to run all the tests, all the time. Therefore we also need to ‘Shift Right’. By implementing staged quality gateways, the first sets have a bias towards speed and minimal but sufficient info, then we can test the functionality in an increasingly exhaustive manner, then move on to the non-functional tests; performance, load, disaster recovery, security, etc.   

It's hard work, but if we do this right then any code that comes out the other side of our pipelines is a shippable, fully validated piece of code that is ready to release.

To prioritize test automation, we must change our mindset

With any kind of change, its success is determined by tools, processes, and mindset. The mindsets we described are not absolute and they can coexist in a project. If you try to achieve the more advanced mindsets then the benefits are very tangible, meaning quicker, cheaper, and better quality.

Less tangible, but no less important, is the improvement in the flow in the software development team. This means less friction in the processes which results in less frustration and more fun at work. 

Maybe you don’t need to be testing everything, all the time, but with a test-first mentality supported by continuous testing then the quality and speed of your software will improve substantially.