With the rise of DevOps movement, there is an increased focus on delivering automated, continuous improvement.
Continuous Integration (CI) and Continuous Delivery (CD) are core software development disciplines that are part of an Agile software development cycle. The idea is to release fully-tested and stable builds early and as often as possible. Automation, specifically automated testing, is a crucial component of this process. In short, Continuous Testing is the core enabler for CI/CD.
Introducing testing early on the “left-side” of the development process, starting with unit tests and complemented with service/functional and integration tests, is fundamental for making CI/CD happen. But that may not be enough. While Shift-Left Testing happens during the development, Shift-Right Testing happens in post-development scenarios (i.e., in or based on production data).
Let’s examine the differences between the two.
Shift-Left Testing
To facilitate CI/CD, testing needs to be prominent from the start, specifically at the requirements specification/story definition phase. Shift-Left Testing is crucial in Agile teams as it allows artifacts to be properly validated through clearer and better communication. The idea is to find bugs and fix them as soon as possible. The sooner, the better.
Activities:
- Discussion of requirements (either the typical requirements or the Agile's user stories), bugs and any other artifact that implements, changes or fixes behavior of the product
- Writing of unit tests at the same time the artifacts (i.e., issues) are being implemented by developers
- Writing service/functional/integration tests (by the developers or the members having the "tester" role); this can be done right after the issue (e.g., story) is clarified and included in the Sprint backlog
- Test execution, which happens as soon as possible (during the Sprint in case of Scrum); this execution is hopefully automated and unit tests run often, upon code commits or merges. Other types of tests (non-UI or end-to-end) run also as soon as they're implemented.
| Pros | Cons |
| Better and clearer requirements/stories | Lack of UI/end-to-end validation |
| Leads to the development of testable features => increase Testability | Misses "real feedback" (i.e., coming from real users on real usage scenarios) |
| Bugs detected sooner | |
| Reduce fix costs (saves time and money) | |
| Prevent bugs at the end of development cycle/release moment | |
| Introduces and embeds quality from the start | |
| Reduces gaps between developing and testing | |
| Early code review of automated tests, letting testers review assertions and usage scenarios |
Shift-Right Testing
The focus of Shift-Right Testing is to get feedback from real users after development is done. It’s like extending and continue testing after the release has occurred.
To achieve this, techniques such as A/B testing, Canary releases or Blue/Green deployments are used to obtain feedback from end-users about the delivered features in a controlled manner.
Active monitoring tools are used in order to detect errors as soon as they appear and also to gather valuable performance metrics.
Because development is done, and all product features have been integrated, end-to-end testing, including UI testing, can be done properly.
| Pros | Cons |
| Use direct feedback from users | More costly bug fixes |
| Allows automated testing at the UI level with end-to-end workflow testing | |
| Features are complete, more integrated and more stable | |
| Allows real performance, stability and security testing | |
| Discover obscure usage scenarios | |
| Test ideias and UI changes with A/B testing | |
| Detect bugs in controlled scenarios | |
| Enables Blue/Green deployments and release rehearsals |
Which testing strategy should you follow?
Agile teams would do best with Shift-Left Testing for clearer specifications, smoother implementations, earlier testing, and reduced costs. However, Shift-Right Testing adds other benefits, such as the ability to incorporate user and system feedback based on real usage scenarios.
Therefore, we see these approaches not as exclusive or contradictory, but complementary.
As depicted in the Test Pyramid, you can focus your effort more at the "ground level" with unit testing, but that does not imply that you shouldn't do other kinds of tests at different levels. The same is true for Shift-Left/Right Testing: you may invest more in requirement reviews and unit tests, but that does not imply or exempt you from doing integrated tests afterwards, especially if they involve the actual end-users in actual usage scenarios.
