We see a lot of teams working with automated testing, and many feel the need to take the next step in order to optimize and excel in their product release cycle. Prior to implementing DevOps, companies can experience a stall in releases and only make new ones every 6 months. This means that development efforts are not bringing value to users quickly enough.
For companies that work in regulated industries, it’s also critical that they set up reporting and traceability into their DevOps pipeline.
In this blog post, we’ll explain how a powerful DevOps setup that includes test automation and reporting can speed up the release cycle.
Integrating automated tests into a DevOps pipeline
For many companies, one of their goals is to reduce the release cycle so that they can release every two weeks, or even more often. Automating the bulk of their tests, helps in two ways:
- Identifies regressions close to the event. As soon as a commit causes a failed test, prompting the developer to fix the problem.
- Shortens the pre-release test cycle. Ultimately shrinking the critical path from the last code commit to a production release.
The automated tests will be one of the final steps in their new DevOps pipeline. The idea is that each software commit would trigger a build and a test cycle. If the test cycle is green, an automatic deployment onto their staging environment will follow.
Out of all the investments in DevOps, the automated tests are the biggest ticket item. Investing so much in automated tests is only justified in light of the business objectives.
Providing reporting and traceability
A company that works in the medical industry and manages an app, for example, can’t ignore their regulatory obligations. To release a new version they also need to provide extensive test and traceability reports. These are the steps to generate these reports using Jira, Xray and Jira Snapshots for Confluence:
- Last commit of software changes triggers a build of a release-candidate
- The pipeline runs the automated tests. If it’s green then the pipeline proceeds to import the test results to Jira
- A manual test cycle takes place. With the huge investment in test automation, manual test cycles take only one day.
- Product Manager generates the final test report and traceability reports. Each takes a single click (using Jira Snapshots for Confluence). They route the reports for review and electronic signature in Confluence.
- And finally the release can happen in less than two days since the last software commit.
Let's take a closer look at how it works. Giving you all the tips and tricks to make this happen for your team too.
Importing the results of automated tests into Jira/Xray
One of the reasons a company facing these issues would choose Xray is that its main benefit is native support for automated tests.
Xray has native support for many testing frameworks. Some companies use a mix of technology stacks, with each generating tests reports in another native format.
Xray offers an integration with JSON format that allows testers to import into Jira through the API. So, for each framework the test team can implement a small converter from the native report to that standard format. The effort involved in implementing this extra step is small and worthwhile.
This allows all tests to generate standardized reports that Xray supports. Importing these reports to Jira is the responsibility of the DevOps team. From their side, they end up having to support the import of one single format. So this is a perfect decoupling of responsibilities (between testers and DevOps engineers).
To streamline the tester's work, not only the test results but also the test cases themselves, need to be created in Jira by the pipeline. When a new test is first run and appears in the test result report, the import to Xray/Jira takes care of creating the corresponding test issue in Jira.
Managing a mix of manual and automated test cases and test executions inside Jira/Xray
Inside Jira, Xray manages manual and automated tests with the same issue types. Both are test issues. The difference is indicated by the Test type: Manual vs. Generic. When the DevOps pipeline creates new Tests in Jira, it sets their Test type to be Generic.
A core topic for companies in the medical industry is their Test Coverage. From a regulatory perspective, they need to show that all requirements are covered by tests.
In Jira, we express coverage by tests as a Jira link from the test issue to a requirement specification issue. This link is manually created in Jira- both for the manual tests and the automated tests. It's a low maintenance high leverage activity. Once added, a link facilitates traceability reports for years to come.
The execution of a manual test in Jira is captured as a Test Run. Each time the pipeline imports a result of an automated test to Jira, this is captured as a Testrun:
Generating tests and traceability reports
Once the last cycle of tests is complete, the next step is to generate the test and traceability reports.
Xray comes out of the box with strong reporting capabilities. But some teams manage all their documentation in Confluence, and want to have a Confluence-based reporting flow. To achieve that they can install Jira Snapshots for Confluence, along with Snapshots traceability extensions for Confluence, which provides the integration with Xray.
A plus is that Jira Snapshots has a “no code” approach to customizing reports. The Product Managers can now generate reports exactly how they need them.
With this setup, Product Managers can prepare the report pages long before the release is ready. It’s then a matter of plugging into them the latest information from Jira. Once the manual test cycle is complete, the tester notifies the Product Owner. With one click, they refresh the snapshot on each page with the latest data, and the report is ready.
Companies can have all their documents approval process set up in Confluence. So when these two last reports are ready, they are immediately routed for review and approval.
This way, it takes a couple of hours from the last test run, to having signed reports and the Software approved for release.
A new product release reality
After integrating with Xray, companies can end up with a optimized DevOps pipeline where:
- Automated tests provide the majority of the test coverage. A limited collection of high-value manual test scripts completes it to provide full coverage.
- All test results, automatic and manual, converge in Jira and Xray.
- One click is all it takes to generate submission-grade test and traceability reports. Jira Snapshot (with Snapshots traceability extensions) provides this capability in Confluence.
With this pipeline in place, the time from code commit to releasing a new version is shorter and sweat-free.
After a while of successfully achieving an optimal release cadence with Xray, companies will see the ripple effects of this change everywhere. Not only are their users raving about the quick progress, also the team shifts into another state. Each team member feels that they have a much more immediate impact on the product.
