6 Steps to Scaling your Software Testing in Jira

Keep your Jira running smoothly whether you have 1,000 test cases or 1,000,000.

At Xray, we have 4.5 million active users across 5,000 customers that trust us to manage over 100 million test cases every month. Ensuring your success by providing you with the ability to work in the most high-demanding scenarios, is in our DNA. 

Currently, we’re aware of some particularly large deployments running Xray successfully with:

• more than 29 million Test Runs in a single Jira instance
• and more than 5K projects or 35K users with over 1 million test cases.

As our customers have grown in terms of users and usage, we have improved Xray along the way. In fact, during 2018 and the first quarter of 2019 there was a huge investment in performance and scalability. Just in the past year, more than half of the enhancements and improvements we delivered relate to performance.

Although performance is influenced by many variables, it is highly dependent on how you configure Jira, the deployment type you choose, and the way you use it. When you ensure that the previous 3 are done well, you guarantee maximum performance at scale.

Here are 6 ways to make sure Xray scales with your development and testing needs:

1. Fine-Tune your Jira Instance

For every tool, there are a set of best practices that ensure a consistent and lightweight process for scaling. If you’re already following best practices for scaling within Jira, then by nature, your instance should be optimized to use with Xray because the tool is built on top of Jira and is mostly dependant on the architecture and technologies followed by Jira itself. Atlassian provides some performance scaling tips, and these also apply to Xray.

1. Limit Custom Fields and configure them properly
2. Moderate workflow usage, in terms of their complexity 
3. Evaluate the plugins/apps you need as some may impact performance.

In addition, if you’re a big organization, or you have large amounts of data and/or many users, you can consider Jira Data Center. To evaluate if Jira Data Center with Xray is right for you, take into account the following benchmarks.

Does your organization have?

•  Geo-distributed teams
• Many integrations with API calls
• Periods of very heavy user load
• Data sovereignty requirements
• 5000+ users

When you're certain that you’re using the right deployment type and have configured your Jira, you can configure Xray and its related entities.

2. Define Your Process

Xray is a flexible tool that you can adapt to your needs. In general, if you have teams working in completely different ways, this complicates communication and leads to improper and unoptimized use of tools.  We recommend that you define a process that you use organization-wide to ensure you sustain optimal performance across all tools including Jira and Xray.

If your team is adopting Agile for managing software development, you can take advantage of Xray for all of your testing-related activities; this is because Xray amplifies Jira and embraces Agile values and principles to enable Agility in teams. Read more about how to use Xray in an agile context here.

Before getting started, familiarize yourself with Xray’s internal terms and concepts and how they relate to common testing language. You can also take a look at this video explaining Xray's features in more detail.

Then, consider the following points to define your process:

1. To create a "Unified" development process, consider what process can be used by all teams in a way to manage the SDLC. Having the Test Process in mind will also help you implement even more quality assurance in your projects. 
2. Are you adopting Agile and Scrum? Check out using Xray in Agile for tips on how you can take advantage of Xray in such scenarios; the Agile software development page provides a high-level overview of Agile and Agile Testing. This page provides some useful tips so your team can be more Agile and avoid doing things that are unnecessary.
3. Each Xray element, or entity, has a purpose/fit that you should try to take advantage of. We’ve built these elements based on best practices in SDLC and specifically test management. You're not obliged to use all of them, but it’s always better to leverage the work we’ve done to build a tool that supports good SDLC.

See this page with further details to define your testing process.

3. Optimize your CI

Any software, regardless of underlying constraints, will tip over with the abuse of the API - for Jira and Xray this is no exception. 

To avoid unnecessary overload, here are 2 things to keep in mind when you’re handling your integrations.

1. Upload only relevant test results (e.g. don’t upload unit test results); choose properly what testing results you want to track within Xray.
2. Constrain upload frequency
   • Don’t create noise by running integrations more often than you need.

Limit the REST API

1. Review the "Max results per request" setting in the Miscellaneous administration settings as it controls the pagination on the REST API calls. 
2. Limit API calls (to Jira and Xray related endpoints) using a reverse proxy
   • Evaluate what REST API calls are being used, discuss their real need with users
   • Restrict access to REST API calls
3. Export results endpoint (i.e. /rest/raven/1.0/testruns) allows you to include custom fields from the Test issues in the response, using includeTestFields parameters; please choose carefully what fields you choose to include, as some of these may be calculated and thus add some additional overhead to the request.
4. Whenever searching for issues using Jira's REST API (i.e. api/2/search), please choose explicitly what fields to return using the fields parameter; that will avoid including unnecessary fields (e.g. "Requirement Status, Test Count, Test Set Status, Test Execution Defects, Test Plan Status) that are included by default and that add overhead to the request. This can be aggravated if this endpoint is used automatically by some integration with an external application. This is relevant for "requirement" like issues, Tests, Test Sets, Test Executions and Test Plans.

4. Manage JQL functions

Xray for Server and Datacenter provide dozens of JQL functions but you have to use them carefully to make sure your instance is not affected. It's recommended that you train your users on JQL before "allowing" them to use JQL functions. Using unoptimized JQL queries can degrade performance substantially. Note: This section does not apply to Xray Cloud users as there are no Xray specific JQL functions.

Some JQL functions, such as the ones dealing with requirement coverage, might be more intensive than other ones, since Xray may have to, for example, load all the related Test Runs in order to obtain relevant data. Some care should be taken with the following JQL functions:

1. testPlanTests() - whenever by tests in a given status; the current workaround is to search using the "TestRunStatus" custom field.
2. requirements() - whenever filtering by dates, as shown in the following example:
     requirements('OK',
               'Calculator', 
               'v1.0',
               'chrome'
               'false'
               '2014-08-30')
3. testExecutionTests() - will depend on the number of Tests you have on the Test Execution

5. Refine your Reporting

Xray offers a range of built-in reports to analyze your projects in different ways, including testing progress, current requirements status, and coverage analysis. Gadgets are also a great way to share information between team members and even amongst different teams. However, if not used carefully, they can degrade Jira performance.

For example; if all users have the same report on their dashboard, they will probably generate multiple requests once users access the dashboard. Thus, use the most intensive gadgets such as the “Historical Daily Requirements Coverage” and the “Tests Evolution” gadget with care. 

Try to use the built-in reports and gadgets for as many of your needs as possible, and use more advanced solutions only when it’s necessary. 

With Xray, it’s possible to keep your reporting simple but powerful.

• If you need custom reports in PDF/Word, try to use a solution that avoids additional work. For example, Xray has a built-in integration with Xporter which you can use to download reports in multiple format types
• Also, if you’re looking for BI reporting with drill-down capabilities, eazyBI is the proper solution.

6. Fine-tune your administration and customization

Xray comes with default settings that are ok for most organizations. Nevertheless, have a look at the different settings; it’s an opportunity for you to get to know the product better and to think about ways to fine-tune it later.

1. Review the settings "Max Test Runs for bulk operations", "Max number of requirements per report or gadget results", "Max number of Tests per Test Plan", "Max results per request" in the Miscellaneous administration settings. The default values should be ok.

Using the requirement coverage strategy "Use versioned Test Sets for Requirement Coverage" will require additional computational resources on daily usage; therefore, the default "Use versioned Test Executions for Requirement Coverage" strategy is the recommended one. Learn more in Requirements Coverage. If you change this setting, then it can take a while until all statuses are re-calculated depending on your instance dimension.

Follow these 6 Best Practices to ensure that your organization scales smoothly in Jira and Xray

As with every app, it's important to stay on top of the latest best practices. Make sure your Jira is configured properly, you have a well-defined process, you're using Xray entities optimally, you're managing your CI effectively and have the necessary reporting capabilities to keep you Agile.

If you have recently adopted Xray, are planning to, or have already been a long-time user, implement these steps to have a peace of mind knowing your performance is optimized and your testing activities will run smoothly. 

Like any app, we are constantly updating and improving performance so that you have the highest quality experience. Haven’t tried Xray for your Test Management yet? Start a free trial and see why 137 of the Global 500 have chosen Xray.