Test execution is one of the most time-sensitive stages of software delivery. Teams are expected to validate functionality, ensure stability, and support release decisions within increasingly shorter development cycles. Even with strong automation in place, there is rarely enough time to execute every Test before a release.
This makes prioritization a critical part of the QA process. Identifying which Tests should be executed first directly impacts how quickly teams uncover issues and how confidently they can move toward a release.
Most teams already have the data they need. Historical Test Runs, Defects, Requirements, and Test metadata are all captured in Jira and Xray, providing valuable insight into application behavior and testing risk. However, this information is not always used consistently to guide execution planning.
Powered by Sembi IQ, Xray's AI Test Prioritization brings these signals into the decision-making process. By analyzing historical testing data together with user-defined context, it intelligently prioritizes Tests within Test Plans and Test Executions, helping teams focus on the areas that matter most during every execution cycle.
Why Test prioritization remains a challenge
Prioritizing Tests is not a new concept, but the way it is often performed creates unnecessary inefficiencies. In many organizations, execution order depends on experience, urgency, or familiarity with the application. While this approach may work for smaller projects, it becomes increasingly difficult to maintain as products grow and teams scale.
Different testers may prioritize Tests differently, and new team members often lack the historical context needed to make consistent execution decisions. Meanwhile, every release generates valuable information through Test Runs, Defects, and Requirements, yet much of that intelligence remains underused because manually interpreting it takes time.
The result is an execution strategy that relies heavily on individual judgment instead of the testing intelligence teams have already collected.
Introducing Xray's AI Test Prioritization
Xray's AI Test Prioritization analyzes multiple signals to determine the most effective execution order for Tests within Test Plans and Test Executions.
The analysis considers a combination of factors, including:
- Historical Test Run results
- Associated Defects
- Recent and overall failure rates
- Test flakiness
- Test metadata, including priority, labels, Test type, and status
- Requirements and prerequisites
- User-defined prompts
- Optional Jira labels and configurable analysis timeframes
Based on these signals, the agent automatically re-ranks Tests according to their predicted relevance and risk, allowing teams to focus on the areas most likely to impact release quality.
The feature also introduces AI Priority Insights, a dedicated field that explains why each Test received its assigned priority. Every prioritization run is recorded in the Test Plan or Test Execution history, providing complete visibility into the prioritization strategy and the resulting execution order.
Availability by Xray plan
AI Test Prioritization is available across all Xray Cloud plans, with usage limits based on your subscription:
- Xray Standard: Up to 75 Tests
- Xray Advanced: Up to 150 Tests
- Xray Enterprise: Up to 300 Tests
How AI Test Prioritization works
Using AI Test Prioritization fits naturally into existing QA workflows.
Teams begin by opening a Test Plan or Test Execution and describing, in natural language, how they want Tests to be prioritized. They can further refine the analysis by selecting an analysis timeframe and specifying up to three Jira labels to provide additional context.
The agent then analyzes the available testing data and automatically re-ranks Tests based on the selected context. The resulting execution order, together with AI Priority Insights, helps teams understand not only which Tests should be executed first, but also why.
What changes during Test execution
The impact of AI Test Prioritization becomes apparent as soon as execution begins.
Instead of reviewing long lists of Test Cases and manually deciding where to start, teams receive a prioritized execution order that highlights the Tests most likely to uncover critical issues. This allows critical functionality to be validated earlier, making initial execution results more meaningful and enabling teams to respond faster when issues are identified.
Even when time constraints prevent the entire Test suite from being executed, teams can proceed with greater confidence knowing that the highest-priority Tests have been executed first.
Using historical QA data to drive smarter Test execution
Every testing cycle generates information that can improve future execution decisions. Historical Test Runs reveal execution trends, Defects identify areas that have caused issues previously, and Requirements provide additional context about the functionality being validated.
AI Test Prioritization brings these signals together into a structured prioritization model. Rather than relying on memory or manually reviewing historical reports, teams receive data-driven recommendations based on actual testing activity.
As more Test executions are completed, the tool continuously analyzes new information, helping teams make smarter prioritization decisions over time.
Supporting consistency across teams
Consistency is essential for maintaining quality at scale. When execution priorities vary between individuals or teams, it becomes more difficult to apply testing standards consistently across projects.
AI Test Prioritization introduces a transparent and repeatable approach to execution planning. Because the feature explains why Tests receive their assigned priority and records every prioritization run, teams gain greater confidence in the process while maintaining full visibility into how execution decisions are made.
This also simplifies onboarding, allowing new team members to rely on a structured prioritization model instead of institutional knowledge alone.
Built for QA Leads, Test Managers, and Agile teams
AI Test Prioritization is designed for teams that need to balance speed with quality.
- QA Leads gain greater visibility into testing risk, supporting more informed release decisions.
- Test Managers can allocate testing effort more effectively by focusing execution on the Tests with the highest potential impact.
- Agile teams receive faster feedback during short sprint cycles by validating the most important functionality first.
AI Test Prioritization helps teams make better use of the testing intelligence they already have. By combining historical Test data, Defects, Requirements, and user-defined context, it provides a smarter, more consistent approach to deciding which Tests should be executed first.
Integrated directly into Jira, AI Test Prioritization enables teams to spend less time deciding where to start and more time validating the functionality that matters most.
FAQ: Xray's AI Test Prioritization
What is Xray's AI Test Prioritization?
AI Test Prioritization uses Sembi IQ to intelligently prioritize Tests within a Test Plan or Test Execution based on historical Test Run data, associated Defects, Test metadata, Requirements, and user-defined context. By automatically re-ranking Tests according to risk and relevance, it helps QA teams focus on the most important Tests first.
How does AI Test Prioritization work?
Users provide a natural language prompt describing how they want Tests to be prioritized and can optionally refine the analysis using Jira labels and a configurable analysis timeframe. The agent analyzes historical Test Runs, failure rates, Test flakiness, associated Defects, and Test metadata before automatically re-ranking Tests and providing AI Priority Insights that explain each assigned priority.
Which Xray plans include AI Test Prioritization?
AI Test Prioritization is available across all Xray Cloud plans:
- Xray Standard: Up to 75 Tests
- Xray Advanced: Up to 150 Tests
- Xray Enterprise: Up to 300 Tests
The feature must first be enabled by a Jira administrator through Xray's AI Hub.
What data does AI Test Prioritization analyze?
AI Test Prioritization analyzes historical Test Runs, associated Defects, recent and overall failure rates, Test flakiness, Test metadata, Requirements, user-defined prompts, optional Jira labels, and configurable analysis timeframes to determine the most effective execution order.
Does AI Test Prioritization replace QA decision-making?
No. AI Test Prioritization supports human decision-making by providing data-driven recommendations. Teams remain in control of execution and can combine AI-generated insights with their own project knowledge, release goals, and business priorities.
How does AI Test Prioritization fit into Xray's AI capabilities?
AI Test Prioritization complements Xray's AI-powered capabilities, including AI Test Case Generation, AI Test Model Generation, AI-powered Test Script Suggestions, and Xray's Rovo Test Plan Summarizer. Together, these capabilities support a connected workflow across Test design, execution planning, reporting, and release readiness.
