•
Drizz raises $2.7M in seed funding •
•
Featured on Forbes
•
Drizz raises $2.7M in seed funding •
•
Featured on Forbes
Automating more tests without careful planning doesn’t improve coverage. It just helps you run tests faster. This is where test prioritization strategies become crucial.
Many teams end up prioritizing tests that are easy to automate, rather than the tests that are most important to the business.
Tests that are critical and need to be done frequently, such as a login test, warrant a greater level of coverage, and test automation. Tests that drive revenue, like important payment flows, deserve greater coverage. Other less important screens, such as an unstable screen that is rarely used, likely don’t need test automation.
Keeping costs in mind, automation of tests that cover the most business critical functions, and are executed frequently and are less stable, should be the focus of test automation strategies.
The goal is not to automate everything. It is to automate tests that reduce maximum risk, manual effort and deliver the highest ROI. It is equally important to determine which tests should not be automated at all. Determining this requires a test prioritization scorecard.
In this article, we will introduce a practical scoring model that mobile teams can use to prioritize tests based on the automation ROI.
A test that is small in a test management tool may encompass several combinations of devices and OS. A checkout test, for example, could include:
If you automate tests without thinking about all of this, your test suite may grow, but your coverage may still be weak in the areas that are the most important to your users.
Prioritization helps you make the best use of your limited time and direct your efforts toward areas with the highest risk, usage, business value, and likelihood of repeatability.
A scoring model needs to be simple enough that the team can apply it consistently. Score each candidate test on a scale from 1 to 5.
A basic priority score is:
Priority Score = Value Score × Execution Score × Feasibility Score
Value Score measures business risk, user impact, and regression risk, while Execution Score measures test frequency and manual effort. Feasibility Score considers how stable the feature is, how testable it is, and the difficulty and cost of automation.
A more complex weighted model is more appropriate for larger test inventories:
Score each factor from 1 to 5, multiply the factor weight by the score, and add the results.
For maintenance cost and automation complexity, reverse the scoring. A low maintenance burden receives a high score. A high maintenance burden receives a low score.
This scoring approach attempts to balance trade-offs. A test for a highly stable but low-impact workflow should not outrank a test for a highly stable payment or authentication workflow, even if the former poses lower business risk.
A measure of business risk is the impact of failure.
Activating users by successfully completing the onboarding workflow is essential. Failing payments prevents revenue. Failing permission or security tests may allow access to sensitive data and information, and expose the organization to legal and compliance risk. Score business risk failures that result in the loss of revenue, trust, compliance, or privacy high.
A simple scale could include:
1: Minor inconvenience with a simple workaround
3: Noticeable user disruption or support impact
5: Revenue loss, security exposure, compliance risk, or major service interruption
A test executed during every pull request leads to greater automation savings than a test executed only once during every release.
Separate user frequency from team execution frequency. A feature frequently used by customers may only be tested during a monthly regression cycle. Both of these should be considered.
Assign higher scores to tests executed with high frequency by users, testers, or automated builds.
Some workflows are more likely to break because of high reliance on shared services and high rates of change.
Features like payment, authentication, search, notifications, account synchronization, and navigation often break and have many dependencies. Consider recent code changes and defects when scoring the likelihood of regression.
A good approximation of regression likelihood is represented by:
Manual effort includes setup, execution, data preparation, device selection, and checking and verifying the results.
A 30-minute test that is executed once may require less effort than a 5-minute test that is executed across multiple devices. Be sure to estimate the effort required for the entire test matrix.
Automate highly manual efforts first, because high manual effort leads to greater savings.
More stable workflows are easier to automate and maintain. Less stable workflows should not be scored too high unless a greater risk is present.
Look at:
Stability should help determine the timing of automation, but a lack of stability should not eliminate the need for coverage altogether. For example, a high-risk test with a low stability score could go in the later stages of automation with better test hooks or API coverage implemented first.
Testability and the system’s ability to expose evidence of success or failure in a reliable manner are closely related. In general, testability is better with mobile applications that:
If testability is low, record the effort that must be undertaken to improve it. Adding test IDs or reset APIs might provide more long-term value than brittle test scripting.
Estimate how much effort it will take to create, troubleshoot, execute, and maintain the test.
Tests with complex biometric flows, third-party payment screens, device-specific behavior, and data that suddenly stops working may need ongoing support. Consider these.
Maintenance cost is often underestimated. A test that fails frequently creates the perception that the entire test suite is unreliable and a waste of time. Such tests lose their value and require repeated effort to resolve. Tests with high flakiness should be scored lower for immediate automation; however, the team should address the underlying instability first.
Let’s consider a case where a group of three mobile tests are being evaluated by a team.
The checkout test has higher business risk and more manual input than the login test, but is less stable. However, it still requires automation sooner rather than later.
The login test may be automated first. This will provide automation for the foundation of the other tests. Next, the checkout test should be the focus with improvements that enhance testability. This may include stable payment mocks and test accounts. The profile theme setting can remain manual until it is used more frequently.
Most mobile teams start with essential tests that include:
The first step is to establish a reliable happy path. Once failure paths become more important, devices may become more of a focus based on the application data.
For example, a checkout test roadmap may focus on successful payments. Next, it may cover declined payments, payment timeouts, lost networks, automatic submission of payment forms, and confirmation of the order.
A well-balanced strategy combines people with automation for exploration, interpretation, and finding risks, while automation performs repetitive verification.
Group tests into three priority bands as follows:
Automate First (Priority 1)
These tests are high impact and have high execution frequency, low execution time, and are stable. Consider including them in pull requests/builds.
Automate Next (Priority 2)
While they may have slightly lower impact, these tests require improved test data, better selectors, better environment and device coverage, or better automation. Create the requisite foundation prior to creating the full automation flow.
Keep Manual for Now (Priority 3)
These tests have a low risk, low frequency, and require high human judgment or are high-maintenance. Consider them during roadmap updates.
Update the scores after large product changes, the introduction of new devices, architecture changes, or significant shifts in release frequency.
Drizz helps in organizing, prioritizing, and strategizing mobile testing and is especially helpful once the execution of the automation strategy and the test coverage is aligned in mobile environments.
Insights into tests, coverage, execution, and maintenance help teams improve test scores over time. This helps teams zero in on tests providing valuable release confidence: good behavior, failing due to product defects, and distracting failures caused by flakiness.
Drizz builds a more realistic automation plan because teams can evaluate test automation and the associated customer devices and environments. This helps teams move beyond viewing every combination of devices as equally important. Instead, they can prioritize coverage based on failure likelihood and risk.
The overall goal is to provide coverage on the user-centric flows that have the most impact on the UX and the business.
In mobile automation, the first step is prioritization. Prioritize business value, risk, frequency, regression likelihood, execution, complexity, and maintenance.
Automate workflows of high value and high repetition first. Prioritize testability of unstable and low-value workflows last. Preserve exploratory and judgment-based testing performed by skilled testers. Review scores when your products, users, devices, and release processes change.
Drizz offers teams the visibility they need to help execute mobile tests, manage devices used for coverage, and track failures and trends for maintenance. Along with a pragmatic scoring framework, Drizz can help teams define automation as a clear investment in release confidence.
©Drizz 2026 • All Rights Reserved
