•
Drizz raises $2.7M in seed funding •
•
Featured on Forbes
•
Drizz raises $2.7M in seed funding •
•
Featured on Forbes
Key takeaways
Test optimization is systematic process of cutting test suite runtime and maintenance overhead without losing bug-catching power that justifies suite existing.
Generic posts on this topic give you a list: prioritize, parallelize, deduplicate, add AI. That list isn't wrong, it's unordered. Apply those tactics in wrong sequence and you'll burn three months building infrastructure on top of a foundation that doesn't hold.
Definition: test optimization is practice of reducing test suite runtime, cost, and maintenance burden while maintaining or improving defect detection.
The two halves matter equally. A suite that runs in 3 minutes but misses regressions isn't optimized, it's broken. A suite that catches every bug but takes 4 hours per CI run isn't optimized either, because nobody waits for it.
Optimization is not:
Optimization is:
The first two require data. The second two require infrastructure. Most teams skip to infrastructure and wonder why savings evaporate.
Six steps, in this order. Skipping any of them undermines next.
Before optimizing anything, you need data on what's in suite. For each test:
This usually surfaces shock data. A typical 1,000-test mobile suite includes 200 tests that haven't failed in 6 months, 80 that have failed every week for a year (all flakes), and 50 that take 5x longer than average.
You can't make optimization decisions without this inventory. Cutting "slow tests" without knowing which ones catch real bugs is how teams lose regression coverage in name of speed.
Fixing flakes is a prerequisite for any real test optimization effort. Parallelizing a flaky suite multiplies flakiness instead of cutting runtime, because failures trigger reruns, which trigger more failures.
The math: a suite with 90% pass rate per test and 200 tests has a 0.9^200 = ~7×10⁻¹⁰ chance of a clean run. In practice, you'll never see green. Add parallelism and you'll see more flakiness, not less.
Fix flakes first. Quarantine worst offenders. Get suite to 97%+ pass rate before you touch anything else.
With inventory in hand, find tests that catch nothing rest of suite doesn't.
Three types of redundancy to look for:
The cut isn't arbitrary. Use mutation testing or coverage analysis to verify that removing a test doesn't reduce bug-detection. Tools like PIT (Java) or Stryker (JS/TS) introduce small code changes and check whether tests catch them. Tests that catch zero mutations are candidates for deletion.
Cut conservatively, but cut. A leaner suite is easier to maintain, faster to run, and clearer in its purpose.
Not every test runs every time. Categorize suite into tiers:
Each tier has a different time budget. Smoke must finish in 5 minutes. Impacted subset in 15. Full regression in 60. Cross-platform whenever, as long as it finishes before release decision.
The point of tiering isn't gatekeeping. It's matching test cost to feedback urgency.
Now you can parallelize without making things worse. Tests are stable, redundant ones are gone, tiers are sequenced. Parallelism multiplies savings from steps 1-4 instead of compounding existing problems.
Parallel execution is fastest path to meaningful test optimization, but only after suite is in shape to handle it. A 200-test stable suite with proper isolation runs in parallel across 20 workers in 5 minutes instead of 90.
Strategies that compound:
This is step nobody writes about. A suite trimmed from 1,000 to 600 tests grows back to 1,000 in 6 months if there's no discipline around what gets added.
Cutting maintenance is one half of test optimization; cutting runtime is other. They're related but not identical. A test that runs in 200ms but breaks every UI change adds more maintenance cost than runtime cost.
Maintenance rules that work:
Without these, optimization is a one-time event, not a practice.
Don't optimize on instinct. Track these four numbers per suite:
Pass rate decides whether to stabilize or parallelize next. Mean runtime decides whether to shard or optimize individual tests. Detection rate decides whether you cut too much. Maintenance cost decides whether suite design is sustainable.
According to 2024 DORA State of DevOps report, elite-performing teams keep change failure rates under 5% and recover from incidents in under an hour. Both of those depend on a test suite that runs fast enough to give pre-merge feedback and reliably enough to be trusted.
Web optimization advice translates partially to mobile. The mobile-specific concerns:
Build install time. A 30-second test takes 90 seconds on first run because app installs first. Across a 200-test suite with no install caching, you've burned 12 minutes on installs alone. App install reuse across tests is single biggest mobile-specific win. Most cloud device platforms support this, but it must be configured explicitly.
Real-device queue depth. A 20-device farm runs 20 tests in parallel. If suite has 500 tests at 1 minute each, that's 25 minutes minimum, plus queue time when multiple PRs land. Optimization here is matching device pool size to peak PR volume, not just average.
OS version matrix combinatorics. A test that runs on iOS 15, 16, 17, 18 and Android 11, 12, 13, 14 = 8 environments per test. Across 200 tests, that's 1,600 test executions. Use risk-based device selection: critical flows on full matrix, secondary flows on latest-version only.
Cold start vs warm start tests. A test that runs first hits a cold app. The same test running 50th hits a warm app with cached state. Different code paths, different timing, different results. Optimize by explicitly choosing cold or warm start per test instead of letting it depend on shard order.
Per-device flakiness aggregation. A test that's 99% stable on Pixel and 70% stable on Galaxy A52 shows up as "85% stable" in aggregate. The aggregate hides where to optimize. Track per-device pass rates.
Some commonly-recommended optimization tactics are oversold.
AI test generation. Generating tests faster doesn't help if you're already drowning in tests. The bottleneck is rarely test authoring speed, it's test maintenance and runtime. Generating 500 more tests makes both worse.
Self-healing locators. Useful, but bigger win is using a framework that doesn't depend on locators at all. Self-healing patches symptom (brittle selectors). Vision-based testing removes cause.
Massively parallel execution beyond a point. Past 30 to 50 parallel workers, marginal speed-up flattens because of orchestration overhead, device pool contention, and serialized setup costs.
More devices in matrix. Each added device class adds cost without proportionally adding coverage if your user base is concentrated on a few models. Pull analytics on actual device usage and prioritize top 80% of installs.
Faster machines. A faster runner cuts wall-clock time linearly. Sharding cuts it logarithmically. Sharding wins for any suite over ~100 tests.
Continuous testing means tests run constantly, on every commit, in background. It only works if tests are optimized. Otherwise, "continuous testing" becomes "continuous waiting."
A continuous testing pipeline that holds up has these properties:
The numbers above assume an optimized suite. On an unoptimized suite, "continuous testing" turns into a flaky 4-hour CI run that engineers learn to ignore.
Google's Engineering Productivity Research has published on diminishing returns of test investment past a certain point. Their finding: each additional minute of test runtime past 15 minutes per PR reduces developer commit frequency more than it improves quality. The point isn't to test less, it's to test smarter.
If you're starting from a 1,000-test mobile suite with 80-minute CI runs and 85% pass rate, here's sequence:
Week 1: Inventory and stabilize. Pull data on every test. Identify 50 worst flakes. Quarantine them. Don't try to fix them yet, just stop letting them block CI.
Week 2: Cut redundancy. Use coverage analysis to find tests that don't add unique value. Cut 10 to 20% of suite. Confirm detection rate stays flat.
Week 3: Tier and sequence. Define smoke, impacted, full regression tiers. Move smoke tier to every PR. Move full regression post-merge. CI per PR should now finish in under 20 minutes.
Week 4: Parallelize and shard. Time-based sharding within each tier. App install caching. Critical path tests first per shard. Target: full regression in under 30 minutes.
Ongoing: Maintain. Owner per test. Flake budget enforced weekly. Quarterly audit. New test policy that requires unique coverage justification.
A team running this sequence cleanly cuts CI time by 60 to 75% over 30 days without losing detection capability. The remaining gains come from infrastructure changes (better hardware, more devices, smarter test selection) and tooling shifts (vision-based testing for UI flake reduction).
For E2E mobile tests, framework decides whether test optimization is a one-time win or a permanent gain.
Selector-based frameworks (Appium, XCUITest, Espresso) tie tests to accessibility IDs, resource IDs, or XPath. Every UI refactor breaks tests. The maintenance cost grows linearly with codebase change velocity. Optimization gains erode as suite ages.
Vision-based frameworks find elements by what they look like, not by selectors. Drizz tests written as Tap "Add to cart" keep working when dev team renames resource ID or restyles button. The optimization gains compound instead of eroding, because suite doesn't accumulate maintenance debt with every UI change.
For high velocity mobile teams, this is difference between a suite that stays optimized and one that drifts back to broken within a release cycle. Optimization isn't just about runtime. It's about whether gains hold.
The process of reducing test suite runtime and maintenance cost while keeping or improving defect detection. Cutting tests without losing coverage.
Inventory and stabilization. Measure what each test catches and runs, then fix flakes. Cutting or parallelizing before this skips foundation.
Parallel execution multiplies failure rate of flaky tests. A 90% stable test that runs 200 times in parallel is statistically certain to flake. Stabilize first.
Most mobile suites can cut runtime 60 to 75% in 30 days through inventory, stabilization, redundancy removal, tiering, and parallelization, without losing detection.
TIA is one tactic inside test optimization. It picks which tests to run for a given change. Test optimization is broader practice including stabilization, cutting, tiering, and maintenance.
For UI maintenance (self healing locators, vision-based element matching), yes. For test generation, less than vendors claim. The bottleneck is usually maintenance and runtime, not authoring speed.
©Drizz 2026 • All Rights Reserved
