Are vision-based tests slower than selector-based tests?

Per step, yes: a vision model spends seconds deciding where to tap, our published p90 is 4.81 second, where a selector lookup takes milliseconds. Per suite, usually no: scripted suites spend most of their wall-clock in waits, retries, and reruns of flaky failures. And speed without accuracy is negative speed, in our benchmark, the fastest models failed hardest.

Most vendors would answer this question with a shrug and a subject change. We'd rather answer it with the numbers, including the ones that don't flatter us.

The per-step truth, conceded up front

Finding an element by selector is a lookup, the framework queries a tree it already has. Milliseconds. A vision-based step is an inference: a model reads the rendered screen and produces coordinates. From UI-TapBench, our open benchmark (570 annotated screenshots, 20 production apps, Apache 2.0):

So yes: per decision, vision costs seconds. If the comparison ended at the step level, selectors would win it. It doesn't end there, for two reasons the step-level view can't see.

Reason one: Scripted suites don't run at lookup speed

Ask anyone who owns a large Appium suite where the time goes. It isn't element lookup. It's the waits padded around every step to survive timing flake; the retries configured because "it usually passes the second time"; and the reruns, the suite executed twice or three times because the first result had failures nobody trusts. A suite that finishes in 40 minutes but has to run three times before anyone believes it is a two-hour suite. Then add the human hours triaging which failures were real, the largest hidden line item, and a big reason maintenance consumes 60–70% of QA time on selector-based stacks. Millisecond lookups, purchased with hours of overhead.

Reason two: Speed without accuracy is negative speed

The sharpest finding in our benchmark wasn't ours to brag about, it belongs to the models that lost. GPT-5.2 posted a faster p90 than several stronger models (5.0s) while scoring 44.83% accuracy. A fast wrong tap is the most expensive event in mobile automation: it doesn't just fail its own step, it corrupts app state and poisons every step after it. Mobile flows are sequential, at 94.51% per-tap accuracy a 15-step flow completes cleanly far more often than at 92.98%, a 1.53-point gap that compounds into a ~30% higher probability of end-to-end success. Now price a wrong tap in time: the flow reruns, or worse, a human investigates a corrupted-state failure. Per-step seconds are noise against that.

The metric that matters: Time to trustworthy signal

The question your release process actually asks is not "how fast does a step execute" but "how long until we have a suite result we believe." That clock includes: execution time × the number of runs it takes to get a clean signal + human triage of the failures. Vision-based steps make the first term modestly larger and collapse the second and third, fewer flake-failures to rerun, and when something does fail, failure reasoning states why, turning triage from an investigation into a read. Selector-based steps minimize the first term and inflate the other two, which is how "fast" suites end up being the slow way to a decision.

Where per-step seconds genuinely matter, the honest section

Symmetry again. If your app's UI is unusually stable, your suite is enormous, and your CI budget is tight and serial, per-step inference cost is real and you should do the math, suite size × steps × seconds is not nothing. Two things blunt it in practice: suite wall-clock is a fleet property (tests parallelize across devices, so throughput scales horizontally in a way per-step latency doesn't gate), and inference latency is an engineering number that keeps falling, while selector flake is structural, it's what happens when tests read code instead of screens. But if your suite is stable, trusted on the first run, and rarely maintained: keep it. That suite is rarer than its owners believe, and the way to find out is to measure.

Don't take our word for it, run both clocks

The benchmark is open, 570 annotated screenshots under Apache 2.0; run your own models against it and publish what you find. And for your own app, run the two-clock test for a sprint: for each stack, log wall-clock per run, number of runs until a trusted result, and human minutes spent on triage. Compare totals, not step timers. That number decides the question for your team better than any vendor's blog, including this one.

FAQs

How much latency does a vision step actually add?

‍Our published p90 is 4.81 seconds per tap decision; comparable models range from ~5 to ~12 seconds. Device actions and app response time exist in both paradigms, the inference is the added cost, and it's the number to ask any vision vendor to publish.

Won't a 500-test suite be unusably slow at seconds per step?

‍Suite wall-clock is governed by parallelism, not per-step time — runs scale across devices. The suites that miss CI windows in practice are the ones being rerun to outvote their own flakiness, not the ones spending seconds per decision once.

Why did faster models score worse?

‍Speed and spatial precision are unrelated capabilities. General models answer quickly but lack pixel-level calibration on dense mobile UIs — high recall, catastrophic precision: they confidently tap the wrong thing. In sequential flows, that's worse than a miss.

‍