•
Drizz raises $2.7M in seed funding •
•
Featured on Forbes
•
Drizz raises $2.7M in seed funding •
•
Featured on Forbes
Most mobile testing tools break when they have to interact with a complicated app flow. Mobile AI and automation tools are often able to understand and interact with screens and elements, only to tap buttons 40 pixels away from the actual element.
With mobile testing, precise interaction matters. The AI Agent has to understand, locate, and interact with the screen element. Wrong clicks also invalidate all the subsequent interactions. The app could open the wrong screen and make the rest of the test yield incorrect results.
For this reason, we do not consider general model intelligence as sufficient evidence of practical CI/CD capability. We also evaluate how well an AI Agent is able to carry out the task after understanding the screen.
To measure the performance of mobile UI and interaction automation systems, we can give a model a screenshot and a tapping instruction, and see if the model can tap where it should.
In this guide, we discuss what matters when measuring mobile automation at the tap level, why tap-level precision is vital to mobile CI/CD, and how UI-TapBench measures performance across 570 screenshots of 20 production apps. It also highlights the difference between understanding a mobile interface and interacting with it reliably.
A test flow typically has 15 to 20 individual actions. Each action introduces the chance for failure. An AI agent that achieved 95% accuracy for each tap would have a 46% chance of completing a 15-step flow with no errors.
Claims of “agent x understands app y” have diminishing returns. The impact of small drops in accuracy of each tap decreases along with the number of flows completed.
Incorrect taps create noisy results. If the third action in our test flow opens the incorrect screen, the next 12 assertions may fail, even though the application has no 12 separate defects. The failure occurs as a result of a single, incorrect action.
Automation that takes work to maintain does not provide any benefits as a result of a fast release pipeline. The goal should be to remove obstacles.
UI-TapBench measures the ability to perform precise taps on the mobile UI. Each benchmark provided includes the following:
The model is presented with the image and the text instruction, and predicts the tap location. A prediction is considered correct if it falls within the target region.
This design simplifies the more complex variables in mobile agent benchmarks and does not measure long-term planning, app state management, API calls, or recovering after a failed action. Instead, the goal is to know if the model clicked on the correct image.
The benchmark consists of 570 screenshots gathered from 20 production applications in the areas of food delivery, travel, communication, entertainment, finance, health, education, productivity, and other apps such as Uber Eats, Airbnb, WhatsApp, Spotify, Netflix, Duolingo, PayPal, Strava, Notion, and Todoist.
The applications include the conditions in which visual agents encounter challenges:
The dataset is available under an open license, Apache 2.0. Researchers and teams are allowed to download the screenshots and annotations, build their own models, and benchmark against them using the same evaluation metrics.
UI-TapBench evaluated six models with zero-shot prompts, and a predictable assessment protocol. From the results, it was clear that the ability to understand the screen differed greatly from the ability to perform precise interactions.
The most important outcome is not just the ranking. It differentiates models that identify the exact tap target from those that understand the instruction conceptually but fail to predict the precise coordinates.
This illustrates why general-purpose language or vision benchmarks are inadequate to reflect the performance of mobile automation. Describing a button and tapping it are different things.
The densely packed interface of mobile applications poses challenges to models. The instruction “Tap the second option” requires not just recognition of elements but an ability to determine their respective boundaries and to map the selected instruction to screen coordinates.
There are several complicating factors.
A list may have six rows with nearly identical design and spacing. The model has to differentiate the second row from the first and the third, and then select the coordinates that are well within the correct bounding box.
Predicting a tap close to the boundary of a row poses a risk. A slight error can place the tap on an adjacent row or on an unfunctional region of the list.
General-purpose visual question answering and multimodal models are not calibrated for mobile automation to include a device coordinate space, screen resolution, touch targets, scrolling, and platform design.
This skill is more narrow than general image understanding. Training with more images may not lead to better tap placement.
A model could have high recall but low precision if it tried to place a tap on almost every screen.
In mobile testing especially, a wrong tap is more problematic than a missed tap. A missed tap may result in a retry or a clear failure while a wrong tap navigates to a different screen and corrupts all subsequent steps.
In CI/CD, the time to perform an action and the time to wait for the action to be confirmed cannot be neglected. A fast incorrect action may take longer than a slow correct one.
Recovery time from failed interactions must also be taken into consideration.
Drizz scored 1.53 percentage points higher than Qwen 3.5-27B. From this perspective, the difference is small.
Assuming a pipeline runs 20 times a day and performs 1000 taps in each run:
Small changes at the action level can have substantial effects at the scale of a large software development and continuous integration pipeline.
Teams should therefore look at the impact that model accuracy has at the flow level, and not just at the tap level. Some important metrics are:
'End-to-end' benchmarks determine if an agent achieves a goal. Such benchmarks help determine planning, navigation, memory, and recovery.
However, such benchmarks do not guarantee that all individual actions are correct. An agent can tap on the incorrect element, recover through another path, and still achieve the goal. The agent may take erroneous steps but still successfully achieve the final goal.
UI-TapBench concerns itself with the correctness of actions. It is therefore complementary to existing high-level benchmarks.
Evaluating mobile app testing should be a combination of both:
AI-powered mobile tests fit best in a layered approach to testing. During pull requests, teams should run a small smoke suite covering important business flows. These tests have to be fast and predictable. If an agent taps inconsistently, the pipeline should capture that as a failure with a screenshot, predicted tap location, tap region, and application state.
Once the pull request is merged, a full regression suite can run across different device models, OS versions, and screen size form factors. This stage is used to identify layout shifts that affect spatial grounding.
Prior to release, teams should run long end-to-end smoke tests involving payment flows, account changes, user onboarding, and other critical business flows. At this stage, the accuracy of taps should be addressed along with the recovery behaviors and failure analysis.
Drizz's platform fits into modern workflows by leveraging AI to reduce the maintenance burden of test automation and reliably interact with interfaces.
Teams evaluate Drizz based on metrics like tap accuracy, flow completion, test stability, and time-to-failure diagnosis.
Evaluation of mobile AI testing tools should occur with measurable goals. Before adoption of AI testing tools, vendors should be asked for evidence in the following areas:
1. Spatial accuracy
What per-action accuracy is provided for mobile interfaces? What does the vendor mean when they say tap is correct?
2. Real application coverage
Is the vendor's app under evaluation a real mobile app (not demo app) with complex UI, such as menus, lists, controls, etc.?
3. Reproducibility
Are the benchmark dataset, prompts, scoring method and limitations available? The publication of data facilitates validation of results.
4. Sequential reliability
Reliability of tool action should not be evaluated by the success of an individual action. A tool that excels in single-step actions, may still struggle with scrolling actions, input of keystrokes, loading actions, and unpredictable dialogs.
5. CI/CD integration
Find out how the tool reports failures. The output must contain screenshots, a history of actions, coordinates, details of the device and sufficient information for an engineer to understand the failure.
AI agents need more than screen reading comprehension to work well in mobile CI/CD. They need accurate spatial grounding, prediction of latency, smart recovery mechanisms and clear indication of failures.
UI-TapBench provides a means to measure essential aspects, such as whether an agent taps the desired location, measuring one of the most important aspects. Its 570 screenshots across 20 production apps show a gap that other broader task completion benchmarks attempt to hide.
The dataset is available for independent evaluation, and its limitations are known. Currently, it focuses on Android, English-based single-action interfaces and works best for simple tasks. Future iterations may include iOS, multilingual screens, additional task difficulty and multiple task flows.
For engineering teams, the solution is simple. Test the model on the screens your product uses, measure each action in isolation and in the context of the entire product, and gauge the impact of modeling errors. Mobile testing of AI begins with discussing the taps.
©Drizz 2026 • All Rights Reserved
