In one minute
Every model did worse on harder tasks, but not by the same amount.
Share of tries that passed, out of 50 per model on each set of tasks.
Fig. 01 — Pass rate out of 50 tries per model on each set of tasks.
Abstract
Six frontier model configurations, driven through Fathom, Drizz’s Android agent, attempted twenty no-login tasks that cross between apps, on two emulator environments. Every verdict was graded from device state or host-side evidence, never from what the agent said it did.
The tasks ran as two suites of ten, on different Android images, months apart. They should be read separately: pooled together they describe neither, and the models’ ranking changes between them.
Fig. 02
Easier tasks vs harder tasks
Finding 1
The easier tasks (Suite I) passed 72.3% of the time. The harder ones (Suite II), which move precise values between apps and are checked against each app’s own data, passed 28.7%. Averaging the two gives 50.5%, but that number only reflects the mix of tasks, so we don’t report it as a score.
Finding 2
That was expected. The two leading Gemini Flash models dropped from almost perfect to around 40%. Each line is one model: its pass rate on the easier tasks on the left, on the harder tasks on the right.
Finding 3
GPT 5.6 Luna fell 60 points, from 64.0% to 4.0%: two passes in fifty runs. Gemini 3.5 Flash, the weakest on the easier tasks, lost only 20 and finished ahead of every GPT configuration. So a single leaderboard can mislead: how good a model looks depends on which kind of task you test it on.
Finding 4
Among the easier tasks, every one was passed by some model. Among the harder ones, five were never passed by any model: zero passes in 150 tries. The other five passed between 20% and 80% of the time. We haven’t yet worked out why those five failed, so we report them as open questions.
Finding 5
Every try gets two marks: what the model did (passed, failed, or broke a rule), and whether our evidence for that try is reliable. On the harder tasks, models broke a rule 11 times. Each of those also left the phone in a state we couldn’t reset, which is the worst kind of failure because it can spoil the next try.
Fig. 07 — Explore
600 tries: 20 tasks, 6 models, 5 tries each. Hover or tap a group of squares for details.
Five of the harder tasks were never passed by any model.
Highlight one model
Each group of five squares is one model’s five tries at one task. Which try passed isn’t recorded here, so passes are drawn first.
Fig. 08 — Specimens
Why this task is interesting.
Method
Setup
Every task is executed by Fathom, the Android agent built at Drizz. The model supplies the planning; Fathom reads the screen, chooses and dispatches actions, retries, and decides when a run is going nowhere. What is measured is therefore the Fathom plus model system, not a model on its own.
Temperature 0.0, cache off, model versions pinned, and a step budget of 60 to 120 steps depending on the task. The harder set scheduled 300 runs and stored 316 attempts: one contaminated run held outside the scored set, eleven second attempts and four third attempts. Nothing was pruned.
Cite this report