Key takeaways
- A false positive is a test that fails when app is working correctly. A false negative is a test that passes when app has a real bug.
- False positives destroy trust in test suite. False negatives give false confidence. Both are damaging, but in different ways.
- Flaky tests are biggest source of false positives in automation. Shallow assertions are biggest source of false negatives.
A false positive means test says "broken" when nothing is actually broken. The alarm went off, but there's no fire. A QA engineer spends 30 minutes investigating, only to find app is fine and test itself was problem.
A false negative means test says "passing" when something is actually broken. The alarm didn't go off, but building is on fire. The team ships build with confidence. A user reports bug from production two days later.
Both are wrong results. They fail in opposite directions, and they damage different things.
What false positives actually look like
A checkout test fails at 2 AM during nightly CI run. The on-call engineer checks logs. The payment sandbox was rate-limited because three teams were running tests against it simultaneously. The app is fine. The test environment wasn't.
That's a false positive. Here's how they show up in mobile testing specifically:
- A button moved 12 pixels down after a design update. The test's XPath selector, anchored to button's position relative to a parent container, can't find it anymore. The button works. The selector is stale.
- A test asserts that home screen loads within 3 seconds. It does on a normal run. But CI machine was under heavy load, emulator ran slow, and test timed out at 3.2 seconds. The app isn't slow. The infrastructure was.
- A push notification test fails because test device's notification permissions were toggled off by a previous test run that didn't clean up after itself. The notification system works. The test state was polluted.
- A login test fails with "element not interactable" because a system-level dialog ("Update available") appeared on top of login screen. The login flow is fine. An unexpected popup blocked test.
In every case, team investigates, finds nothing wrong with product, and marks failure as "not a bug." Multiply this by 5-10 times a week and you've lost a full engineering day to chasing ghosts.
What false negatives actually look like
A test suite runs across 200 test cases. All green. The build ships. Within 48 hours, three users report that app crashes when they rotate their phone during payment flow on Samsung Galaxy A-series devices.
The team checks test suite. There's a payment flow test. It passes. But it only tests portrait mode, on a Pixel emulator, with a single payment method. The rotation edge case on a Samsung-specific Android skin was never covered.
That's a false negative. The test existed. It passed. The bug shipped anyway. More examples:
- A test checks that order confirmation screen appears after checkout. It does appear. But test doesn't verify that actual charge was processed by payment gateway. The screen shows "Order confirmed," but backend silently failed. The test says pass. The user never gets their order.
- A test for image upload runs on an emulator. It passes because emulator uses a synthetic camera feed. On a real device, upload fails because app doesn't handle HEIF image format from the iPhone camera. The emulator can't reproduce bug because it doesn't generate HEIF files.
- A test validates app's behavior on a 4G connection. It passes. But 30% of user base is on 3G or spotty Wi-Fi, and the app shows a blank screen instead of a loading indicator when an API call takes longer than 5 seconds. The test never simulated slow conditions, so it never found bug.
- A regression test for a search feature asserts that results appear. They do. But results are wrong, returning items from a different category. The assertion checked for presence of results, not their correctness.
The common thread: test looked at the surface. It didn't look deep enough to catch what was actually wrong.
Why false positives are more dangerous than they seem
The obvious cost of a false positive is wasted investigation time. A QA engineer or developer spends 20-40 minutes looking into a failure that turns out to be nothing. That's annoying but manageable if it happens once.
The real damage is behavioral. False positives change how teams respond to test failures over time.
When a test suite produces false positives regularly (more than 2-3 per week), team develops a reflex: "it's probably just a flaky test." Engineers stop investigating failures promptly. They re-run test instead. If it passes on retry, they move on without root-causing original failure.
This is where false positives cause false negatives. A real bug triggers a test failure. The engineer sees failure, assumes it's another flaky test, re-runs it. The re run happens to pass (timing-dependent bugs do this). The engineer marks build as green. The bug ships.
This spiral is most expensive testing failure mode. It combines time cost of false positives with production cost of false negatives. Teams with high false positive rates almost always have hidden false negative problems because their triage muscle has atrophied.
The cost difference
The costs are asymmetric, and gap is wide.
False positive cost:
- 20-40 minutes of engineer investigation time
- CI pipeline delay if build is blocked pending investigation
- Gradual erosion of team trust in test suite (compounding cost)
False negative cost:
- A production bug report that needs triage, reproduction, and prioritization
- A hotfix development cycle (hours to days, depending on severity)
- An emergency release through app stores (2-24 hours for iOS App Store review, faster for Android)
- User-facing impact: crashes, data loss, broken transactions
- App store rating damage from one-star reviews during exposure window
- If false negative involved payment or data handling, potential compliance or legal exposure
One false negative in a payment flow can cost more than a thousand false positives combined. The math isn't close.
How to measure your rates
Most teams don't track either rate. Here's how to start.
False positive rate. For one week, categorize every test failure: real bug or false alarm. Divide false alarms by total failures.
- Under 10%: healthy suite. Trust is intact.
- 10-25%: manageable but trending toward alert fatigue.
- Above 30%: suite is actively hurting. Engineers are ignoring real failures.
False negative rate. Track production bugs for a month. For each one, check if a test existed for that flow. If it did and passed, that's a false negative.
This is harder to measure because you're counting things you missed. But even a rough number is useful. If 3 out of 10 production bugs had passing tests in suite, your false negative rate for covered flows is 30%, and your assertions need an audit.
Reducing false positives
- Replace fragile selectors (XPath based on layout hierarchy, CSS tied to class names that change with builds) with Vision AI or stable accessibility identifiers. Selector breakage is single biggest false positive source in mobile UI testing.
- Use explicit waits tied to element state ("wait until button is clickable") instead of fixed sleep timers. A 3-second sleep passes in your local run and fails in CI when machine is slow.
- Isolate test data per run. Tests that share a database account, a test user, or a payment sandbox token interfere with each other in parallel execution.
- Run tests on dedicated, stable infrastructure. Shared staging servers where another team's deployment can break environment will produce false positives that have nothing to do with your code.
- Handle system-level interruptions. On Android, unexpected dialogs (app update prompts, low battery warnings, "System UI isn't responding") break tests that don't account for them. Self-healing frameworks and popup agents handle these automatically.
Reducing false negatives
- Write assertions that verify outcomes, not just state transitions. Don't assert "confirmation screen appeared." Assert "confirmation screen shows order ID matching submitted order, with correct total and delivery address."
- Test on real devices across your actual user base's device distribution. The Galaxy A14 and Redmi Note 12 behave differently from Pixel 8 your test suite runs on. Bugs that only reproduce on specific hardware, camera modules, or Android skins are invisible on emulators.
- Cover flows where bugs actually hide: network interruptions mid-transaction, background-to-foreground restoration, screen rotation during form input, low storage, and multi-step flows where step 3 depends on state from step 1.
- Audit mocks quarterly. When real API adds a field, changes a status code, or modifies error handling, mock should reflect it. Stale mocks are false negative factories.
- After every production bug, do a retroactive: why didn't the test catch it? Add missing assertion or scenario to suite. Over time, this closes gaps that false negatives exploit.
Drizz reduces both rates by attacking root causes. Vision AI removes selector fragility that drives most false positives. Real device execution removes the emulator gap that hides false negatives. A built-in popup agent handles system dialogs that derail tests on real hardware. And because tests are written in plain English describing what the user does, assertions naturally match user visible outcomes rather than implementation details.
FAQ
What is a false positive in testing?
A test fails even though the app is working correctly. The failure is in the test, not the product.
What is a false negative in testing?
A test passes even though the app has a real bug. The defect ships to production undetected.
Which is worse, false positive or false negative?
False negatives ship bugs to users. False positives waste investigation time. The production cost is higher.
What causes most false positives in mobile testing?
Brittle selectors, timing-dependent waits, shared test environments, and unhandled system-level popups on real devices.
How do false positives cause false negatives?
Teams stop trusting failures, start ignoring them, and real bugs get dismissed as "probably flaky."
