Takeaways:
- Pete Hodgson's taxonomy on martinfowler.com identifies four types of feature flags: release, experiment, ops, and permission. Each type has different testing requirements, lifecycle expectations, and risk profiles.
- Martin Fowler's practical advice on combinatorial explosion: you don't need to test every flag combination. For release flags, test two states: all flags on as expected for next release, and all flags off (fallback). That covers 90%+ of real-world risk.
- Mobile apps add problems that web apps don't have with flags: SDK fetches flag values asynchronously on launch, cached flag values can persist across app updates, and a flag that evaluates before UI renders can flash wrong state for a split second.
Feature flag testing verifies that your app behaves correctly in every flag state your users will encounter, and degrades gracefully when flag service is unavailable.
Why feature flags make mobile testing harder than web testing
On web, a feature flag evaluation is fast. The page loads, flag SDK initializes, flag value is returned, and UI renders accordingly. If flag service is down, page can fall back to defaults on same request.
Mobile apps don't have that luxury. The complications stack up:
- Async SDK initialization. The flag SDK fetches values from server on app launch. If network is slow or app cold-starts from a killed state, flag values may not be ready when first screen renders. The app has to decide: show a loading state, use cached values, or use hardcoded defaults. Each choice has different testing implications.
- Cached flag values between updates. When a user updates app from version 3.1 to 3.2, locally cached flag values from 3.1 may persist. If 3.2 introduces a new flag or changes behavior behind an existing flag, cached value can put app in an unexpected state.
- Flag evaluation during backgrounding. A user opens app, gets flag state A, backgrounds app for an hour, and comes back. The flag service may have changed flag to state B in meantime. Does app re-evaluate flags on foreground, or does it use stale value?
- No instant updates. Web flags take effect on next page load. Mobile flags might not take effect until next app launch, next session, or next SDK sync interval. QA needs to know refresh behavior to test correctly.
- App Store review timing. If a flag is turned on while app is in App Store review, Apple's testers may see new feature before it's ready. Flag targeting rules need to exclude review environments or internal test accounts.
These aren't theoretical problems. Teams using feature flag management tools like LaunchDarkly, Firebase Remote Config, Statsig, or Unleash encounter them regularly on mobile.
Four flag types and what each means for QA
Pete Hodgson's feature toggle taxonomy on martinfowler.com categorizes flags into four types. Each one creates different testing demands.
Release flags
Purpose: hide incomplete or untested code behind a flag so it can be deployed to production without being visible to users. Turned off by default. Turned on when feature is ready.
What QA tests:
- Flag OFF: feature is completely hidden. No UI element, no API call, no side effect.
- Flag ON: feature works as specified.
- Flag OFF → ON transition: turning flag on mid-session doesn't break current user flow.
- Flag cleanup: after feature is stable, flag code is removed. Regression test that feature still works without flag wrapping it.
Lifecycle: short. Days to weeks. If a release flag lives longer than a sprint or two, it's becoming technical debt.
Experiment flags
Purpose: A/B testing. Users are bucketed into cohorts, and each cohort sees a different variant. The flag SDK handles bucketing.
What QA tests:
- Each variant renders correctly and functions as expected.
- The bucketing is deterministic: same user gets same variant across sessions (unless experiment is reconfigured).
- Analytics events fire correctly for each variant so experiment data is valid.
- Edge case: what happens when a user is in variant A, updates app, and experiment has ended? Do they see winning variant or default?
Lifecycle: medium. Weeks to months while experiment collects data.
Ops flags
Purpose: circuit breakers and kill switches. If a feature is causing performance problems or errors in production, ops can turn it off without deploying a new build.
What QA tests:
- Flag ON: feature works normally.
- Flag OFF: feature degrades gracefully. No crash, no error screen, no broken navigation. The feature simply isn't there.
- Rapid toggle: turning flag on and off repeatedly doesn't corrupt state or cause race conditions.
Lifecycle: permanent or long-lived. These flags stay in codebase as operational controls.
Permission flags
Purpose: gate features based on user attributes (subscription tier, geography, account age). Premium users see feature X; free users don't.
What QA tests:
- Each user segment sees correct set of features.
- Upgrading from free to premium mid-session enables gated feature without requiring a restart.
- Downgrading or expiring a subscription disables feature correctly.
- No UI artifacts from gated feature leak into free experience (empty menus, broken navigation links, placeholder text).
Lifecycle: permanent. These are product features, not temporary toggles.
Which flag combinations to test (and which to skip)
The combinatorial explosion is real. Ten binary flags create 1,024 possible states. Nobody tests all of them. Martin Fowler's practical advice: focus on combinations that will actually exist in production.
For most teams, three combinations cover risk:
- Current production state. Whatever flags are currently active for real users. This is your baseline regression testing configuration.
- Next release state. The flag configuration planned for upcoming release. All new flags turned on, with everything else matching production.
- Fallback state. All flags off. This tests default behavior when flag service is unreachable or returns errors.
Beyond these three, test specific flag interactions only when two flags affect same screen, same user flow, or same backend service. If flag A controls checkout UI and flag B controls search algorithm, they don't interact. Testing them independently is sufficient. If flag A controls checkout UI and flag C controls a new payment method in checkout, those two interact and need combined testing.
A simple rule: if two flags can both be active and both modify same screen or flow, test combination. If they live in different parts of app, test them independently.
Mobile-specific feature flag testing checklist
Beyond flag state testing above, mobile apps need testing for timing and delivery issues unique to mobile:
Cold start with no cached flags:
- Kill app, clear app data, launch on a slow network.
- Does app show a reasonable default state while flags load?
- Does UI update correctly once flag values arrive, without visual glitches?
Stale cache after app update:
- Install version N with flags in state A.
- Update to version N+1 which expects flags in state B.
- Does app handle stale cache gracefully, or does it show broken UI until SDK refreshes?
Flag service outage:
- Disable network connectivity after app is running.
- Does app continue to function with last known flag values?
- If app is launched with no network and no cache, does it fall back to hardcoded defaults?
Flag change during backgrounding:
- Launch app, verify flag state A.
- Background app, change flag remotely.
- Foreground app. Does app reflect new flag state, or does it require a restart?
Flag targeting in test environments:
- Confirm that internal test accounts see flags they're supposed to.
- Confirm that App Store review accounts are excluded from unreleased features.
- Confirm that staging and production flag configurations don't leak into each other.
This checklist sits alongside your regular smoke testing in CI. Run smoke suite once with production flags and once with next-release flag configuration.
Tools and platforms for flag management
The feature flag ecosystem has matured. These are platforms QA teams most commonly encounter:
For QA teams evaluating feature flag tools, mobile SDK quality matters as much as dashboard features. Check for:
- Offline support (does SDK work without network?)
- Cache management (how are stale values handled?)
- Evaluation latency (how long until flags are available after launch?)
- Targeting precision (can you target by device model, OS version, app version?)
How Drizz tests feature-flagged mobile apps
Drizz doesn't evaluate feature flags directly. It tests what user sees on screen. That makes it well-suited for feature flag testing because same test can validate both flag states by running it against different flag configurations.
A test for a release flag that controls a new checkout experience:
Tap on "Add to Cart"
Tap on "Checkout"
Validate "Express Checkout" is visible
Type "4111111111111111" in "Card Number"
Tap on "Pay Now"
Validate "Order Confirmed" is visible
Run this test twice: once with flag ON (should see "Express Checkout") and once with flag OFF (should not). Drizz's Vision AI validates what's actually on screen, so it catches case where a flag-off state still leaks a UI element that should be hidden.
For fallback scenario (flag service down):
Launch app
Validate home screen loads within 5 seconds
Tap on "Search"
Validate search results are visible
Tap on first result
Validate product details are visible
This end-to-end smoke test runs with flag service disabled or unreachable. If app hangs on launch because flag SDK is blocking main thread, this test fails immediately and surfaces problem before users encounter it.
Drizz Cloud runs tests in parallel across device configurations. For feature flag testing, that means running same suite against multiple flag states simultaneously:
- Suite 1: production flag configuration on Pixel 8 (Android 15)
- Suite 2: next-release flag configuration on Samsung Galaxy S23 (Android 14)
- Suite 3: all flags off (fallback) on iPhone 15 (iOS 18)
Three parallel runs, three flag configurations, three device profiles. Results in minutes, not hours. That's practical answer to combinatorial problem: don't test 1,024 states. Test three that matter, on devices your users actually have.
FAQ
What is feature flag testing?
Verifying that an app behaves correctly in each flag state users will encounter, including flag-off fallback and transitions between states.
Do you need to test every flag combination?
No. Test production state, next-release state, and all-off fallback. Only test specific interactions when two flags affect same flow.
What mobile-specific problems do feature flags cause?
Async SDK loading, stale cached values after updates, flag changes during backgrounding, and UI flicker before flag evaluation completes.
What are best feature flag management tools?
LaunchDarkly, Firebase Remote Config, Statsig, Unleash, Flagsmith, and GrowthBook. Each offers mobile SDKs with varying offline support.
How do feature flags create technical debt?
FlagShark research found 73% of flags are never removed. Stale flags increase testing burden and code complexity without providing value.
Should QA test flag cleanup after a feature launches?
Yes. Removing flag code requires regression testing to confirm feature still works without toggle wrapping it.
