A/B test sample size calculator
work this out before you run the test, not after. most tests that produce no clear answer were doomed on the day they launched, because the traffic was never going to be enough to detect the difference being looked for.
this calculator needs javascript. the explanation below works without it.
Why small lifts are so expensive
the traffic needed rises with the square of the difference you are trying to detect. halving the lift you want to catch roughly quadruples the users you need.
that is why testing button colours is usually a waste at seed stage. the effect is real but tiny, and you do not have the traffic to see it. test things that could plausibly move the rate by a fifth or more.
The assumptions here
this uses 95% confidence and 80% power, which are the standard defaults. in plain terms: a 5% chance of calling a winner that was not real, and a 20% chance of missing a real one.
it also assumes you decide the sample size first and look at the result at the end. peeking daily and stopping when it turns green produces false winners far more often than the 5% suggests.
Numbers tell you what. They do not tell you what to do.
bring your real numbers to the breakdown call and leave with a plan for them. free, no pitch deck.
Common questions
Can I stop early if one side is clearly winning?
not without inflating your error rate substantially. early leads reverse constantly at low sample sizes. if you genuinely need to stop early, that requires a sequential testing method, not just a judgement call.
What if I do not have enough traffic for anything?
then stop A/B testing and use qualitative methods instead. watch session recordings, talk to ten users, look at where people drop off. at low traffic those find bigger problems faster than a test that cannot conclude.
Other execute tools
Ad funnel calculator
turn CPM, click rate and install rate into real cost per install, and see which step of the funnel is costing you money.
ExecuteAd budget split planner
split one budget across TikTok, Meta and Reddit, forecast installs per channel and see your blended cost per install.
ExecuteChannel CAC comparison
compare cost per user across three channels and see how many extra users you would get by moving budget to the best one.
ExecuteCreative testing planner
work out how many creatives your budget can actually test, and whether each one gets enough data to trust the result.
Bring one real growth problem. Leave with the fix.
the breakdown call is free. no pitch deck, no agency handoff.