A/B test significance calculator
you ran the test and B is ahead. this tells you whether that lead means anything. it is the same two proportion test the paid tools use, without the signup.
this calculator needs javascript. the explanation below works without it.
What confidence actually means
95% confidence does not mean B is 95% likely to be better. it means that if the two variants were truly identical, you would see a gap this large or larger only 5% of the time.
that distinction matters because it explains why running many tests produces occasional false winners. test twenty things that do nothing and about one will look significant anyway.
Before you trust the verdict
check that you set the sample size in advance. if you have been watching daily and stopped when it went green, the confidence number above is optimistic, because you gave yourself many chances to catch a random high point.
also check the test ran for whole weeks. a test covering one weekend and not the next is comparing different populations, not different variants.
Numbers tell you what. They do not tell you what to do.
bring your real numbers to the breakdown call and leave with a plan for them. free, no pitch deck.
Common questions
What if I get 93% confidence?
that is not a winner, it is an unfinished test. keep it running to the sample size you planned. if you have already hit that and it is still at 93%, the difference is probably too small to matter commercially anyway.
Is 95% always the right bar?
it is a convention rather than a law. for a cheap reversible change, 90% is a reasonable risk. for something expensive to build or hard to undo, wait for more. the cost of being wrong should set the bar.
Other execute tools
Ad funnel calculator
turn CPM, click rate and install rate into real cost per install, and see which step of the funnel is costing you money.
ExecuteAd budget split planner
split one budget across TikTok, Meta and Reddit, forecast installs per channel and see your blended cost per install.
ExecuteChannel CAC comparison
compare cost per user across three channels and see how many extra users you would get by moving budget to the best one.
ExecuteCreative testing planner
work out how many creatives your budget can actually test, and whether each one gets enough data to trust the result.
Bring one real growth problem. Leave with the fix.
the breakdown call is free. no pitch deck, no agency handoff.