Growth marketer operator · San Jose, CA | ZuAI: 10K → 2M users at $0.02 CAC | $300k/mo ad spend managed
Ar.Bhavesh Pro
free tool · execute

A/B test sample size calculator

work this out before you run the test, not after. most tests that produce no clear answer were doomed on the day they launched, because the traffic was never going to be enough to detect the difference being looked for.

Users needed per variant ...
Days to get there ...
Total users needed ...
Read ...

this calculator needs javascript. the explanation below works without it.

example

Worked example

an illustration with round, made-up numbers, not anyone's real data. they are the calculator's default inputs, so you can follow each step in the form above.

  1. your sign-up page converts 5% of visitors.
  2. the smallest change worth detecting is a 20% relative lift, which means moving 5% to 6%.
  3. about 800 users a day enter the test, split across both variants.
  4. at 95% confidence and 80% power, each variant needs about 8,155 users, so 16,310 in total.
  5. at 800 a day that is about 20 days, so run three full weeks and end on the same weekday you started.

now halve the ambition to a 10% lift, 5% to 5.5%. each variant needs about 31,231 users and the test takes about 78 days: nearly four times the traffic for half the lift. that is why small lifts are rarely worth testing at seed stage traffic.

Why small lifts are so expensive

the traffic needed rises with the square of the difference you are trying to detect. halving the lift you want to catch roughly quadruples the users you need.

that is why testing button colours is usually a waste at seed stage. the effect is real but tiny, and you do not have the traffic to see it. test things that could plausibly move the rate by a fifth or more.

The assumptions here

this uses 95% confidence and 80% power, which are the standard defaults. in plain terms: a 5% chance of calling a winner that was not real, and a 20% chance of missing a real one.

it also assumes you decide the sample size first and look at the result at the end. peeking daily and stopping when it turns green produces false winners far more often than the 5% suggests.

Numbers tell you what. They do not tell you what to do.

bring your real numbers to the breakdown call and leave with a plan for them. free, no pitch deck.

Let's talk Growth
FAQ

Common questions

Can I stop early if one side is clearly winning?

not without inflating your error rate substantially. early leads reverse constantly at low sample sizes. if you genuinely need to stop early, that requires a sequential testing method, not just a judgement call.

What if I do not have enough traffic for anything?

then stop A/B testing and use qualitative methods instead. watch session recordings, talk to ten users, look at where people drop off. at low traffic those find bigger problems faster than a test that cannot conclude.

How many users do I need for an A/B test?

at a 5% conversion rate, detecting a 20% relative lift needs about 8,155 users per variant at 95% confidence and 80% power. the exact number depends on your current rate and the smallest lift you care about, and detecting a 10% lift at the same rate needs about 31,231 per variant.

How long should an A/B test run?

until each variant reaches the sample size you set before launch, rounded up to whole weeks. behaviour differs by day, so run full seven day cycles, end on the same weekday you started, and fix the end date in advance so looking at the results cannot change it.

What do 95% confidence and 80% power mean?

95% confidence means a 5% chance of calling a winner that was not real, and 80% power means a 20% chance of missing a real one. they are the standard defaults, and this calculator uses both.

Why does a smaller lift need so many more users?

because the users you need rise with the square of the difference you are trying to detect, so halving the lift roughly quadruples the sample. that is why testing small tweaks such as button colours rarely concludes on seed stage traffic.

next

Other execute tools

Bring one real growth problem. Leave with the fix.

the breakdown call is free. no pitch deck, no agency handoff.

Let's talk Growth all 20 tools