How long to run an A/B test before you call it
Why checking a test daily is the fastest way to a wrong answer, how to size it before you start, and why creative testing does not follow these rules at all.
the short answer: decide the sample size and the end date before the test starts, then do not look at it in a way that lets you stop early. a test you stop the moment it looks significant is not a test, it is a search for a moment that flatters the thing you already wanted to ship.
Why watching it daily produces wrong answers
if you check a running test repeatedly and stop the first time it crosses the significance line, you will cross that line eventually by chance alone. random noise wanders. wait long enough, look often enough, and it wanders across your threshold.
this is the single most common way teams ship changes that do nothing. the test was real, the maths was real, and the stopping rule quietly invalidated both. the variant that “won” was noise caught at its most flattering moment.
the fix is not discipline about looking. it is deciding the stopping point in advance so that looking cannot change anything.
Size it before you start
the required sample depends on your current conversion rate and the size of the change you care about, and the relationship is brutal: detecting a small lift costs enormously more traffic than detecting a large one.
this is worth knowing before rather than after, because the honest answer for a lot of early stage products is that they do not have the traffic to detect a 5% change this quarter. that is not a failure. it tells you to test bigger swings, which is better advice anyway, since a 5% lift was never going to save the company.
the sample size calculator gives the number before you commit, and the significance calculator tells you whether the result you ended up with means anything.
Run whole weeks, not convenient windows
behaviour differs by day. a test that runs tuesday to friday has measured a working week and nothing else.
run full seven day cycles, and end on the same weekday you started. if a promotion, a press hit or a seasonal spike lands mid test, the cleanest thing is usually to throw the test away rather than reason about which days to exclude, because deciding which days to exclude after seeing the results is the same problem as stopping early.
Creative testing does not work like this
this matters, because most of what i do day to day is creative testing, and applying A/B rules to it produces paralysis.
testing 150+ creatives a month is not a hypothesis test with a control. it is a search for outliers in a population where roughly one in twenty is worth keeping. you are not asking whether variant B beats variant A by a statistically defensible margin. you are asking which two of this week’s batch are obviously alive, killing the rest quickly, and spending the saved budget on more batches.
the decision rule is different too: creative gets killed on clearly bad early numbers long before significance, because the cost of running a dead ad for another week is real and the cost of wrongly killing one is a cheap reshoot. what that system looks like in practice is in 150 creatives a month.
know which of the two you are doing. statistical rigour applied to creative volume is expensive, and creative intuition applied to a pricing page test is how you ship a change that quietly costs you money for a year.
want the testing setup reviewed before the spend scales? the breakdown call is free. Let’s talk Growth.