An A/B test is won or lost before the first visitor is split; everything after launch is either the execution of a plan or the manufacture of an anecdote.
India should be the best market in the world for experimentation. Traffic volumes that European brands would envy, digital-first categories, and product teams — in fintech, quick commerce, direct-to-consumer — that ship weekly. Yet the testing culture in most Indian organisations I encounter is a dashboard culture: launch two variants, watch the live results, and declare a winner the day the significance indicator turns green or the sprint ends, whichever comes first. High traffic makes this worse, not better, because large numbers make noise look authoritative. The result is a steady output of “winning” variants that fail to move the quarterly number, followed by a slow institutional loss of faith in testing itself. The problem was never testing. It was that almost none of these were tests.
The statistics here are old, settled, and blunt. An A/B test is defined by five parameters: statistical significance, statistical power, minimum reliably detectable lift, baseline conversion rate, and number of visitors. These parameters are interlinked — when four are established, the fifth can be calculated. The disciplined workflow fixes the first four before launch and calculates the fifth: the analyst sets the significance level (the tolerance for declaring a false winner), the power (the probability of catching a real winner), the minimum lift worth detecting (a business judgment, not a statistical one), and the known baseline rate — and out comes the required number of visitors, which, divided by daily traffic, is the test’s duration. That number is the contract. The test runs until the contract is fulfilled, and only then are results read.
| Parameter | Who sets it | Nature |
|---|---|---|
| Statistical significance | Analyst, pre-test | Tolerance for a false positive |
| Statistical power | Analyst, pre-test | Probability of detecting a true effect |
| Minimum reliably detectable lift | Business, pre-test | Smallest effect worth acting on |
| Baseline conversion rate | Observed, pre-test | Current performance of the control |
| Number of visitors | Calculated output | Determines duration; not chosen |
Notice what that table implies: the number of visitors is the one parameter the team does not get to choose. Yet in practice it is the only one most teams do choose — implicitly, by stopping when the dashboard looks good. This is the peeking problem, and it is not a minor sin. Checking a live test repeatedly and stopping at the first green reading inflates the false-positive rate far beyond the advertised significance level; run enough short, peeked-at tests and a healthy fraction of your “winners” are coin flips wearing lab coats. Across a decade of deployments, this is the single most common failure I have seen in testing programmes at otherwise sophisticated companies: excellent tooling, correct arithmetic inside the tool, and a stopping rule of “whenever the number looks persuasive.”
The pre-test conversation is also where the business and the statistics meet honestly — the dual lens again. The brand voice proposes the variant and the hypothesis: this creative, this price framing, this onboarding flow should perform better, for an articulable reason. The evidence voice asks the uncomfortable arithmetic question: what is the minimum lift that would justify shipping this, and given our baseline rate, how many visitors does detecting that lift require? Frequently the answer ends the test before it starts — the honest sample size implies a fourteen-week test for a page that sees modest traffic, at which point the correct decision is not to run a doomed test but to test something bigger, on a busier page, or to accept the change as a judgment call and say so out loud. A test that cannot reach its required sample in a tolerable window is not a test; it is a delay with graphs. Killing it at the planning stage is the discipline working, not failing.
There is a genuine turn coming at the end of this argument, and it is worth naming rather than resisting. The run-test-analyse-ship-a-winner workflow is being displaced at the edges by automated, machine-learning-driven personalisation — systems in the mould of Adobe Target’s automated personalisation, where ensemble models decide continuously which content each visitor sees, without a marketer ever declaring a single winner. Where these systems fit, they genuinely remove the wait-for-significance loop for routine content decisions. But they do not repeal the discipline; they relocate it. The questions move up a level: what is the model optimising for, is that metric actually the business outcome or a convenient proxy, what is being held out as a control to prove the system beats a static experience, and who reviews the exploration policy? A team that never internalised power and sample-size thinking will not suddenly acquire judgment because the deciding is now automated — it will simply stop being able to tell whether the machine is earning its keep. The pre-test mathematics was never really about tests. It was about refusing to act on noise, and that refusal transfers intact to the algorithmic era.
On Monday: institute a one-page pre-registration for every experiment — hypothesis, the four chosen parameters, the calculated visitor requirement and end date — written before launch and stored where the results will later be published. Remove mid-test readouts from every screen that a decision-maker sees; the dashboard opens on the pre-registered end date. Ask your optimisation lead for the current quarter’s tests that ended early, and treat each one as unread. And where an automated personalisation system is already running, ask it the same question you would ask an analyst: show me the holdout, and show me the lift against it. If nobody can, you are not experimenting. You are decorating decisions that were already made.