A/B Test Significance Calculator
Check whether a variant really beat the control: p-value and confidence interval for the lift, a Bayesian chance to win, a traffic split check against the split you planned, and whether the test had enough visitors. Up to four variants, with Holm correction.
Press Calculate.
What this A/B test calculator does differently
Most significance calculators take two rows of visitors and conversions and return a percentage. The number they return is only as good as the test behind it, and the common ways a test goes wrong, an uneven split, stopping early, too few visitors, testing several variants at once, are invisible in that one figure. This calculator runs those checks beside the result.
| Elementor | Neil Patel | Wisepops | This tool | |
|---|---|---|---|---|
| Test named | No | No | Two-proportion z-test | Two-proportion z-test, formulas shown |
| Variants | Two | Two, with an add button | Control plus variants | Control plus up to four |
| Several variants corrected | No | Not explained | Sidak | Holm, Bonferroni or none |
| Interval for the relative lift | No | No | Absolute difference, shown relative | Both, the lift computed on the log ratio |
| Traffic split check | No | No | Against an even split | Against the split you planned, such as 90/10 |
| Stopping early | Not covered | Not covered | Warning in the text | Checks against your planned sample size |
| Sample size | No | No | Observed power | Smallest detectable lift, and visitors needed |
| Bayesian view | No | A probability, method not stated | No | Chance to beat control and to be best |
| Export | None | None | Copy results | CSV, Markdown, share link |
Calculator pages as fetched on 1 November 2026. Anything not shown or described on the page is not counted.
How to read the result
Each variant is compared with the control. The card shows its conversion rate, the relative lift with a confidence interval, the difference in percentage points, the z statistic and the p-value, and a Bayesian chance that it beats the control.
The interval matters more than the verdict. A lift of +12% with an interval of +1% to +24% says the change probably helps, by an amount that could be small. Plan on the low end. An interval that crosses zero means the data cannot tell the variant from the control at the confidence level you chose, which is different from showing the two are the same.
The p-value is the probability of seeing a difference at least this large if the variants really converted at the same rate. It is not the probability that the variant is better. That second figure is what the Bayesian chance to beat control estimates, using a flat prior, so the two are shown side by side rather than mixed up.
The checks, and why each one matters
Sample ratio mismatch
If you split traffic 50/50 and see 10,000 visitors in one group and 9,400 in the other, something other than chance is usually at work: a redirect that loses visitors in one arm, a bot filter applied unevenly, tracking that fires late on the variant. Microsoft’s experimentation team, writing about diagnosing sample ratio mismatch, reported that about 6% of its A/B tests had one, and uses a chi-square test with a threshold of p < 0.0005. This calculator uses the same test and threshold against the split you planned, so a deliberate 90/10 test is not flagged for being uneven.
Stopping early
Checking a running test every day and stopping when it crosses 95% is the most common way to get a false winner. Evan Miller’s How Not To Run an A/B Test showed that checking after every visitor and stopping at the first significant reading pushes a nominal 5% false positive rate to 26.1%. The fix is to fix the sample size before starting and read the result once. Enter that planned size and the calculator tells you whether you have reached it.
Sample size and power
A result that is not significant is only informative if the test could have detected a lift worth having. The calculator reports the smallest relative lift your sample could detect at 80% or 90% power, and how many visitors per group you would need to detect the lift you observed. It does not report “observed power”, which some calculators show: it is a direct function of the p-value and adds nothing to it, a point Hoenig and Heisey made in The American Statistician.
Several variants
Every extra variant is another chance of a false winner. With four variants each tested at 95%, the chance that at least one looks significant by luck alone is close to one in five. The Holm correction controls that error across all the comparisons and rejects at least as often as Bonferroni, so it is the default.
The method
The test is a two-proportion z-test with a pooled standard error, the usual choice for conversion rates. The interval for the difference uses the unpooled standard error, and the interval for the relative lift is built on the log of the ratio of the two rates and transformed back, which keeps it asymmetric the way a ratio should be. The Bayesian figures come from a Beta(1 + conversions, 1 + non-conversions) distribution for each group, sampled 20,000 times with a fixed seed so the same inputs always give the same figures. The normal approximation behind the z-test needs a handful of conversions and non-conversions in every group; with very few conversions, treat every figure as rough.
Everything runs in your browser. Nothing you type is sent anywhere, and the share link keeps the numbers in the part of the address after the #, which browsers do not send to the server.
How to run an A/B test that this calculator can trust
- Choose one primary metricDecide before starting which conversion decides the test. Picking the metric that moved after the fact is another way to find a false winner.
- Fix the sample sizePick the smallest lift worth acting on and work out the visitors per group needed to detect it. Run until every group has that many, ideally across whole weeks.
- Check the splitBefore reading the result, confirm the groups match the split you planned. A mismatch means the comparison is broken.
- Read onceEnter the totals here at the planned end. If you looked earlier, say so when you report it.
- Report the intervalQuote the lift with its interval, not just the verdict, and plan on the low end.