A/B Test Significance Calculator

You ran an A/B test — now check whether the result is statistically significant. Enter the visitors and conversions of your control (A) and your variation (B), pick a confidence level, and get the conversion rate, uplift, significance and p-value instantly.

Control Visitors

Control Conversions

Variation Visitors

Variation Conversions

Confidence Level

Warning: a possible Sample Ratio Mismatch (SRM) was detected. Your control and variation traffic differ by more than 10% — double-check your baseline and variation split before trusting the result.

Enter your visitors and conversions above, then click Calculate to see your result and the distribution chart.

What the calculator tells you

Three answers in one click — no spreadsheet, no statistics degree required.

Conversion rate & uplift

See the conversion rate of both versions side by side and the relative uplift of your variation over the control.

Statistical significance

A chi-square test tells you whether the difference is real or just noise, at the confidence level you choose (90%, 95% or 99%).

Sample Ratio Mismatch check

An automatic SRM check flags when your traffic split is off — a common cause of misleading test results.

Understanding A/B test significance

When is my A/B test result significant?

A result is statistically significant when the difference between your control and variation is unlikely to be caused by chance. You define how strict that bar is with the confidence level: at 95%, the result counts as significant once the calculator reaches at least 95% confidence.

Worked example: the control (A) has 85,000 visitors and 3,326 conversions — a 3.91% conversion rate. The variation (B) has 83,000 visitors and 3,403 conversions — a 4.10% conversion rate, a 4.78% uplift. The result exceeds the 95% threshold (about 97.5% confidence), so this test is significant.

What is the p-value?

The p-value is the probability of seeing a difference at least this large if the variation had no real effect. A smaller p-value means stronger evidence. At a 95% confidence level you look for a p-value of 0.05 or lower. This calculator reports a one-sided p-value — it tests whether the variation is better than the control in the observed direction.

Which confidence level should I choose?

95% is the common default in conversion optimization — a good balance between catching real winners and avoiding false positives. Choose 99% when a wrong decision would be expensive and you want to be extra sure; 90% only for early, low-stakes reads where you accept more uncertainty.

What is a Sample Ratio Mismatch (SRM)?

SRM happens when visitors are not split between control and variation the way you intended (for example, a planned 50/50 split that ends up 55/45). It usually points to a technical problem — redirect issues, bots, or tracking gaps — and it can quietly invalidate your result. If the calculator flags SRM, fix the split before trusting any significance number.

Is a significant result enough to ship the variation?

Significance tells you the difference is probably real, not whether it is meaningful for your business or stable over time. Also consider the size of the uplift, the test duration (cover full business cycles), sample size, and whether the result holds across key segments before you roll a variation out to everyone.

Disclaimer

Varify accepts no responsibility for the functionality of the code or the information provided. It is the user's responsibility to test their implementation thoroughly and implement it correctly.