The High Stakes of Test Selection
Real companies have lost millions shipping features that won their A/B tests.
In documented case studies, product teams rolled out variants that looked statistically significant during testing, then underperformed the original by 5 to 10 percent after full rollout. The culprit wasn't which testing framework they chose. It was skipping the fundamentals.
This happens far more often than it should. Teams get excited about Bayesian versus frequentist debates while their experiment pipelines leak value through preventable errors: underpowered tests, peeking at early results, and running experiments too short to distinguish real effects from novelty bounces.
The philosophical framework you choose matters far less than executing the basics correctly. Bayesian methods offer real advantages, and I'll show you what they are and where they help. But they're an optimization layer sitting atop solid experimental foundations, not a replacement for those foundations.
Here's a systematic workflow. You'll learn how to match your test to your metric type, calculate the sample size you actually need, guard against the validity threats that destroy experiments from the inside, and apply Bayesian optimization only after you've got the fundamentals locked down.
But before we get to Bayesian elegance or advanced corrections, we need to nail the first decision. What are you actually measuring?
Your metric type determines everything that follows. Binary outcomes, those simple yes-no conversions, call for a chi-squared test or a Z-test for two proportions. These two tests answer the same question; they're mathematically equivalent for comparing conversion rates. The critical error to avoid is using a t-test on conversion data. T-tests compare means across continuous distributions; conversion rates are proportions, and applying t-tests violates their underlying assumptions in ways that silently corrupt your results.
Continuous metrics like revenue per user, session duration, or pages per visit require a different approach. Here you need a t-test or Welch's variant when comparing group means. These tests perform reliably once your sample exceeds about 30 per group, thanks to the Central Limit Theorem. The sampling distribution of the mean approaches normal regardless of the underlying distribution, which is why these tests remain robust even when your raw data looks messy.
Here's a quick heuristic to lock this in: two possible values for your metric means chi-squared or Z-test; many possible values means t-test, or Mann-Whitney U if the data looks skewed.
| Metric Type | Recommended Test | Common Mistake |
|---|---|---|
| Conversion (yes/no) | Chi-squared or Z-test | Using t-test |
| Revenue, duration, counts | T-test | Ignoring skewness |
| Skewed or ordinal data | Mann-Whitney U | Pretending it's normal |
| Small samples (fewer than 5 per cell) | Fisher's Exact | Chi-squared with sparse cells |
For chi-squared tests, ensure your expected cell counts exceed 5; when they don't, Fisher's Exact Test handles the small-sample case correctly. Once you've identified the right test for your metric, the next question becomes how many users you need to make that test meaningful.
The Four Parameters That Determine Sample Size
Calculating sample size isn't optional math you can eyeball. Underpowered tests, those run with too few observations, are actively harmful. They produce wide confidence intervals that make it nearly impossible to detect real effects while simultaneously increasing your chance of detecting spurious ones. You're better off not running the experiment at all than running one that misleads you.
Four parameters interlock to determine how many users you need. First, your baseline conversion rate or current mean for continuous metrics. Second, your Minimum Detectable Effect, or MDE, which is the smallest improvement worth detecting. Third, your significance level alpha, typically 0.05, which controls your false positive rate. Fourth, your statistical power, typically 80 to 90 percent, which represents your probability of detecting a real effect of size MDE.
The trade-off between MDE and sample size is exponential. Smaller effects demand dramatically larger samples. This means your MDE shouldn't be "what can we detect?" It should be "what effect size actually matters for our business?"
Let me walk through a concrete calculation. Say your baseline conversion sits at 5 percent, and you want to detect a 1-percentage-point improvement to 6 percent. With alpha at 0.05 and power at 0.80, you'd need roughly 13,000 users per variant, or 26,000 total. Plug those numbers into any decent calculator; analytics-toolkit.com and statsig.com both offer solid options, and you'll get there in seconds.
Sample size calculations must happen before launch, not after you start wondering if you're close to significance. Decide your MDE based on business impact, not statistical convenience. But knowing how many users you need is only half the battle. The other half is resisting the urge to stop the moment you see promising results.
The Peeking Problem and the Minimum Duration Requirement
You've calculated your sample size, hit go on the experiment, and now it's running. The temptation builds: check the results early. See if you're close to significance. Just one quick look.
This is peeking, and it's endemic in practice despite being fundamentally invalid for frequentist tests.
Here's what happens: every time you check interim results and apply a significance test, you give yourself another chance to observe a false positive. Run one test at the planned endpoint, and your false positive rate sits at 5 percent. Check results weekly for 8 weeks and apply significance tests each time, and your false positive rate balloons to somewhere between 15 and 30 percent. That's a 200 to 300 percent increase in spurious winners.
Those early "winners" that pop up in week 2 or 3? Most are false positives that would resolve to no difference, or actual losers, with continued data collection. Teams that peek ship variants that are actually money-losers, then spend months or quarters undoing the damage.
The solution requires different approaches depending on your framework. These adjust your significance thresholds as you accumulate data, preserving the overall false positive rate. Alternatively, you can commit to a fixed sample size and simply refuse to look until you reach it.
Bayesian methods offer more flexibility for continuous monitoring, but only with proper stopping rules. You need calibrated posteriors using metrics like P(B > A) > 0.95, or expected loss thresholds where you stop if expected loss falls below a defined dollar amount per user. Monitoring Bayesian results without these guardrails is just peeking with different vocabulary.
But looking too early isn't the only danger. Even if you never peek at your results until you hit your sample size, running the test for too short a calendar period can lead you equally astray.
New designs frequently trigger a novelty effect: users interact differently simply because the interface feels fresh. This can generate temporary lifts of 5 to 15 percent in conversion or engagement, which typically fade within one to two weeks as the experience becomes routine. If you stop the test too early, you risk shipping a feature that appears victorious only because of this transient buzz.
A real-world case illustrates the danger.
The antidote is disciplined duration. Most experimentation guides recommend a minimum run time of two to four weeks, with three to four weeks preferred for high-traffic sites. This window captures at least one full business cycle and helps average out day-of-week fluctuations.
A complementary tactic is to segment users by tenure. Compare the behavior of brand-new visitors, who are most prone to novelty, against established users. If the lift lives only in the new-user cohort, you have strong evidence that the effect is ephemeral.
Running longer tests may feel like it slows you down, but rolling back a feature that never delivered lasting value costs far more in engineering time, customer confusion, and missed opportunities. Patience here actually speeds things up in the long run.
Once you've mastered these fundamentals, matching tests to metrics, sizing samples correctly, and running experiments long enough for behavior to stabilize, you might wonder if there's a more elegant way to handle the uncertainty. This is where Bayesian methods enter the picture, offering a different way to think about probability and decision-making.
Bayesian Methods: Benefits, Requirements, and Pitfalls
The intuitive advantage is real. Bayesian outputs answer business questions directly. Instead of explaining what a p-value of 0.03 means, you say "P(B beats A) equals 94 percent" or "the expected loss if we ship B and it's actually worse is $0.12 per user." These statements map directly to decisions your stakeholders understand.
Industry implementations bear this out. Teams at Wix, Netflix, and Microsoft report 23 to 31 percent faster decision velocity after adopting proper Bayesian methods. That's meaningful when your roadmap velocity depends on experiment throughput.
But here's what the Bayesian evangelists don't tell you: these benefits require rigorous implementation, and the failure modes are real.
Prior specification trips up nearly half of practitioners. About 47 percent of teams mis-specify their priors, introducing systematic bias that contaminates results. Uninformative (objective) priors work well when you lack strong domain knowledge; these let the data speak. Informative priors require genuine expertise about your metric's likely behavior; using them without that expertise is guessing dressed up as statistics.
Expected loss thresholds offer a principled stopping rule for Bayesian experiments. You define the maximum loss you're willing to accept per user if you ship a loser, then stop the experiment when the posterior's expected loss falls below that threshold. This is more intuitive than fixed sample sizes, but it requires you to think carefully about your actual business constraints.
What Bayesian methods don't fix: they can't compensate for underpowered tests, novelty effects, or peeking without proper stopping rules. If your experiment is too small to detect meaningful effects, a Bayesian framework won't save you. If you're monitoring results without calibrated stopping rules, you're still peeking. Bayesian is an optimization layer atop rigorous design, not a replacement for it.
Of course, even with the right framework and proper stopping rules, your data might violate the assumptions of standard tests. When your metrics refuse to behave normally, or when you're testing multiple variants at once, you need additional tools to maintain validity.
Your t-tests assume normality. Sometimes that assumption fails.
Highly skewed metrics, revenue per user being the classic offender, violate normality in ways that undermine t-test reliability. When your distribution looks nothing like a bell curve, the Mann-Whitney U test steps in. It compares whether observations from one group tend to be larger than observations from another, without assuming any particular distribution shape. It maintains power and Type I error control for non-normal data. One practical note: when ties in your data exceed 10 percent of observations, apply continuity corrections to keep your error rates calibrated.
A/B/n tests, experiments comparing three, four, or more variants, introduce a different problem. Every additional test you run adds another chance for a false positive. Run a test with 4 variants and alpha at 0.05, and your chance of at least one false positive jumps to roughly 18 percent. You're not controlling your error rate anymore; you're inflating it dramatically.
Multiple comparison corrections bring that error rate back under control. Bonferroni divides alpha by the number of tests; it's conservative and straightforward, but reduces your power significantly. Holm-Bonferroni applies a sequential correction that's uniformly more powerful than Bonferroni while maintaining the same family-wise error rate. Benjamini-Hochberg FDR controls the false discovery rate instead of the family-wise error rate; this is appropriate when you're in exploration mode and can tolerate some false discoveries.
For most product experiments, start with Mann-Whitney for skewed continuous metrics, and use Holm correction for A/B/n designs unless false discoveries are explicitly acceptable for your use case. Bayesian hierarchical models can share information across variants more elegantly, but they still require correction for decision-making; the framework doesn't eliminate the multiple comparisons problem.
Here's a practical workflow to apply starting tomorrow.
Building Your A/B Testing Foundation
The framework is sequential: match your test to your metric type before you calculate anything else. Calculate your sample size using baseline, MDE, alpha, and power; never eyeball it. Guard against peeking by committing to stopping rules before you launch. Run tests long enough for novelty effects to settle, typically 3 to 4 weeks minimum. Apply Bayesian optimization only after these fundamentals are consistent in your process. Handle multiple variants and non-normal data with appropriate corrections as your testing program matures.
The key takeaway is this: Bayesian methods are an advanced optimization for teams already running rigorous frequentist experiments. They're not a shortcut that lets you skip sample size calculations, ignore validity threats, or avoid understanding your data's distribution.
Here's your audit checklist. Before you invest in Bayesian infrastructure or argue about priors, confirm your team consistently selects tests matching metric types, calculates sample sizes before running any experiment, resists the urge to peek at interim results, and runs tests long enough for behavior to stabilize, not just until significance appears.
If you're checking these boxes reliably, Bayesian methods can accelerate your decisions and make results more interpretable. If you're skipping any of them, fix those gaps first. Bayesian engines won't rescue underpowered tests or experiments ruined by peeking.
Audit your current process against these fundamentals. Address the gaps before you invest in new infrastructure. Your shipping decisions, and your company's revenue, depend on getting these basics right.