The Statistical Optimality Trap
The standard statistical advice for A/B testing is unequivocal: allocate users equally to treatment and control. That 50/50 allocation is mathematically optimal. It minimizes the total number of observations you need to detect a given effect and squeezes the most power out of every visitor. Yet in the wild, product teams routinely run 10/90, 20/80, or 70/30 splits. They do it not because they have forgotten the theory, but because they are juggling risk. A pricing experiment could tank revenue. A checkout redesign might frustrate loyal users. A regulatory rollout demands limited exposure until safety signals appear.
That tension is deliberate: teams knowingly trade statistical efficiency for business safety. Teams accept a statistical efficiency penalty in exchange for protecting the business from downside. The real question is not "should we ever deviate from 50/50?" but "how much are we willing to pay for that safety, and do we know the price tag?"
Counting the Cost: From Formula to Traffic
Knowing the price tag requires understanding exactly how the mathematics punishes imbalance. Under the classic difference-in-means framework, the estimator for the treatment effect is
[ \hat{\Delta}= \bar{Y}\text{treat}-\bar{Y}\text{control}. ]
If the two groups share the same variance (\sigma^2) (homoscedasticity) and the total sample size is (N=n_1+n_2), the variance of (\hat{\Delta}) becomes
[ \operatorname{Var}(\hat{\Delta})=\sigma^2!\left(\frac{1}{n_1}+\frac{1}{n_2}\right). ]
Let (k=n_1/n_2) be the allocation ratio (treatment to control). Substituting (n_1 = kN/(1+k)) and (n_2 = N/(1+k)) yields
[ \operatorname{Var}(\hat{\Delta})=\sigma^2\frac{(1+k)^2}{kN}. ]
When (k=1) (the 50/50 split) the variance collapses to (2\sigma^2/N), the smallest possible value for a given (N). The relative efficiency of an unequal design compared with the optimal equal design is therefore
[ E(k)=\frac{4k}{(1+k)^2}. ]
This formula translates directly into real traffic numbers. A 2:1 split ((k=2)) operates at 0.89 efficiency, requiring roughly 12% more observations than a balanced test. Push to 5:1 and you need around 80% more traffic. At 9:1 you are looking at nearly double the sample size. The curve is nonlinear: moving from 50/50 to 60/40 costs little (roughly 5% extra traffic), but pushing to 90/10 almost doubles your requirements. This asymmetry catches many teams off guard. They assume that splitting 70/30 is only slightly worse than 50/50, when in fact they have sacrificed over 16% of their statistical power. If you want a 10% holdout, the test will run three weeks instead of two.
| Allocation (T:C) | (k) | Efficiency (E(k)) | Approx. traffic increase |
|---|---|---|---|
| 2:1 | 2 | 0.89 | +12-14% |
| 70/30 | 2.33 | 0.82 | +22% |
| 90/10 | 9 | 0.36 | +178% |
That means a test that would finish in two weeks with a 50/50 split could stretch to three weeks under a 2:1 design, or creep past a month with a 90/10 holdout. When variances differ between arms ((\sigma_1\neq\sigma_2)), the Neyman allocation becomes optimal: (n_1/n_2 = \sigma_1/\sigma_2). In that scenario an unequal split can actually reduce variance, because the noisier arm receives more observations. This statistical justification is distinct from the business-driven reasons that dominate most production rollouts, but it reinforces that the mathematics always points to a single, well-defined optimum.
A common pitfall complicates this math. Many online calculators and legacy scripts default to equal splits. If you plug your desired power, alpha, and minimum detectable effect into one of those tools without specifying the allocation ratio, you will get a sample size that is too low for your actual skew, leaving the experiment underpowered. The fix is simple. Most modern libraries, such as R's pwr package or Python's statsmodels.stats.power.TTestIndPower, expose a ratio argument that applies the (1/E(k)) correction automatically. Once you have calculated the true cost, the next question becomes whether your specific situation justifies paying it.
Unequal splits are not a statistical sin when they are chosen for a clear, pre-experiment reason and held constant throughout the test. Four patterns appear repeatedly in production.
First, risk mitigation through holdouts. High-stakes changes, such as a new pricing engine or a radical checkout overhaul, are often launched to a tiny fraction of users (10/90 or 20/80). The goal is to bound potential loss while still gathering enough signal to decide whether to proceed.
Second, phased rollouts. Teams start with a conservative split (say 70/30) to watch for anomalies, then gradually ramp to 50/50 once the treatment proves stable. This blast-radius containment lets you catch bugs before they hit the full audience.
Third, heteroscedastic optimization. If early data suggest the treatment variance differs from control, Neyman allocation prescribes a ratio proportional to the standard deviations. For example, a recommendation algorithm that occasionally serves wildly off-target items might merit a larger treatment group to average out that noise.
Fourth, regulatory or compliance mandates. Finance, healthcare, or aviation experiments sometimes face rules that cap exposure to unproven changes until safety thresholds are met. In those domains, the allocation ratio is a legal requirement, not a convenience.
The critical constraint across all cases is that the ratio must be locked before the first visitor arrives and never altered mid-test. Pre-commitment transforms what could look like a mistake into a principled design choice. But how do you distinguish this principled choice from a broken experiment?
The SRM Check: Separating Intention from Error
This is where Sample Ratio Mismatch (SRM) detection becomes crucial. SRM is the statistical alarm that sounds when observed bucket proportions diverge significantly from what you planned. A significant SRM, usually flagged by a chi-square goodness-of-fit test with p-value < 0.01, points to bucketing failures, device-specific bugs, or faulty ramp-up scripts. Ronny Kohavi's trustworthiness criteria list SRM detection as one of five non-negotiable pillars. If an experiment fails this check, any outcome is suspect regardless of the p-value.
The diagnostic rule is straightforward: if you didn't plan it, it's a bug. When you do intend a 10/90 split, your monitoring framework must test against those exact expectations, not the default 50/50. Building dashboards that display both the planned ratio and the observed ratio, with a clear pass/fail flag based on the appropriate chi-square test, prevents alert fatigue and keeps teams from chasing false positives.
Ignoring SRM can masquerade as "unequal allocation," leading teams to celebrate a win that is actually an artifact of broken randomization. Conversely, over-alerting on intentional skew erodes trust in the monitoring system itself. The solution is to encode the planned probabilities directly into your SRM checks and to document the rationale prominently in the experiment spec. Once you have secured your bucketing integrity, you might wonder if fixed ratios are even the right tool for the job.
Multi-Armed Bandits as an Alternative Philosophy
Fixed unequal allocation is not the only approach to managing the trade-off between exploration and exploitation. Multi-Armed Bandit (MAB) algorithms offer a fundamentally different mechanism.
The distinction is philosophical. A fixed 80/20 allocation commits you upfront to exposing 80% of traffic to the treatment, regardless of early results. A bandit algorithm continuously shifts traffic toward whichever variant appears to be winning, based on accumulated data. The goal is not final inference precision; it is cumulative regret minimization, reducing the total exposure of users to underperforming variants during the experiment itself.
The statistical trade-off is real. MABs sacrifice the clean, fixed-horizon inference that makes traditional A/B tests defensible. You can run a bandit and emerge with a winner, but your p-values and confidence intervals will be biased because the traffic allocation itself influenced which data you collected. Traditional statistical inference assumes fixed assignments; bandits violate that assumption by design.
Choose fixed allocation when your organizational standards or regulatory context demand high-confidence inference, when you need defensible evidence for consequential decisions, or when the cost of reaching the wrong conclusion exceeds the opportunity cost of extended testing. Choose bandits when you are optimizing for exploratory features where rapid loss reduction outweighs final significance testing, when speed matters more than precision.
Hybrid approaches exist and deserve consideration. An explore-then-exploit schedule might start with a conservative 20/80 fixed allocation to gather initial signal, then transition to a bandit after you have established that no variant is catastrophically worse than the other. This combines the safety of limited initial exposure with the efficiency gains of dynamic allocation once you are in stable territory. The transition point itself requires discipline. Set the criteria upfront, perhaps after reaching a minimum sample size or after a fixed number of days, to avoid cherry-picking the transition moment.
Whatever approach you choose, MABs still require SRM checks. Bucketing failures and configuration errors do not become acceptable just because you have switched algorithms. Bandits also introduce their own risks. Time-varying confounders can mislead early winners, making aggressive exploitation dangerous in volatile environments. With these nuances in mind, the mechanics of implementation remain similar regardless of your philosophical choice.
Here’s how to move from theory to a test that runs in production. Start by writing the allocation ratio into the experiment specification, alongside a brief business justification (risk mitigation, phased rollout, heteroscedasticity note, or regulatory requirement).
Next, run your power analysis using a library that accepts an allocation ratio. In R, use pwr.t2n.test(n1 = NULL, n2 = NULL, d = effect.size, sig.level = 0.05, power = 0.8, ratio = k). In Python, call TTestIndPower().solve_power(effect_size, nobs1=None, ratio=k, alpha=0.05, power=0.8). The solver will return the required n1 and n2; multiply by `(1+k)) to get total traffic.
Then plan your duration. Divide the inflated total sample by your expected daily active users to estimate test length. Communicate the timeline uplift to stakeholders early. A 70/30 split will add roughly three days to a two-week test.
Lock the ratio in your feature-flag or experimentation platform. Avoid "approximately" settings; most systems let you enter exact probabilities or weights. Document the configuration in your experiment ticket so that future analysts understand why the groups are imbalanced. This documentation serves as another guardrail against the SRM confusion that can plague poorly documented tests.
Build an SRM test that compares observed counts to the expected probabilities (not 0.5/0.5). A simple chi-square with chisq.test(observed, p = expected) does the job. Alert only when the test fails; otherwise, treat the skew as intentional.
After the test ends, compare the actual group variances to the assumptions used in the design. If heteroscedasticity was misestimated, record the observed (\sigma_T/\sigma_C) ratio to inform future Neyman allocations. With these steps complete, you can launch with confidence.
Conclusion: Making the Trade-Off Deliberately
Unequal allocation is not a statistical error; it is a deliberate lever for managing business risk. The real dangers lie in ignoring the efficiency penalty (running underpowered tests because you forgot to inflate the sample size) or confusing a planned skew with an SRM bug that invalidates your inference.
When you weigh a decision, ask: What am I buying with the extra traffic? Is it protection against a costly pricing mistake, compliance with a regulator, or a cleaner estimate under heterogeneous variance? Know the cost from the formula (E(k)=4k/(1+k)^2), pay it consciously by adjusting your power calculations, document the rationale, and monitor for the right signals. That’s how a theoretical trade-off becomes a repeatable practice.