The Inverse MDE Problem

The Real Problem with Fixed Traffic

Every A/B testing textbook tells you the same story. First, you define your Minimum Detectable Effect, the smallest improvement worth shipping. Then, you calculate how many visitors you need to detect it reliably. Finally, you run the test and wait.

But this workflow assumes you have unlimited traffic, which most teams don't.

For most product teams, reality looks nothing like this. You know exactly how many visitors hit your product tomorrow. You know your daily active users, your monthly uniques, your traffic budget for the quarter. The sample size isn't a variable you control; it's a constraint you inherit.

This mismatch often leads to wasted effort. When you work backwards from a fixed traffic pool, you often discover your test can only detect effects so large they'd represent game-changing wins, not incremental improvements. Running such an experiment guarantees one outcome: weeks of waiting followed by inconclusive results. You'll declare the test "not significant" and move on, never knowing whether a real 5% improvement existed but fell below your detection threshold.

Instead of asking how many people you need, ask: given your actual traffic, what effect size can you detect, and does that matter for your business?s?**

This shift turns experiment design into a practical triage tool. Let's look at how the standard formula works in reverse.

The Mathematics of Precision

The classic sample size formula for a two-proportion Z-test looks like this:

n = (z₁₋α/₂ + z₁₋β)² × 2p̄(1 − p̄) / MDE²

Here, n is the required sample size per variant, z values represent critical values for your significance level (α) and statistical power, p̄ is the pooled baseline proportion, and MDE is your target minimum detectable effect. This answers the textbook question: given a desired MDE, find the required traffic.

Now rearrange it to solve for MDE instead:

MDE ≈ (z₁₋α/₂ + z₁₋β) × √(2p̄(1 − p̄) / n)

This version answers the real-world question: given fixed traffic, what effect size can I detect?

Notice the quadratic relationship embedded in these formulas. Sample size doesn't scale linearly with precision; it scales with the square of the inverse. The notation n ∝ 1/MDE² tells the story: cutting your detectable effect in half requires multiplying your sample size by four. Halve it again, and you need sixteen times the original traffic.

This quadratic relationship makes precision costly. In low-traffic environments, your detection threshold rises steeply. A team with 10,000 monthly visitors might detect a 10% relative lift reliably. Detecting a 5% lift requires 40,000 visitors. A 2.5% lift? You'd need 160,000. This relationship underpins the calculations below.

For conversion rates specifically, the achievable MDE simplifies to a more workable formula:

[ \text{MDE} \approx (z{1-\alpha/2} + z{1-\beta}) \cdot \sqrt{2 \cdot \bar{p} \cdot (1-\bar{p}) / n} ]

• (z_{1-\alpha/2}): critical value for the chosen (\alpha) (1.96 for 0.05).
• (z_{1-\beta}): critical value for the desired power (0.84 for 80%, 1.28 for 90%).
• (\bar{p}): pooled proportion ((p_1 + p_2)/2).
• (n): sample size per variant (assuming equal split).

Standard practice uses (\alpha = 0.05) and power = 0.80 unless the decision is high stakes, in which case many teams bump power to 0.90.

Concrete example: 1,000 visitors per variant, baseline conversion (p_0 = 15\%). Assuming a small effect, we start with (\bar{p} \approx p_0 = 0.15). Plugging the numbers:

[ \text{MDE} \approx (1.96 + 0.84) \sqrt{\frac{2 \times 0.15 \times 0.85}{1000}} \approx 2.80 \times \sqrt{0.000255} \approx 2.80 \times 0.01597 \approx 0.0447 \; (\text{or } 4.5\% \text{ absolute lift}) ]

Rounded, the achievable MDE is about 4.2-4.5%.

However, there's a nuance in the calculation. That pooled proportion (\bar{p}) technically depends on the unknown treatment proportion, which itself depends on the MDE. Because MDE appears on both sides, the formula cannot be solved algebraically. You can use a baseline approximation for modest lifts or a fixed-point iteration for more precision. Start with (\bar p^{(0)} = p_{0}), compute an MDE, update (\bar p) with the average of the implied control and treatment rates, and repeat until the change in MDE is negligible. In practice, this usually converges in two or three iterations.

import math

def achievable_mde(n, p0, alpha=0.05, power=0.80, max_iter=10, tol=1e-6):
    z_alpha = 1.96                     # two-tailed 0.05
    z_beta  = 0.84 if power == 0.80 else 1.28
    z_sum   = z_alpha + z_beta

p_bar = p0                         # initial guess
    for _ in range(max_iter):
        mde = z_sum * math.sqrt(2 * p_bar * (1 - p_bar) / n)
        new_p_bar = (p0 + (p0 + mde)) / 2   # assume a positive lift
        if abs(mde - z_sum * math.sqrt(2 * new_p_bar * (1 - new_p_bar) / n)) < tol:
            break
        p_bar = new_p_bar
    return mde

print(achievable_mde(n=1000, p0=0.15))

A spreadsheet version replaces the loop with a few rows: start with the baseline, compute MDE, recalc (\bar p), and copy the formula down until the cell values converge.

This same principle extends beyond binary outcomes. When your metric is continuous, like revenue per user or session duration, the logic holds but variance replaces the proportion term. The two-sample t-test gives:

[ \text{MDE} \approx (z{1-\alpha/2} + z{1-\beta}) \cdot \sqrt{\frac{2\sigma^2}{n}} ]

where (\sigma^2) is the (assumed equal) variance of the metric in each grouFirst, get a realistic estimate of σ².(\sigma^2). Use pre-experiment data: compute the standard deviation of the metric over a recent window, or run a short pilot to estimate variability.

If you suspect unequal variances, common with revenue where a few high spenders inflate the treatment spread, switch to Welch's correction. The denominator becomes (\sqrt{\sigma_1^2/n_1 + \sigma_2^2/n_2}) and the degrees of freedom adjust accordingly. Most statistical packages handle this automatically when you request a Welch t-test.

If you're unsure about variance, inflate your estimate to avoid overly optimistic MDE projections.

Example: 800 users per arm, observed standard deviation (\sigma = \$5), (\alpha=0.05), power=0.80.

[ \text{MDE} \approx (1.96 + 0.84) \sqrt{\frac{2 \times 5^2}{800}} \approx 2.80 \times \sqrt{0.0625} \approx 2.80 \times 0.25 \approx \$0.70 ]

If you anticipate unequal variances and use Welch's formula with (\sigma_1=4, \sigma_2=6), the MDE rises to roughly \$0.85. Your variance estimate directly affects how trustworthy the inverse MDE calculation is. After you've calculated your threshold, you need to decide what to do with it. Is this detectable effect actually worth finding?

The Triage Decision

You now have the smallest effect your traffic can detect.nd power. Compare that to your minimum business impact threshold, the effect size that would justify the engineering, design, and opportunity costs of launching a change. If the achievable MDE is larger than that threshold, the test will likely be inconclusive, wasting traffic and time.

Consider a SaaS team that requires at least a 1% absolute lift in monthly recurring revenue to cover the cost of a new feature experiment. Their current campaign allocates 500 users per variant, and the inverse MDE calculation (based on historical revenue variance) shows they can only reliably detect a \$3 lift, far above the 1% threshold. This often leads to high Type II error rates and peeking bias as analysts extend the test hoping for significance. Underpowered tests just produce noise, not useful insights.

compute your achievable MDE, define your business threshold, and only run the test if the MDE is at or below that threshold.red.

If the MDE is only slightly above your threshold, you can adjust α or power, but be aware of the trade-offs.ications. This triage step makes experimentation more efficient and protects your resources. If the calculation says no and you can't increase traffic, there are still alternatives.

Alternatives for Traffic-Constrained Environments

When your traffic is too low, these methods can reduce the sample size needed without sacrificing statistical rigor.

Approach	How it helps	Key trade-off
Sequential testing (Wald SPRT, O'Brien-Fleming)	Allows early stopping when evidence is strong, reducing average sample size for large effects	Requires pre-planned boundaries; implementation complexity
Bayesian ROPE (Region of Practical Equivalence)	Directly asks "is the effect larger than the business-relevant threshold?" and reports posterior probability	Needs a prior for the effect size; results are probabilistic, not p-values
Multi-armed bandits	Dynamically reallocates traffic toward promising variants, cutting wasted exposure	Provides regret-minimization rather than a fixed-sample hypothesis test; inference about the exact effect size is indirect

Getting the implementation right takes some thought. The gsDesign package in R offers exact binomial sequential designs with O'Brien-Fleming boundaries. For Bayesian approaches, bayesAB in Python or PyMC allow you to define a ROPE and obtain the posterior probability that the lift exceeds it. For bandits, the MAB package or Facebook's Vowpal Wabbit provide production-ready algorithms.

These tools don't create traffic; they just help you ask a better question: given what you have, how much can you actually learn? Properly set priors or stopping rules are essential to avoid over-optimistic conclusions. That's the real constraint behind any experiment.

Know Your Resolution Limit

When traffic is fixed, the usual A/B testing math gets turned around. Instead of deriving sample size from a chosen MDE, you derive achievable MDE from your traffic constraints, then compare that threshold to what actually matters to your business.

Because of the quadratic relationship, most teams will only reliably detect effects between 2% and 10% unless they have a lot of traffic. A 10x improvement in detection precision requires 100x the traffic. In low-traffic settings, this resolution limit is a hard reality, not a choice.

Adding inverse MDE calculations to your testing process saves time and helps you make faster decisions. Before launching your next test, calculate the achievable MDE. If it exceeds your minimum meaningful impact, abort or redesign before spending traffic you won't recover.

Sample size calculations aren't just a box to check after the fact. Use them upfront to protect your resources.