You stare at the dashboard, the blue glow reflecting off your coffee cup. Variant A converted 4% of 50 visitors. Variant B converted 6% of 50 visitors. Your gut says there's a lift, a real improvement that could matter for the business. But the p-value from the chi-square test comes back 0.21, stamped with the dreaded "not significant." The software has spoken, and its verdict is clear: nothing to see here. Move along.
This tension, this gap between what your intuition tells you and what the statistics report, pops up whenever you run experiments on low-traffic pages or features with conversion rates below 1%. Small samples and sparse counts violate the assumptions that underlie the classic Pearson chi-square test, prompting many tools to apply Yates' correction automatically. You didn't ask for it. You might not even know it's happening. But there it is, silently nudging your p-value upward, making your promising result vanish into the fog of statistical noise.
Why does this correction appear in SciPy by default, in textbooks, and in teaching slides? It was invented in the 1930s to bridge a gap between discrete count data and the continuous chi-square distribution when computers were nowhere near as powerful as they are today. The central question for modern practitioners is simple: Is Yates' correction a helpful safeguard against false positives, or an outdated habit that needlessly hides real effects? The answer shapes how often you overlook genuine lifts or chase mirages.
The correction was born from computational limits that no longer exist; today it should serve as a diagnostic check, not a default adjustment. Treating it as a routine step inflates Type II errors, making you miss real improvements, while blindly discarding it can raise Type I risk in very sparse tables. Understanding this trade-off directly shapes how often you ship valuable features or leave money on the table.
The Math and the History: How Yates Fixed a Discrete-Continuous Mismatch
To understand what the correction actually does, you need to see the formula in action. The standard chi-square statistic sums (Observed − Expected)²/Expected across all cells of your table. For a 2×2 contingency table, which is what you have when comparing two A/B variants, you're making four comparisons between what you observed and what you'd expect if the null hypothesis were true. The math is straightforward. The interpretation is not.
Yates' modified formula subtracts 0.5 from each absolute difference before squaring: Σ(|O − E| − 0.5)²/E. That "half-point" adjustment pushes the test statistic downward. A smaller statistic means a larger p-value. A larger p-value means you're less likely to declare significance. The effect is subtle but decisive, especially when your sample sizes are small. In the example from your dashboard, that 0.5 subtraction might be the difference between p=0.08 and p=0.21, between investigating further and moving on entirely.
The correction emerged from a specific historical moment. In 1934, English statistician Frank Yates published his method in the Journal of the Royal Statistical Society. His goal was practical: reduce approximation error when using a continuous chi-square distribution to model discrete binomial counts. Before modern computers, researchers relied on published statistical tables and hand calculations. Fisher's exact test existed, but enumerating all possible tables by hand? That was an impractical burden for anyone without a research team and months to spare. Yates was solving a problem of his era, not setting eternal statistical law.
Imagine a statistician in 1935, hunched over a desk with a mechanical calculator, a pad of graph paper, and a single chi-square distribution table printed in a handbook. That researcher couldn't run ten thousand simulations to see how often their method produced false negatives. They couldn't bootstrap confidence intervals or enumerate exact probabilities for a table with more than a handful of observations. They made do with mathematical insight and practical constraints. Yates himself framed his correction as an approximation improvement, not a gold-standard solution. He was trying to bridge the gap between what was computationally feasible and what was statistically desirable. The problem is that approximation improvements from 1934 became institutional defaults by 2024, not because they were proven superior, but because they were already there, baked into textbooks and software that inherited them like family heirlooms.
Why the Correction Fails and What to Use Instead
Cochran's rule warns that the chi-square approximation becomes dubious when any expected cell frequency falls below 1, or when more than 20% of cells have expectations under 5. In typical A/B tests, think a landing page with 200 visitors per variant and a 1% baseline conversion, many cells hover near these thresholds, making the approximation questionable. Because Yates' correction subtracts 0.5 from each |O−E|, it systematically reduces the chi-square statistic. Larger p-values follow, which inflates Type II error, the false negative rate. Empirical work by Grizzle (1967) and Conover (1974) showed that the correction often overcorrects, producing p-values that overstate the true probability under the null. For very small samples, the typical p-value inflation ranges from 0.02 to 0.05, enough to turn a marginally significant result into a non-significant one precisely when you need power the most.
The controversy hinges on the error trade-off you value. Medical trials may prioritize avoiding false positives, Type I errors, whereas digital experiments often favor detecting real lifts quickly, accepting a modest rise in false positives to avoid costly missed opportunities. In the latter context, Yates' conservatism can be a costly liability. When you're testing a new checkout flow that could generate millions in revenue, waiting for 20% more traffic to achieve the same power as an uncorrected test means leaving money on the table every single day.
When the chi-square approximation frays, you have several stronger options that modern computing makes trivial to implement. Fisher's exact test computes the exact probability of observing a table as extreme as the one you saw, conditioning on the marginal totals. It works for any sample size but can be conservative, especially when the table is balanced, leading to slightly higher p-values than necessary. Barnard's exact test drops the conditioning on margins, making it an unconditional alternative that frequently yields more power than Fisher's. Modern laptops handle the extra computation effortlessly, so the cost is negligible for most A/B test sizes. The two-proportion z-test relies on the normal approximation and shines when each variant contributes more than 500 observations and conversion rates exceed 5%. No continuity correction is needed, and the test aligns closely with the chi-square statistic without Yates' adjustment. Bayesian A/B testing places prior beliefs on conversion rates and updates them with data to produce posterior distributions. The result is a direct probability statement, such as "there's a 92% chance Variant B outperforms Variant A," without invoking asymptotic approximations. Priors can be weakly informative to let the data speak, and the method naturally handles rare events.
Simulation studies consistently show that exact tests and Bayesian approaches detect true effects more often than Yates' corrected chi-square when samples are small or conversion rates are low, while maintaining comparable Type I error control. The choice among them depends on your context. Fisher's is the safe default for tiny samples. Barnard's gives you a bit more power when you need it. The z-test is your workhorse for large samples. Bayesian methods give you interpretability that stakeholders actually understand.
Software Defaults, Code, and the Legacy Trap
SciPy's chi2_contingency function applies Yates' correction by default whenever the input is a 2×2 table, unless you set correction=False. This behavior mirrors a long-standing convention in statistical packages rather than a deliberate methodological recommendation. Consequently, many analysts run a test, see a non-significant p-value, and assume they have made a principled choice, when in fact they have simply accepted a legacy setting. R's chisq.test takes the opposite stance: it does not apply the correction unless you explicitly request correct = TRUE. The divergence highlights how software defaults shape practice, leaving Python users with an extra responsibility to know what they are toggling.
Industry experimentation platforms at large tech companies typically bypass Yates' correction altogether, favoring sequential testing, Bayesian frameworks, or plain z-tests. Their internal documentation rarely mentions the correction, signaling that the field has moved beyond the 1930s workaround. When you read the experiment review guides from companies that run thousands of tests daily, you see discussions of prior selection, sequential analysis boundaries, and variance reduction techniques. Yates' name is conspicuously absent. That silence is its own form of consensus.
When documenting decisions, add a note in your experiment logs whenever Yates' correction was applied and why. If you're using it as a sensitivity check, say so. If you disabled it, explain that the sample size warranted an uncorrected test. Consider creating a small flowchart for your team's wiki, a visual decision tree based on the framework below. Make it visible. New team members shouldn't have to discover this through painful trial and error. One final audit task: search your existing codebase for chi2_contingency calls without an explicit correction parameter. Those legacy defaults might be silently influencing your results. Assess whether disabling the correction would change any conclusions you've drawn.
Here is exactly how to handle this in code, with explicit commentary on what each line does.
Python/SciPy:
from scipy.stats import chi2_contingency, fisher_exact
# Your 2x2 table: [conversions, non-conversions] for each variant
table = [[2, 48], [3, 47]]
# Default (with Yates' correction) - explicitly for clarity
chi2_corr, p_corr, dof, expected = chi2_contingency(table, correction=True)
# Without correction
chi2_uncorr, p_uncorr, dof, expected = chi2_contingency(table, correction=False)
# Fisher's exact test for comparison
fisher_result = fisher_exact(table)
print(f"Yates corrected p-value: {p_corr:.4f}")
print(f"Uncorrected p-value: {p_uncorr:.4f}")
print(f"Fisher's exact p-value: {fisher_result[1]:.4f}")
# Expected counts help you diagnose approximation quality
print(f"Expected counts:\n{expected}")R:
# Your 2x2 table
table <- matrix(c(2, 48, 3, 47), nrow = 2)
# Chi-square without correction
chisq.test(table, correct = FALSE)
# Fisher's exact test
fisher.test(table)
# For larger samples, the two-proportion z-test
# prop.test automatically uses chi-square, equivalent to z-test for 2x2
prop.test(table, correct = FALSE)The key is explicitness. Don't let defaults decide for you. State your choice and justify it.
A Practical Decision Framework
Here is how to choose the right tool for your specific situation.
Step 1: Check expected counts. Compute the expected frequency for each cell. If any cell is less than 1 or more than 20% of cells are under 5, flag the chi-square approximation as suspect. This is your early warning system.
Step 2: Very small samples (total <30). Prefer Fisher's exact test. If power is critical and you can afford the extra computation, use Barnard's exact test. These are your precision instruments for tiny datasets.
Step 3: Moderate samples (30-200). Run both the uncorrected chi-square and Yates' corrected version. Treat the Yates' p-value as an upper bound, a conservative check, and the uncorrected p-value as a liberal estimate. If they agree, you have confidence. If they diverge, look to an exact test. This is your belt-and-suspenders approach.
Step 4: Large samples (>200) or solid normal approximation. Use the two-proportion z-test (no correction). The chi-square and z-test will give virtually identical results. This is your workhorse.
Step 5: Extremely rare events (<1% conversion) with moderate traffic. Apply Bayesian A/B testing with a weakly informative beta prior (e.g., Beta(1,1)). The posterior probability of improvement is intuitive and avoids the multiple-testing pitfalls of repeated significance checks. This is your modern, interpretable alternative.
Sample-size inflation. Simulations indicate that using Yates' correction can require roughly 10-20% more subjects to achieve the same power as an uncorrected test or an exact method. Factor this into your power calculations if you ever feel compelled to retain the correction. That 20% translates directly into opportunity cost.
Reporting best practices. Whenever you compare corrected and uncorrected results, present both p-values, note the correction's conservatism, and always accompany the test with the observed difference in proportions and a confidence (or credible) interval. A p-value without context is a number without meaning.
From Default Fix to Diagnostic Tool
Yates' correction answered a genuine problem in 1934: how to approximate a chi-square distribution when your data are discrete counts and your only tools are paper tables and a slide rule. Today, laptops can enumerate exact probabilities in milliseconds, and Bayesian machinery delivers intuitive probability statements without leaning on asymptotics. The correction's origin was a compromise born of necessity, not a statement of statistical optimality.
Treating it as a default adjustment throws away power. Studies using Yates' correction need roughly 10-20% more traffic to detect the same lift that an uncorrected test or an exact method would catch with fewer visitors. In the fast-moving world of digital experimentation, that inflation translates into missed opportunities and slower iteration cycles. When your competitor ships a better feature three weeks earlier because their statistical toolkit didn't hold them back, that 20% sample size penalty becomes a competitive disadvantage.
The evidence is clear: for tiny samples, exact tests (Fisher or Barnard) give you the most reliable inference; for moderate samples, run both corrected and uncorrected chi-square as a sanity check; for large samples, the plain two-proportion z-test suffices; and for rare events, Bayesian methods provide a natural, interpretable alternative. The field's consensus is growing: exact tests and Bayesian approaches outperform Yates' correction in the small-sample regimes that dominate many online experiments.
Call to action: Audit your A/B testing scripts, switch off the automatic Yates' correction unless you have a specific, documented reason, and adopt the decision framework outlined above. Let your statistical tool match the data context, not a legacy setting. By doing so, you reduce false negatives, accelerate learning, and keep your experimentation engine running at full throttle. The next time you stare at a dashboard showing a promising but non-significant result, you'll know exactly which button to click, which test to run, and which conclusion to trust. That knowledge is the difference between shipping value and shipping statistical superstition.
Now that the computational barriers have vanished, we can finally move from a "one-size-fits-all" default to a nuanced, context-driven toolbox, one that serves the goal of detecting real improvements, not hiding them behind a historical footnote. Your 1930s statistician, slide rule in hand, would be proud.